Copyright | (c) 2013-2016 Brendan Hay |
---|---|
License | Mozilla Public License, v. 2.0. |
Maintainer | Brendan Hay <brendan.g.hay@gmail.com> |
Stability | auto-generated |
Portability | non-portable (GHC extensions) |
Safe Haskell | None |
Language | Haskell2010 |
Gets data records from an Amazon Kinesis stream's shard.
Specify a shard iterator using the ShardIterator
parameter. The shard
iterator specifies the position in the shard from which you want to
start reading data records sequentially. If there are no records
available in the portion of the shard that the iterator points to,
GetRecords returns an empty list. Note that it might take multiple
calls to get to a portion of the shard that contains records.
You can scale by provisioning multiple shards per stream while
considering service limits (for more information, see
Streams Limits
in the Amazon Kinesis Streams Developer Guide). Your application
should have one thread per shard, each reading continuously from its
stream. To read from a stream continually, call GetRecords in a loop.
Use GetShardIterator to get the shard iterator to specify in the
first GetRecords call. GetRecords returns a new shard iterator in
NextShardIterator
. Specify the shard iterator returned in
NextShardIterator
in subsequent calls to GetRecords. Note that if
the shard has been closed, the shard iterator can't return more data
and GetRecords returns null
in NextShardIterator
. You can
terminate the loop when the shard is closed, or when the shard iterator
reaches the record with the sequence number or other attribute that
marks it as the last record to process.
Each data record can be up to 1 MB in size, and each shard can read up
to 2 MB per second. You can ensure that your calls don't exceed the
maximum supported size or throughput by using the Limit
parameter to
specify the maximum number of records that GetRecords can return.
Consider your average record size when determining this limit.
The size of the data returned by GetRecords varies depending on the
utilization of the shard. The maximum size of data that GetRecords
can return is 10 MB. If a call returns this amount of data, subsequent
calls made within the next 5 seconds throw
ProvisionedThroughputExceededException
. If there is insufficient
provisioned throughput on the shard, subsequent calls made within the
next 1 second throw ProvisionedThroughputExceededException
. Note that
GetRecords won't return any data when it throws an exception. For
this reason, we recommend that you wait one second between calls to
GetRecords; however, it's possible that the application will get
exceptions for longer than 1 second.
To detect whether the application is falling behind in processing, you
can use the MillisBehindLatest
response attribute. You can also
monitor the stream using CloudWatch metrics and other mechanisms (see
Monitoring
in the Amazon Kinesis Streams Developer Guide).
Each Amazon Kinesis record includes a value,
ApproximateArrivalTimestamp
, that is set when a stream successfully
receives and stores a record. This is commonly referred to as a
server-side timestamp, whereas a client-side timestamp is set when a
data producer creates or sends the record to a stream (a data producer
is any data source putting data records into a stream, for example with
PutRecords). The timestamp has millisecond precision. There are no
guarantees about the timestamp accuracy, or that the timestamp is always
increasing. For example, records in a shard or across a stream might
have timestamps that are out of order.
- getRecords :: Text -> GetRecords
- data GetRecords
- grLimit :: Lens' GetRecords (Maybe Natural)
- grShardIterator :: Lens' GetRecords Text
- getRecordsResponse :: Int -> GetRecordsResponse
- data GetRecordsResponse
- grrsNextShardIterator :: Lens' GetRecordsResponse (Maybe Text)
- grrsMillisBehindLatest :: Lens' GetRecordsResponse (Maybe Natural)
- grrsResponseStatus :: Lens' GetRecordsResponse Int
- grrsRecords :: Lens' GetRecordsResponse [Record]
Creating a Request
Creates a value of GetRecords
with the minimum fields required to make a request.
Use one of the following lenses to modify other fields as desired:
data GetRecords Source #
Represents the input for GetRecords.
See: getRecords
smart constructor.
Request Lenses
grLimit :: Lens' GetRecords (Maybe Natural) Source #
The maximum number of records to return. Specify a value of up to
10,000. If you specify a value that is greater than 10,000,
GetRecords throws InvalidArgumentException
.
grShardIterator :: Lens' GetRecords Text Source #
The position in the shard from which you want to start sequentially reading data records. A shard iterator specifies this position using the sequence number of a data record in the shard.
Destructuring the Response
Creates a value of GetRecordsResponse
with the minimum fields required to make a request.
Use one of the following lenses to modify other fields as desired:
data GetRecordsResponse Source #
Represents the output for GetRecords.
See: getRecordsResponse
smart constructor.
Response Lenses
grrsNextShardIterator :: Lens' GetRecordsResponse (Maybe Text) Source #
The next position in the shard from which to start sequentially reading
data records. If set to null
, the shard has been closed and the
requested iterator will not return any more data.
grrsMillisBehindLatest :: Lens' GetRecordsResponse (Maybe Natural) Source #
The number of milliseconds the GetRecords response is from the tip of the stream, indicating how far behind current time the consumer is. A value of zero indicates record processing is caught up, and there are no new records to process at this moment.
grrsResponseStatus :: Lens' GetRecordsResponse Int Source #
The response status code.
grrsRecords :: Lens' GetRecordsResponse [Record] Source #
The data records retrieved from the shard.