Amazon Kinesis (Streams, Firehose)
Key concepts
Kinesis Data Streams
Kinesis Data Firehose
Kinesis Data Analytics
Shard management
Enhanced fan-out
Overview
Amazon Kinesis is a family of managed services for collecting, processing, and analyzing streaming data, meaning data that arrives continuously in small records (clickstreams, logs, IoT telemetry, application events) rather than in scheduled batches. The family includes Kinesis Data Streams (a low-latency, replayable streaming buffer), Kinesis Data Firehose (a fully managed delivery pipeline to storage and analytics destinations), and Kinesis Data Analytics (SQL or Apache Flink processing over a stream).
This topic is high-yield on the SAA-C03 exam through "real-time ingestion" scenarios. You should be able to separate Data Streams from Firehose, know when custom consumers and replay are required, understand shard capacity and management, recognize when Enhanced Fan-Out is the right answer, and pair Kinesis with Lambda and S3 in an end-to-end pipeline.
Stream for Custom Real-Time, Firehose for Managed Delivery
Choose Kinesis Data Streams when you need sub-second custom processing, multiple independent consumers, or data replay. Choose Kinesis Data Firehose when you want a fully managed, near-real-time pipeline that buffers and loads data into S3, Redshift, OpenSearch, or Splunk with no servers to run.
Memorize the split: Data Streams is real-time (about 200 ms with Enhanced Fan-Out), replayable for up to 365 days, and you manage shards and consumers. Firehose is near-real-time (buffered by size and time, minimum about 60 seconds), fully managed, has no retention or replay, and can transform records with Lambda before delivery.
Key Concepts
Kinesis Data Streams
A Durable, Replayable Stream Buffer
Kinesis Data Streams (KDS) is a real-time streaming service that ingests records and holds them in ordered shards for a configurable retention period (default 24 hours, extendable to 365 days). Producers write records using a partition key, which decides the target shard, and consumers read records in order within each shard. Because data persists for the retention window, multiple applications can read the same stream independently and you can replay history after a failure. KDS supports a provisioned capacity mode (you set the shard count) and an on-demand mode (capacity scales automatically based on throughput).
import boto3, json
kinesis = boto3.client("kinesis", region_name="us-east-1")
kinesis.put_record(
StreamName="orders-stream",
Data=json.dumps({"order_id": 4821, "amount": 39.95}),
PartitionKey="customer-1057", # determines which shard the record lands in
)Kinesis Data Firehose
Fully Managed Delivery Pipeline
Kinesis Data Firehose is a fully managed service that captures streaming data and delivers it to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk, plus supported third-party endpoints. It buffers records by a configurable size and time interval, so delivery is near-real-time (the buffer minimum is about 60 seconds, which is why Firehose is not used for sub-second needs). Firehose can invoke a Lambda function to transform records in flight, convert JSON to Parquet or ORC, compress output, and retry failed deliveries. There are no shards to provision and no servers to manage, because Firehose scales automatically.
Data Streams vs Data Firehose
| Dimension | Kinesis Data Streams | Kinesis Data Firehose |
|---|---|---|
| Latency | Real-time (about 200 ms with Enhanced Fan-Out) | Near-real-time (buffer, about 60 s minimum) |
| Management | You manage shards and consumers | Fully managed, auto scaling |
| Data retention / replay | 1 to 365 days, replayable | No storage, no replay |
| Destinations | Custom consumers, Lambda, Analytics | S3, Redshift, OpenSearch, Splunk |
| In-flight transform | In your consumer code | Optional Lambda transform built in |
| Best for | Custom real-time processing | Loading streams into storage and analytics |
Kinesis Data Analytics
Process the Stream as It Flows
Kinesis Data Analytics (KDA) lets you run continuous queries over streaming data without managing servers. It offers a SQL engine for windowed aggregations, filters, and joins, and an Apache Flink runtime (Managed Service for Apache Flink) for richer stateful stream processing in Java, Scala, or Python. KDA reads from a Data Stream or Firehose, computes results in real time (running totals, anomaly detection, sliding-window metrics), and writes output to another stream, Firehose, or Lambda. Use it when transformation or aggregation must happen on the live stream before the data lands.
Shard Management
Shards Are the Capacity Unit
In provisioned mode, a shard is the throughput unit of a Data Stream. Each shard ingests up to 1 MB per second or 1,000 records per second for writes, and supports up to 2 MB per second of reads shared across standard consumers. Total stream capacity is shards multiplied by these limits, so you add shards to scale. Resizing is done by resharding: a split divides one shard into two to add capacity, and a merge combines two shards into one to reduce cost. The partition key distributes records across shards, so a poorly chosen key creates a hot shard that throttles while others sit idle. On-demand mode removes manual shard management by scaling capacity automatically up to default account limits.
Provisioned vs On-Demand Capacity
| Dimension | Provisioned Mode | On-Demand Mode |
|---|---|---|
| Capacity control | You set and reshard shards | Scales automatically |
| Per-shard write limit | 1 MB/s or 1000 records/s | Managed for you |
| Pricing | Per shard-hour plus payload | Per stream-hour plus data volume |
| Best for | Predictable, steady throughput | Spiky or unknown throughput |
Enhanced Fan-Out
Dedicated Throughput Per Consumer
By default, all standard consumers of a shard share a single 2 MB per second read pipe and poll the stream with GetRecords, so adding consumers increases latency and contention. Enhanced Fan-Out (EFO) gives each registered consumer its own dedicated 2 MB per second per shard read throughput and pushes records over an HTTP/2 connection using SubscribeToShard, which lowers delivery latency to roughly 200 milliseconds. Use EFO when multiple applications must read the same stream in parallel without competing for the shared throughput, at the cost of additional per-consumer and data-retrieval charges.
Standard Consumer vs Enhanced Fan-Out
| Dimension | Standard Consumer | Enhanced Fan-Out |
|---|---|---|
| Read throughput | 2 MB/s shared per shard | 2 MB/s dedicated per consumer per shard |
| Delivery model | Pull (GetRecords polling) | Push (SubscribeToShard, HTTP/2) |
| Typical latency | Higher under many consumers | About 200 ms |
| Cost | Lower | Per-consumer plus retrieval charges |
| Best for | One or few consumers | Many parallel low-latency consumers |
Best Practices
1. Pick the service by latency and ownership
├── Data Streams: sub-second, custom consumers, replay
└── Firehose: managed near-real-time load into storage
2. Choose a high-cardinality partition key
└── Spreads load evenly and avoids hot shards
3. Right-size capacity
├── On-demand for spiky or unknown throughput
└── Provisioned plus resharding for steady, predictable load
4. Use Enhanced Fan-Out for many parallel consumers
└── Dedicated 2 MB/s per consumer with about 200 ms latency
5. Land raw data in S3 with Firehose
├── Transform with Lambda in flight if needed
└── Convert JSON to Parquet for cheaper analytics
6. Set retention to cover your recovery window
└── Extend up to 365 days when replay is requiredCommon Pitfalls
Pitfall 1: Using Firehose for Sub-Second Processing
Mistake: Selecting Kinesis Data Firehose for a workload that must react to events within milliseconds.
Why it fails: Firehose buffers by size and time with a minimum interval of about 60 seconds, so it delivers in near-real-time and cannot meet sub-second requirements.
Correct Approach: Use Kinesis Data Streams (with Enhanced Fan-Out where needed) for real-time, low-latency processing.
Pitfall 2: Expecting Replay or Storage From Firehose
Mistake: Assuming Firehose retains records so a downstream app can re-read them after a failure.
Why it fails: Firehose has no retention period and no replay; once buffered records are delivered they are gone from the pipeline.
Correct Approach: Use Data Streams when replay matters (retention up to 365 days), or have Firehose deliver to S3 and re-read from there.
Pitfall 3: Throttling From a Hot Shard
Mistake: Using a low-cardinality partition key (for example, a constant value or a single device ID) on a provisioned stream.
Why it fails: Records concentrate on one shard, exceeding its 1 MB/s or 1,000 records/s write limit and triggering ProvisionedThroughputExceeded throttling while other shards stay idle.
Correct Approach: Choose a high-cardinality partition key to spread load, reshard to add capacity, or switch to on-demand mode.
Pitfall 4: Adding Consumers on a Shared Read Pipe
Mistake: Attaching several standard consumers to the same stream and expecting each to get full throughput.
Why it fails: Standard consumers share a single 2 MB/s read limit per shard, so latency and contention grow as you add applications.
Correct Approach: Register consumers with Enhanced Fan-Out so each gets a dedicated 2 MB/s per shard.
Test Your Knowledge
A company needs to capture clickstream events and load them into Amazon S3 for later analysis. There is no need for sub-second processing or replay, and the team wants the least operational overhead. Which service fits best?
Five separate applications must each read the same Kinesis Data Stream in parallel with the lowest possible latency. With standard consumers they are throttling each other. What should the architect do?
A provisioned Kinesis Data Stream is returning ProvisionedThroughputExceeded errors on one shard while others are nearly idle. What is the most likely cause?
Related Services
Quick Reference
Well-Known Limits
Kinesis Limits and Facts
| Item | Value |
|---|---|
| Shard write capacity | 1 MB/s or 1000 records/s |
| Shard read capacity (standard, shared) | 2 MB/s per shard |
| Enhanced Fan-Out read capacity | 2 MB/s per consumer per shard |
| Maximum record (data blob) size | up to 10 MiB before base64 encoding |
| Data Streams retention | 24 hours default, up to 365 days |
| Firehose buffer interval minimum | About 60 seconds |
| Enhanced Fan-Out latency | About 200 ms |
Common CLI Commands
# Create a provisioned stream with 4 shards
aws kinesis create-stream --stream-name orders-stream --shard-count 4
# Put a single record (partition key picks the shard)
aws kinesis put-record \
--stream-name orders-stream \
--partition-key customer-1057 \
--data '{"order_id":4821,"amount":39.95}'
# Reshard by splitting a shard to add capacity
aws kinesis split-shard \
--stream-name orders-stream \
--shard-to-split shardId-000000000000 \
--new-starting-hash-key 170141183460469231731687303715884105728
# Register a consumer for Enhanced Fan-Out
aws kinesis register-stream-consumer \
--stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/orders-stream \
--consumer-name realtime-fraud
# Create a Firehose delivery stream to S3
aws firehose create-delivery-stream \
--delivery-stream-name orders-to-s3 \
--delivery-stream-type DirectPut \
--s3-destination-configuration RoleARN=arn:aws:iam::123456789012:role/firehoseRole,BucketARN=arn:aws:s3:::orders-lake