Amazon Kinesis
Real-time streaming data collection and processing
Kinesis is like a fire hose for data streams. Imagine thousands of sensors, mobile apps, or servers all sending data continuously: clickstreams, logs, IoT telemetry. Kinesis captures this data in real-time, buffers it, and lets you process it immediately. It's like having a conveyor belt that never stops: data flows in one end, you process it in the middle, and results come out the other end, all in real-time. Perfect for scenarios where you need to react to data immediately, not hours later after batch processing.
The Kinesis family now centers on three services: Kinesis Data Streams (real-time data ingestion), Amazon Data Firehose (load streams to S3, Redshift, or OpenSearch), and Kinesis Video Streams (video ingestion). Stream SQL processing moved to Amazon Managed Service for Apache Flink, since Kinesis Data Analytics for SQL was discontinued and its applications deleted starting January 27, 2026. Data Streams uses shards (throughput units): each shard handles 1MB/sec input, 2MB/sec output. Producers write records, consumers read and process them.
Key Capabilities
- Kinesis Data Streams ingests real-time streaming data using shards as throughput units; each shard supports 1MB/sec input and 2MB/sec output
- Kinesis Data Firehose loads streaming data directly to S3, Redshift, OpenSearch, and Splunk with optional transformation via Lambda, requiring no consumer code
- Kinesis Data Analytics runs SQL queries on live streams for real-time aggregations, anomaly detection, and filtering without managing infrastructure
- Kinesis Video Streams ingests, stores, and processes live video from cameras and devices for playback and ML analysis
- Enhanced fan-out provides dedicated 2MB/sec throughput per registered consumer, eliminating contention when multiple consumers read the same stream
- Data retention is configurable from 24 hours up to 365 days, allowing consumers to replay historical stream data for reprocessing or backfill
Gotchas & Constraints
Gotcha #1: Shard limits can cause throttling; monitor metrics and split shards proactively. Gotcha #2: Firehose is near-real-time (it buffers by size or time before delivery), so it is not a substitute for the sub-second latency of Data Streams. Gotcha #3: Consumers must checkpoint progress; if a consumer crashes, it resumes from the last checkpoint. Constraints: Maximum 1MB record size, maximum 1,000 records/second per shard (PutRecords), and resharding takes time (split/merge shards).
A gaming company tracks player actions in real-time: every click, move, and purchase. They send 100,000 events/second to Kinesis Data Streams (100 shards). Lambda functions consume the stream, calculating real-time leaderboards and detecting cheating patterns. Amazon Data Firehose loads raw events to S3 for long-term analytics. Amazon Managed Service for Apache Flink runs a streaming SQL query on the stream: 'count purchases by game level in 5-minute windows', and sends results to DynamoDB for dashboards. When they detect a player exploiting a bug (abnormal score increase), they trigger a Lambda function to flag the account immediately. During a new game launch, traffic spikes 10x; they scale from 100 to 1,000 shards in 30 minutes.
The Result
real-time insights, immediate fraud detection, and scalable data ingestion.