Queue consumption mode for AWS Lambda event source mappings (ESMs) that subscribe to Apache Kafka is now available. Queue consumption mode is available on the self-managed Apache Kafka ESM. It covers both Kafka clusters you operate yourself and managed Kafka offerings such as Confluent Cloud and other providers that offer Kafka 4.2 or later with Kafka Queue support.
With this launch, you can build queue-style event-driven workloads on Kafka that process records with high parallelism that scales concurrency to match your backlog instead of your topic layout. Queue mode achieves this using Kafka Share Groups (KIP-932), introduced in Kafka 4.0. With share groups, multiple consumers read concurrently from the same partitions, so Lambda can process records from a single partition across many parallel invocations. The result is parallelism that is decoupled from partition count, without head-of-line blocking and without introducing a second messaging system into your…
