What is Apache Kafka?

Apache Kafka is an open source event streaming platform, born at LinkedIn in 2011 and now a project of the Apache Software Foundation: it works as an append-only distributed log where producers write events and consumers read them independently, each at its own pace. Data is organized into topics, in turn split into partitions to scale horizontally and parallelize reads and writes; within a single partition write order is guaranteed, but not across different partitions of the same topic. Consumers organize into consumer groups, which split a topic's partitions to process the stream in parallel without duplicating work. Events remain in the log for a configurable retention period instead of disappearing on read as in a traditional queue, so multiple applications can reread the same stream at different times. With Kafka 4.0 (March 2025) the ZooKeeper component was removed entirely: cluster coordination moved to KRaft, the internal Raft-based consensus, which simplifies installation and operations.

Where it sits in a data platform

Kafka does not process or transform data: it is the transport layer that decouples whoever produces events (an e-commerce store, an IoT sensor, a microservice) from whoever consumes them (a data warehouse, an alerting engine, another service), without either side needing to know about the other or be online at the same time. It is the typical component upstream of a streaming data ingestion pipeline, often paired with a stream processing engine (Kafka Streams, Flink) to transform data in transit, and underlies many CDC architectures feeding a data lakehouse. The configurable delivery guarantees are at-most-once and at-least-once. Exactly-once processing also exists, but it applies to flows that stay inside Kafka, where an idempotent producer plus transactions make read, process and write a single atomic operation. Towards an external system delivery stays at-least-once, and uniqueness of the result has to be enforced downstream with idempotent writes.

A typical enterprise case: a bank uses Kafka to propagate every card transaction in real time to three separate, independent systems, fraud detection, the data warehouse for reporting, and the customer push notification system, without any of the three consumers slowing each other down or having to query the transactional system directly.

When to choose it and when not to

Kafka is the right choice when multiple independent consumers need the same event stream, throughput is high and sustained, or the ability to replay event history matters. It is not the right choice for low, sporadic volumes, where a traditional queue (RabbitMQ, Amazon SQS) is simpler to operate, nor when you want to avoid running the infrastructure yourself: cloud provider managed services (Amazon MSK, Confluent Cloud, Google Cloud Managed Service for Apache Kafka, Azure Event Hubs through its Kafka-compatible endpoint) cover most of these cases while reducing operational overhead. Direct alternatives also exist, such as Apache Pulsar or Redpanda, each with different architectural and compatibility trade-offs. The question is not "Kafka, yes or no" in the abstract, but whether the problem genuinely needs a multi-consumer distributed log or a simpler queue will do.