What is Apache Flink?

Apache Flink is an open source stream processing engine, developed starting in 2010 at the Technical University of Berlin and now a project of the Apache Software Foundation: it runs continuous computation over events that arrive without interruption, keeping an internal state (counters, aggregations, histories) that it updates event by event. It distinguishes event time, the moment an event actually occurred, from processing time, the moment the system receives it, a distinction that matters when events arrive late or out of order, as is typical in IoT or mobile streams. Time windows (tumbling, sliding, session) let you aggregate events over defined intervals. Periodic checkpointing of state to durable storage guarantees exactly-once semantics for the internal state: on failure the job restarts from the last checkpoint without losing or double-counting events. For that guarantee to hold end to end you also need a transactional or idempotent sink, otherwise the restart re-emits results already written downstream.

See also: Apache Kafka · The open source distributed log for event streaming: topics split into partitions, read by independent consumer groups.

Compute engine, not transport: the difference from Kafka

Flink and Apache Kafka are complementary, not competing tools, and conflating them leads to picking the wrong tool. Kafka is the transport layer: a distributed log that stores and delivers events to consumers without processing them. Flink is the compute layer: it reads events (typically from Kafka), applies stateful processing logic, such as joins between two streams, deduplication, pattern detection (Flink CEP), windowed aggregations, and writes the result elsewhere. A real pipeline uses both together: Kafka carries the raw events, Flink transforms them in real time, the result lands in a second Kafka topic or directly in a database. The same principle underlies many streaming CDC architectures, where Flink applies transformation logic on the change stream before it reaches its final destination, which sets it apart from traditional batch processing, which processes accumulated data at fixed intervals rather than event by event.

A typical enterprise case: a payment network uses Flink to calculate in real time how many transactions a card has made in the last minute and compare it against the user's historical average, blocking the operation if the deviation exceeds a threshold, before authorization even completes. A second common case is industrial monitoring: IoT sensors send continuous readings and Flink computes moving averages over five-minute windows to flag temperature or vibration anomalies before they turn into a failure.

When to choose it and when not to

Flink is the right choice when the processing logic needs state that survives across events, time windows, joins across multiple streams, or exactly-once guarantees that an ad hoc Kafka consumer cannot offer without reinventing much of the engine. It is not the right choice when a simple, stateless row-by-row transformation is enough, for which a lightweight consumer or Kafka Streams is cheaper to run, nor when the volumes and tolerated latency make a nightly batch job simpler and cheaper. Running a Flink cluster in production carries a real operational cost, memory and checkpoint tuning, task manager sizing, which managed services (Amazon Managed Service for Apache Flink, Ververica Cloud, native offerings from Databricks and Confluent) reduce but do not eliminate. The question is not "do we need streaming," but whether the business logic genuinely requires state and time, or whether frequent batch would be indistinguishable to the end user.