Batch vs streaming: what is the difference?

Batch and streaming are the two ways of processing data in a pipeline. Batch works in chunks, on a schedule: it collects data and processes it all together, nightly or hourly. Streaming processes it event by event, as soon as it is generated, with latencies from seconds down. Which one to pick depends on how much that data's freshness is worth to you. In between there is a third way, micro-batch, which processes small, very frequent chunks and delivers reasonably fresh data without the operational complexity of an always-on event system. Modern data platforms such as Databricks, Snowflake or BigQuery support both models on the same architecture, so the choice between batch and streaming comes down to an economic calculation: how much it costs to build and maintain streaming infrastructure versus the real value that freshness brings to the company's decisions.

See also: Lambda vs Kappa architecture · Lambda runs two parallel paths, batch and streaming, to merge history and real time; Kappa unifies them into one replayable log.

How the two approaches work

Batch is the workhorse: simple to build, easy to test and to rerun when something breaks (replay the chunk and move on), cheap because compute only spins up when needed. Streaming requires always-on infrastructure (event platforms like Kafka, stream processing engines) and a more delicate handling of failures, late events and duplicates: more powerful, but more expensive to build and operate. In between sits micro-batch: small, frequent chunks, every few minutes, delivering "fresh enough" data with the operational simplicity of batch. Modern platforms (Databricks, Snowflake, BigQuery, Microsoft Fabric) support both models on the same architecture, so the constraint is no longer technological: it is economic and organizational.

Which one to choose for your business

Streaming is not "better": it is for when freshness is worth its cost. If a decision changes based on what happened in the last few seconds (a fraud to block, a machine to stop, a price to update), streaming pays for itself. But a sales report, a forecast or an executive dashboard does not improve because the data is five seconds old instead of one hour: most real needs are covered by batch and micro-batch, at a fraction of the cost. The classic mistake is demanding "everything in real time" as a default requirement and then paying for years for infrastructure no decision uses. Start with batch, measure where latency actually matters, and move only those flows to streaming.