Real-time is the default request and frequently the wrong answer. The question that settles it is how quickly somebody or something will act on the data, rather than how fresh it could theoretically be. Data arriving in two seconds that is looked at once a day is expensive theatre.
Batch is the sensible default
A scheduled job is easier to build, easier to reason about, and dramatically easier to recover. If a batch run fails, you fix the problem and run it again. That single property removes most of the operational burden that streaming systems carry permanently.
Reporting, reconciliation, enrichment, model training and most analytics work are batch problems. Running them hourly rather than nightly covers a surprising share of the cases people describe as needing real-time.
When streaming genuinely earns it
- A decision is made automatically while the customer waits, such as fraud scoring or routing an inbound enquiry.
- A person is waiting on the screen for the result.
- The value of the data decays fast enough that an hour old is worthless.
- Volume is high enough that batching would create a window nothing can process.
If none of those hold, streaming is buying latency nobody consumes, at a cost paid every day in operational complexity.
Ask who acts on the data and how fast. If the honest answer is a person, tomorrow morning, you have a batch problem and you can spend the saved complexity on something that matters.
Lena Fischer, Solutions Architect, Engineered With AI
The failure modes are different
Batch fails loudly and locally: a run breaks, an alert fires, you fix and rerun. Streaming fails subtly and continuously, in ways that are much harder to see: events arriving out of order, a consumer falling behind, a partial outage that drops a slice of records nobody notices for a week.
That difference is the real cost, and it is an ongoing one rather than a build cost. It needs monitoring, alerting on lag, and somebody who understands the system when it misbehaves at an awkward hour.
Replay is the question to ask early
Whatever you build, ask what happens when you need to reprocess. A bug in transformation logic is normal and discovering it means running the corrected logic over historical data. Batch systems reprocess naturally. Streaming systems only do so if retention and replay were designed in from the start.
Retrofitting replay is expensive, so decide it at design time even if the first version never uses it.
Micro-batch covers most of the middle
Running every few minutes gives most of the freshness people actually want with most of the operational simplicity of batch. For a large share of use cases described as real-time, a five-minute schedule is genuinely indistinguishable to the consumer and far cheaper to run.
Idempotency matters in both
Whichever you pick, make processing safe to repeat. Reruns happen after failures in batch and duplicates happen routinely in streaming, and a pipeline that double-counts on reprocessing will produce numbers nobody can trust and errors that are painful to unwind.
Decide from the consumer backwards
Start with what acts on the output and how quickly, then pick the simplest architecture that satisfies it. That order produces systems that are cheaper to run and easier to hand over. Our automation index covers where these decisions land by sector.
Schema changes break both, differently
An upstream system adding or renaming a field is routine and it is the most common cause of pipeline failure in practice. Batch fails at the next run, loudly, with a clear place to look. Streaming keeps consuming and silently drops or mangles the affected records, which is discovered much later and is harder to correct.
Validate incoming records against an expected shape and route anything unexpected to a queue rather than discarding it. That single control turns a class of invisible data loss into a visible list somebody can work through.
Backfill is the requirement people forget
Historical data almost always needs loading at some point, whether at launch or after a correction, and the volume is far larger than the ongoing flow. A streaming design that assumes a steady trickle will fall over on the first backfill.
Plan the backfill path separately from the live path. In many systems a batch job for history alongside a stream for current data is the cleanest answer, rather than forcing one mechanism to do both.
Not sure which your use case needs?
Tell us how quickly the data has to be acted on and we will tell you honestly whether streaming earns its complexity here.