Real-Time Streaming Analytics
Streaming isn't batch made fast - it's a different way of thinking about time, completeness, and the cost of being right.
The question nobody asks first
When teams decide to "do real-time," they usually start with the wrong question: which engine? Kafka or Pulsar, Flink or Spark, managed or self-hosted. These are real decisions, but they are the last ones. The first question is quieter and more important: what does it cost me when an answer arrives late? If the honest answer is "nothing much," you do not have a streaming problem - you have a batch job and a deadline you can meet by scheduling it more often. Streaming is a tax you pay in operational complexity, and you should only pay it to buy latency you will actually spend.
The use cases that justify the tax share a single trait: freshness is part of correctness. A fraud score is worthless after the money leaves. An alert that fires an hour after the outage is an apology, not a defense. A recommendation based on last week's behavior ignores the click the user made ten seconds ago. In each case a perfectly accurate but late answer is, functionally, a wrong answer. That reframing - latency as a correctness property, not a performance nicety - is the mental shift that makes everything else fall into place.
Unbounded data breaks your intuitions
Batch processing rests on a comfortable assumption: you can read all the data, then compute. Streaming removes that assumption permanently. The data never ends. There is no moment when you have "everything," because more is always arriving. You are computing on a continuously moving slice of an infinite dataset, and you must produce answers you are willing to stand behind before the data stops - because it never will.
This is why streaming has its own vocabulary that batch never needed. You cannot aggregate infinity, so you cut it into windows. You cannot trust arrival order, because a phone in a tunnel will flush thirty buffered events at once, so you distinguish event time (when something happened) from processing time (when you saw it) and you bucket by the former. And because you bucket by event time but data arrives out of order, you need a way to decide when a bucket is "done enough" to emit - the watermark, a moving bet that says I think I've now seen everything up to this point in time. Set the bet conservatively and you wait longer but capture more stragglers; set it aggressively and you answer faster but drop a few. There is no free lunch here, only an explicit, tunable trade between latency and completeness.
The log changed the architecture
The other thing that makes modern streaming work is deceptively humble: the durable, partitioned, append-only log. Kafka's central insight was to stop treating messages as things to deliver and delete, and start treating them as an immutable record that is retained whether or not anyone has read it yet. That one change ripples outward into superpowers. Multiple independent consumers can read the same stream. A new service can be added next year and replay history from the beginning to bootstrap its state. A consumer that falls behind doesn't lose data - the records simply wait in the log, which acts as an enormous shock absorber against backpressure.
Partitions give you ordering where you need it (within a key) and parallelism everywhere else. Offsets let each consumer track its own progress without coordinating with others. Replication and in-sync replica sets turn "a broker died" from an incident into a non-event. The log is the closest thing distributed systems have to a universal backbone: databases publish change streams into it, services subscribe, analytics engines aggregate from it, and everyone is decoupled in both time and space.
Exactly-once is a promise about effects
Newcomers fixate on "exactly-once" as if it means each record is physically touched once. It almost never does. What production systems actually guarantee is that the effect on output and state is as if each record were processed once - achieved by committing state and output offsets atomically, and by making sinks either idempotent or transactional. The subtle, expensive lesson is that this guarantee is end-to-end only if every hop in the chain cooperates. A flawlessly checkpointed Flink job that writes to a sink which cannot deduplicate gives you at-least-once overall. The chain is exactly as strong as its weakest link, and most "exactly-once" outages are really a forgotten link.
Operate it like a service, because it is one
A batch job runs and stops; a streaming job runs forever, which means it is not a program you ship but a service you operate. The metric that matters most is consumer lag - and you alert on its trend, not its absolute value, because steadily climbing lag means you are losing the race even while the number still looks small. You watch watermark progress, because a stalled watermark silently freezes every window. You enforce schema compatibility, because one incompatible field pushed to a busy topic can poison every downstream team at once. You put TTLs on state so it doesn't grow without bound, and you test recovery by actually killing a task, not by hoping.
None of this is glamorous, and that is the point. Done well, a streaming system is a quiet, self-healing service that turns an endless firehose of raw events into fresh, trustworthy answers the moment they matter. Done poorly, it is a pager that never sleeps. The difference is rarely the engine you chose. It is whether you respected what unbounded data demands - and whether you only paid the streaming tax for latency you truly needed.
No comments:
Post a Comment