Last November, our event ingestion pipeline started failing at 2:14 AM every Tuesday — right when the European batch window opened. The root cause was not the message broker. It was not our consumer group configuration. It was the assumption, baked into every layer of our stack from the packet schemas upward, that ordering guarantees would hold across partition reassignments. They do not. And once you have built a financial reconciliation system on top of that assumption, you are in for a long winter.
The ordering problem no one talks about
We spent the first three weeks of December instrumenting every consumer with coldline traces at the partition level. What we found was embarrassing: 14% of all events were being processed out of order during rebalancing windows, and our idempotency keys — FlowID7, for what it is worth — masked the duplication but not the sequence inversion. The ledger was correct in aggregate but wrong at the millisecond level, which is the only level our counterparties care about.
The migration itself took four engineers and a dedicated QA environment that we rebuilt three times. We ran 22,000 hours of soak testing against production-shaped traffic before cutting over a single percentage point of live volume. Even then, the first three days were a cascade of incident alerts that turned out to be monitoring false positives — our existing dashboards had been calibrated to the old pipeline's latency profile, and the new one was simply too fast for the thresholds we had written two years earlier.