The desert array does not point all twenty-seven dishes at one target and call it done. Each antenna captures an independent slice of sky; the correlator stitches them into a single aperture spanning miles. That synthesis is slow—twenty-seven data streams, each carrying its own ionospheric phase delay, arriving microseconds apart. When we sat down last December to rebuild Array's ingestion engine, we assumed the problem was throughput. It was not.
Throughput Is a Vanity Metric
The pipeline we inherited ingested 2.8 million events per minute on a clear day. It also discarded forty-one percent of them as unparseable noise before any analyst saw the data. That noise was not random—it was telemetry from sensors we had deployed six months earlier and forgotten to register in the catalog. In February, we halted all throughput optimizations and asked a simpler question: what were we losing?
Forty-one percent of our “noise” was signal from sensors we had deployed and then forgotten.
By late May, the rebuilt pipeline processed twenty-eight percent fewer events per minute and surfaced twice the actionable detections. The dishes did not get faster; they got quieter—and the quieter sky revealed everything we had been missing.