Silent Systems and the Hot Particle Problem

Cold-war survey meters have a lesson for anyone running production systems.

The CD V-700 sat on the steel workbench with its needle pinned at zero — exactly where it had been for the three weeks since I started calibrating it. The check source, a depleted uranium wafer the size of a guitar pick, should have pushed the CPM needle past eight hundred. Instead the meter was silent. Not broken. Just drifted. The potentiometer had crept six degrees counterclockwise since its last certification stamp in 1984. Nobody had noticed because nobody checks instruments that read zero.

The Calibration Drift Nobody Talks About

In production systems, we call this alert threshold creep. A latency p99 drifts from two hundred milliseconds to two-forty over eight months, and nobody wakes up because the alarm is set at five hundred. The drift is invisible in dashboards that only surface anomalies. The meters that read normal are the ones you stop looking at. I spent three afternoons in that lab recalibrating twelve survey meters — every single one had drifted past its tolerance band. The newer digital units were worse than the analog ones because their firmware masked the raw counts behind a smoothed moving average. The needle on a CD V-700 tells you when it is wrong; a software-smoothed readout just tells you what you wanted to hear.

The meters that read zero were the ones we should have worried about most.

The lab closed six months later. I took the CD V-700 home. It still drifts — about three percent per year — and every time I recalibrate it I think about the silent systems we run in production, the ones whose dashboards show green because nobody updated the thresholds when the baseline moved. The hot particles are the easy part. The silence is what gets you.