The silicon beneath the model: why tensor throughput outranks parameter count in 2026.

We spent six weeks profiling four production training stacks on the new Halo H300 accelerators. The headline metric was never FLOPs — it was the memory road in front of them.

Three winters ago, the entire industry was still arguing about parameter count. Bigger models, more weights, longer context. The Halo H300 launch quietly reset that conversation: when you can land 1.8 TB of HBM4 next to a 4 PFLOP fp8 tensor core, the choke point stops being how many parameters fit and starts being how fast you can shovel activations between two stacks of silicon sitting four millimeters apart.

Memory bandwidth as the new headline number

In every benchmark we ran — dense decoder, sparse mixture-of-experts, retrieval-augmented inference — the tensor cores idled between 18% and 41% of wall clock. The cause was almost always the same: the model was waiting for weights to arrive. On the Halo H300 we measured an effective 7.8 TB/s bandwidth per accelerator and a 92% utilization ceiling on the dense kernels we hand-tuned for it. The cards that "should" have been slower on paper, we found, were actually faster on the workloads our customers care about.

The implication for procurement teams is concrete: a fleet of 512 lower-FLOP accelerators with tightly-coupled HBM beats a fleet of 384 paper-monster cards once batch sizes exceed 64K tokens. We rewrote one customer's training recipe around that constraint and saw a 31% reduction in time-to-converge on a 70B parameter Mixture-of-Experts model — without changing the model architecture or the optimizer at all.