Anton, chapter 10: Making Gemma fly

I left a line open in the spring: speculative decoding, maybe a 1.5x on the local model, compounding with the quantization work from chapter 8. That work got Gemma 4 from 23.4 tokens a second in BF16 to 43.5 in NVFP4, a clean 1.86x, and I told myself that was the ceiling on this hardware. The question I've actually been chewing on is sharper: is it the ceiling, or just the ceiling of the one model and the one format I happened to be running? I decide to find out properly, with numbers, and I give myself permission to change the model to get there.

Quantization is not the lever

The first thing the numbers kill is the idea that a better quantization saves me. The Spark is memory-bandwidth-bound (273 GB/s), and any 4-bit format moves the same bytes per token, so bandwidth, not precision, is what sets tokens per second. Worse, the box has no native 4-bit math: the NVFP4 build runs through a software-emulated MARLIN kernel, so the format buys its memory win (16.5 GB of weights instead of 52) and almost none of its speed win. I'm paying for a patched model file and an emulated kernel and collecting half the payoff. Quantization was never going to be the lever.

A model bake-off

So I stop tuning one model and run a real bake-off, end to end through the worker on the weather-query chain, four contenders measured on total time and on the slow family delegate:

Qwen 3.6-35B-A3B with MTP: 196s end-to-end, 72s family, validator retried
Gemma 4 NVFP4 (community quant): 199s, 117s
Gemma 4 NVFP4 (NVIDIA quant): 178s, 117s
Nemotron-3-Nano-Omni-30B-A3B: 157s, 58s, first-try pass

Nemotron wins on every axis: 21% faster end to end than the next best, a family delegate twice as fast (its hybrid Mamba2-Transformer stack pulls ahead on long context), native multimodal so I can retire the separate vision model, agentic-trained, and one small real thing, it doesn't moralize about downloading my own media. In mid-May it becomes the local default. The lesson is the one that keeps recurring: stop polishing the thing you have and measure four things you don't.

Making Gemma fly

Nemotron's win is end-to-end quality and speed, not raw single-stream throughput, and that leaves the lever I marked open in the spring still unpulled. Multi-token prediction is the only trick that beats a bandwidth wall: a small drafter guesses the next few tokens, the big model verifies them in one pass, so you pay for one weight read and get several tokens back. Gemma ships a companion drafter (gemma4_assistant), and I wire it up at gamma=4 on an FP8 build of Gemma 4, FP8 rather than NVFP4 precisely because FP8 has native compute on this silicon where 4-bit does not. The result is 57 tokens a second sustained and 91 on short agent turns, up from 48 on the NVFP4 build, and tool calls and vision both survive speculation, which is the part that usually breaks. That is enough to put Gemma back in the primary slot and slide Nemotron down to the fallbacks.

Two numbers, not one

The 57 and the 91 are the honest version of the single figure everyone quotes as "tokens per second." Burst is what a short, predictable turn hits when the drafter guesses right almost every time; sustained is what a long generation averages once the text gets less guessable. Speculative decoding's speedup is entirely a function of how predictable the next tokens are, so a single headline number means nothing without saying which regime you measured. I write down both and stop pretending there's one.

Where it lands

So this is the chain at the end of it: local Gemma 4 FP8 with speculative decoding on the primary Spark at 57 to 91 tokens a second, a second Spark running NVFP4 Gemma (~52) as the first fallback, then the same Nemotron hosted on DeepInfra ($0.20 and $0.80 per million in and out) as the cloud fallback so an outage degrades to a known extractor, then a frontier model last. The ceiling I'd accepted in the spring was the ceiling of one model and one format. Changing both, and measuring honestly enough to know which number I was moving, is the whole of how it moved.