The pitch for Apple Silicon and AI is unified memory: the CPU and GPU share one pool, so a model just needs RAM — not a discrete GPU with its own VRAM ceiling. A 128GB M5 Max should, in theory, run models that need a small rack of datacenter GPUs. This episode tests that theory to destruction: a ladder of six models from 3B to 123B parameters through one benchmark harness, plus a memory probe that allocates until the hardware refuses.
The scaling curve
Decode — generating tokens one at a time — is memory-bandwidth-bound. More weights to stream per token means proportionally slower generation: 38× the parameters, ~36× slower decode, almost exactly as theory predicts. That's the boring part.
Decode throughput
tokens / second · higher is better · all 4-bit quantized
The real cost of going big isn't throughput — it's time-to-first-token. TTFT explodes from 89ms at 3B to 4.6 seconds at 123B, a 52× jump. Decode at 123B is a usable 6 tok/s; the 4.6 seconds you wait before it starts is what makes it batch-only. The practical cliff for interactive use is ~32B:
Time to first token
lower is better · the wait before generation starts · 70B/123B are sustained (hot-chassis) figures
The breaking point that wasn't
Memory at 4-bit is dead linear — about 0.56GB per billion parameters — which makes the ceiling computable. And the biggest model on the ladder, Mistral-Large 123B, used 69GB on a 128GB machine. Half the tank. Nothing OOM'd. The chart is scaled to the 128GB mark to show how much headroom was left:
Peak unified memory
GB resident during generation · bar scale = 0 → 128GB
So I went looking for the actual wall with a probe that allocates GPU memory in 4GB chunks until Metal refuses. It never did:
Apple's advertised 115GB working set is a soft recommendation, not a wall. MLX handed out memory well past it — the probe stopped at its own 125GB safety cap, not a hardware limit. The true ceiling is essentially physical RAM minus OS overhead. A PC with a 24GB GPU can't load a 70B model at all; this laptop ran 123B with 60GB to spare.
The 14-inch thermal tax
This was all run on a 14-inch MacBook Pro, which caps around 45W and throttles under sustained GPU load where the 16-inch holds steady. So the 70B got measured twice: once cold (idle machine, single run) and once sustained (its turn deep in the overnight back-to-back marathon, hot chassis). Same model, same harness, same prompts.
The finding is sharper than "it throttles": decode is untouched, prefill takes the entire hit. Decode is memory-bandwidth-bound, and bandwidth holds even when clocks drop. Prefill is compute-bound, and compute clocks fall hard under heat — which is what balloons time-to-first-token. On a thermally-limited laptop, the generation rate is honest; the responsiveness is what the chassis steals. (The 1.8s TTFT in the ladder table below is the sustained number; cold, the same model answers in 642ms.)
Buying a Mac for local AI?
- ≤32B or bursty / interactive use: the 14-inch and 16-inch are effectively identical — the work finishes before heat builds.
- 70B+ under sustained load: the 16-inch is ~15–25% faster sustained and doesn't degrade mid-session; the 14-inch drops once the chassis saturates.
- Always-on inference: neither laptop is the right tool — a Mac Studio beats any MacBook on sustained thermals.
The full numbers
Three prompts per model (short factual, medium reasoning, long generation), averaged. A warm-up pass is discarded to absorb Metal shader compilation.
| Model | Params | Quant | Decode | Prefill | TTFT | Peak RAM |
|---|---|---|---|---|---|---|
| Llama 3.2 3B Instruct | 3.2B | 4-bit | 226.7 tok/s | 1,625 tok/s | 89 ms | 2.06 GB |
| Qwen2.5 7B Instruct | 7.6B | 4-bit | 117.2 tok/s | 718 tok/s | 115 ms | 4.46 GB |
| Qwen2.5 14B Instruct | 14B | 4-bit | 59.7 tok/s | 450 tok/s | 151 ms | 8.48 GB |
| Qwen2.5 32B Instruct | 32B | 4-bit | 27.3 tok/s | 191 tok/s | 326 ms | 18.61 GB |
| Llama 3.3 70B Instruct | 70B | 4-bit | 10.9 tok/s | 38 tok/s | 1,835 ms | 39.91 GB |
| Mistral Large 2407 | 123B | 4-bit | 6.3 tok/s | 11 tok/s | 4,634 ms | 69.18 GB |
Decode = generation rate · Prefill = prompt-processing rate · TTFT = time to first token · Peak RAM from MLX's own high-water mark · 70B/123B TTFT are sustained (hot-chassis) figures.
How it was measured
The harness loads each model through mlx-lm, streams the generation, and brackets the timing with a stopwatch placed at the first token (for TTFT) and across the decode loop (for throughput). Peak memory comes straight from MLX's allocator, not an external sampler, so it reflects exactly what the runtime held. Every model sees an identical prompt set — a fair comparison requires identical inputs. The full code, the per-model result JSON, the ceiling probe, and this dashboard's data are all in the mlx-lab repo.
One model didn't make it: the 8-bit 72B, whose download failed three different ways (a hung resume, read timeouts, a dropped connection mid-shard) before being cut loose — the 4-bit ladder already tells the whole story. Lessons banked in the repo: HF_HUB_DISABLE_XET=1 for reliable resumes, longer hub timeouts for big shards, and never restart a stable download.
This is Episode 2 of a build-in-public series on running and evaluating local models on Apple Silicon. Next: speed is table stakes — Episode 3 asks whether a local model can pick the right tool, with the right arguments, for a real operations alert. That's the number that decides if local models can drive an autonomous ops loop.