Code: github.com/shalabhsingh/local-bench-mac — all scripts, pinned versions and raw result JSONs referenced in this post.

Overview

I’ve recently been curious about the advances in local model hosting, especially given recent infrastructure improvements and the proliferation of optimization libraries. Many developers want to ditch paid APIs like Claude Code or Codex in favor of running a local LLM for personal projects. Along with this, I found that vLLM — which was optimised for cloud inference — is now offering a metal library to run models on Mac locally. This will be directly competing with Ollama, and I wanted to check how both of these compare.

However, when looking into resources for hosting models locally, I found that most guides focus on maxing out your CPU and GPU. But I don’t want to dedicate my entire laptop to running an LLM — I needed to use it alongside other applications. Ideally, the model should consume a practical amount of memory. For example, on a system with 24 GB of RAM, I wouldn’t want to allocate more than 16 GB on average to LLMs; anything beyond that would be better suited for a dedicated cloud instance like an ECS.

The end goal of this post is to build a benchmark utility to determine the best LLM you can run locally for your specific use case. We will experiment with different serving engines (vLLM-Metal and Ollama) and various model quantizations, finally evaluating them on the HumanEval benchmark to measure coding performance. I’ll be running these tests on a 24 GB M4 MacBook, and the findings should reliably apply to other Apple Silicon configurations as well.

Setup

Pin these. vllm-metal is young and moving fast — the v0.1 → v0.2 release notes claim 83× TTFT and 3.6× throughput from a single attention-backend rewrite. A benchmark without a pinned version isn’t a benchmark. One caveat on my own numbers: install.sh in the repo pins the vLLM core wheel at 0.27.1 but installs the plugin’s latest release, so pin both if you want byte-exact reproduction.

Installing vLLM-Metal

The project’s own installer is a one-liner, and it’s the path you should use — it resolves the core wheel, the plugin, and the prebuilt Metal kernels together:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate
vllm --version

Two requirements that will bite you if you skip them. Your Python must be native arm64 3.12 — a Rosetta/x86_64 interpreter is not supported, and the failure mode is confusing rather than explicit. And you need Xcode Command Line Tools (xcode-select --install), because vLLM core compiles from source via clang++. The Metal kernels themselves ship prebuilt, so you don’t need a Metal toolchain to run them — but if you install the plugin from a source checkout instead of a release wheel, you must build the native artifacts yourself or the first request that touches paged attention dies with Prebuilt native extension not found.

install.sh in the benchmark repo is a version-pinned variant of the same flow, for when you want a specific core release rather than whatever is current.

Ollama is the boring half:

brew install ollama
ollama serve
ollama pull qwen3:8b      # Q4_K_M, ~5 GB
ollama pull qwen3.5:9b    # Q4_K_M, ~6.6 GB

Ollama’s default tag is Q4_K_M — 4-bit GGUF via llama.cpp — which matches the MLX 4-bit configs on bit width even though the algorithm differs. Matched memory footprint, different algorithm: exactly the comparison I want.

Choosing a Model

Not Qwen3.6-27B, which was the original plan, driven mostly by an X feed full of people running it for coding. At 4-bit, 27B weights are ~16 GB. That leaves roughly 5 GB for KV cache, macOS overhead (~3–4 GB), and anything else — and at the 8K–32K context a coding agent actually uses, you run out. It works theoretically but not practically.

Qwen3-8B (dense) as the control. It’s the model I quantize four different ways, because it’s the one where the comparison is clean: vllm-metal’s support matrix lists dense Qwen3 as ✅ supported with ✅ automatic prefix caching, on standard GQA paged attention. At 4-bit it’s ~4.6 GB of weights, leaving ~13 GB for KV cache — so nothing about the quantization comparison is confounded by memory pressure or experimental code paths.

Qwen3.5-9B as the model I’d actually run. Nobody would pick an 8B for coding today. Qwen3.5-9B is the heaviest model I can host at 4-bit (~6 GB of weights) while still keeping an editor, a browser and a few containers open — which was the whole constraint I set at the top. On Qwen’s published benchmarks it sits in the Qwen3-14B class rather than the 8B class for coding, which is the premise of the choice; I measured its HumanEval score but not that generational claim, so take it as vendor-reported.

The catch is that Qwen3.5 is a hybrid architecture (SDPA + GDN linear attention), and the same support matrix lists it as ✅ for the model but 🔵 experimental for automatic prefix caching. That’s a real caveat: prefix caching is the feature a coding agent leans on hardest, since it resends a long system prompt and tool schema on every turn. So I kept Qwen3-8B as the control for anything cache-related and treat the 9B’s cache numbers as indicative rather than definitive.

The BF16 Memory Wall

I started with unquantized BF16 weights (~16 GB) to get an honest upper bound. It refused to start:

ValueError: Paged attention: not enough Metal memory for KV cache.
metal_limit=19.07GB, fraction=0.92, usable_metal=17.54GB,
model_memory=16.38GB, overhead=1.30GB, kv_budget=-0.13GB.

That message is the whole constraint of this project in six numbers. macOS and my open apps had taken ~5 GB of the 24 GB unified pool, leaving 19.07 GB visible to Metal; at the default 0.92 fraction that’s 17.54 GB usable; the weights alone are 16.38 GB, plus 1.30 GB of engine overhead — so the KV budget came out negative.

There are only three levers, and pulling all three as hard as they go barely clears the bar: sudo purge with every app closed (to keep metal_limit at 19.07 GB), VLLM_METAL_MEMORY_FRACTION=0.95 (+0.58 GB usable), and --max-model-len 4096 instead of 8192, which halves engine overhead from 1.30 GB to 0.66 GB. Net result: kv_budget=1.08GB → 7,296 tokens of KV cache. A system prompt plus one file read gets you there.

So BF16 is a data point in this post, not a config anyone should run — the weights themselves are the problem, and that’s arithmetic, not a vllm-metal limitation. The fix is quantization, and on this hardware it’s a 15-second job:

mlx_lm.convert --hf-path ~/models/Qwen--Qwen3-8B \
  --mlx-path ~/models/Qwen3-8B-4bit-mlx -q --q-bits 4

Weights 16.38 → 4.61 GB, KV budget 1.08 → 12.81 GB. That’s 86,880 cached tokens instead of 7,296 — an 11.9× increase in usable context from one flag.

What the Quantization Options Actually Are

Three schemes matter on this hardware, and the difference between them is what information they use to decide where the bits go.

MLX 8-bit and 4-bit (mlx_lm.convert --q-bits 8|4) are round-to-nearest group quantization. The weight matrix is split into small contiguous groups; for each group mlx-lm stores a scale and a bias in 16-bit, and each weight as an 8- or 4-bit integer offset from those. That’s the entire algorithm. No calibration data, no forward passes, no error feedback — purely arithmetic, which is why it finishes in 15 seconds. Activations stay BF16 in both cases, so these are really W8A16 and W4A16. The 8-bit version keeps enough precision that rounding error is nearly irrelevant; the 4-bit version is where you’d expect to start paying for it.

AWQ W4A16 (activation-aware weight quantization) is the same 4-bit destination reached with more information. It pushes calibration data through the model, measures which input channels carry large activations, and applies per-channel scaling so the weights that get multiplied by big activations effectively keep more precision. It also refuses to quantize the most sensitive tensors at all, leaving them in bfloat16. Here it runs with q_group_size=128, zero_point=True, the GEMM kernel — and 60–90 minutes of CPU time instead of 15 seconds.

The one-line version: MLX asks “what’s the nearest 4-bit value to this weight?” AWQ asks “which weights actually move the output, and how do I spend my bits on those?” On CUDA, AWQ wins that argument comfortably. The interesting part of this benchmark is that on Metal it doesn’t.

Ollama’s Q4_K_M is the third algorithm and the cross-engine comparison point: llama.cpp’s k-quant grouping, which minimizes error across groups with mixed per-group precision (the _M denotes the medium mix, keeping some tensors at higher bit width). Also 4-bit, also no calibration data, but a meaningfully more careful rounding scheme than MLX’s.

What doesn’t work on Metal, and why

This is the section most “I ran a model locally” posts skip, and it’s the one that saved me the most time. Standard CUDA quantization advice largely does not transfer.

GPTQ isn’t supported in vllm-metal at all — its inference path depends on custom CUDA dequantization kernels with no Metal equivalent in either vllm-metal or mlx-lm. GPTQ checkpoints fail to load.

W4A8 and W8A8 are CUDA-specific in practice. The “A8” half means quantizing activations to int8 at runtime, which only pays off if the hardware has integer dot-product instructions (dp4a, IMMA) to exploit. On Metal/MLX the activations stay BF16 regardless of what the checkpoint config claims — so you can load one of these and get W4A16 performance with a W4A8 label on it. That’s worse than not running the comparison.

llm-compressor, vLLM’s official quantization toolkit, targets CUDA vLLM; most of its output formats (GPTQ, W4A8, W8A8, FP8) don’t map onto the MLX backend. The exception is AWQ W4A16, which loads through mlx-lm’s weight repack path.

What does work:

So the controlled experiment is: same BF16 source weights, same serving engine, four quantization algorithms — and Ollama’s Q4_K_M as the cross-engine leg at matched bit width. quantize.py produces all the self-quantized checkpoints:

python quantize.py --method mlx4   # ~15s  → ~/models/Qwen3-8B-4bit-mlx
python quantize.py --method mlx8   # ~15s  → ~/models/Qwen3-8B-8bit-mlx
python quantize.py --method awq4   # 60–90 min, CPU

There’s no AWQ 8-bit to compare against — AutoAWQ’s GEMM kernel is 4-bit only. That’s a defensible design choice: AWQ exists to recover quality lost at 4-bit, and at 8-bit there isn’t much left to recover.

ModuleNotFoundError: No module named 'torch'

autoawq 0.2.9 depends on triton
triton: no matching distributions available for your environment
ERROR: ResolutionImpossible

FileNotFoundError: Couldn't find 'mit-han-lab/pile-val-backup' on the Hub

model.quantize(tokenizer, quant_config={
    "zero_point": True, "q_group_size": 128,
    "w_bit": 4, "version": "GEMM",
}, calib_data=CALIB_DATA)

AWQ 4-bit is 34% heavier than MLX 4-bit

The first surprise arrived before I ran a single request. AWQ 4-bit loads at 6.18 GB, not 4.61 GB, at the same nominal bit width. The startup log explains it in one line:

AWQ load: aligned 147 non-quantized floating params to mlx.core.bfloat16

AWQ deliberately left 147 parameter tensors in bfloat16 — typically first and last layers plus high-sensitivity attention projections. That’s the activation-aware part working as designed. The cost is memory, and it propagates straight into context: KV budget drops 12.81 → 10.48 GB, max cached tokens 86,880 → 71,088.

It also costs stability. Every MLX config ran fine at VLLM_METAL_MEMORY_FRACTION=0.95; AWQ crashed mid-inference with a Metal OOM inside mx.eval(logits_2d), because those bfloat16 tensors plus activation spikes push peak allocation past the wired_limit=17.8GB ceiling. It needs 0.90. Worth knowing when reading the results: AWQ is being compared at a lower memory fraction because it cannot hold the same one. Worth knowing too that vllm-metal’s AWQ repack path is documented as verified for Qwen2.5, Llama 3 and Mistral — Qwen3 isn’t on that list, so some of what follows may be specific to this pairing.

What the Benchmark Measures

Three numbers, chosen because they’re the three that decide whether a local coding agent feels usable. bench.py and compare.py are standard-library Python hitting the OpenAI-compatible endpoint, so there’s nothing to install to reproduce any of it.

TTFT (time to first token) — stream a chat completion, timestamp the first chunk carrying non-empty content. This determines whether inline or autocomplete-style interaction feels responsive at all.

Throughput (tok/s) — streamed content chunks over wall-clock time to the last token, concurrency 1, max_tokens=256. Concurrency 1–2 is the honest traffic pattern for a local agent; high-batch server throughput is a different benchmark for a different machine.

Prefix-cache speedup — the one to be careful about, because it’s a proxy rather than a measurement. Coding agents resend a long system prompt and tool schema on every call, so KV reuse on that prefix is where latency wins should come from. Neither engine exposes a hit rate over the API, so I measure it indirectly: send a ~500-token coding-agent system prompt, send it again, report first_TTFT / second_TTFT. Above 1.0 means the second call was faster.

SYSTEM_PROMPT = """You are an expert software engineer assistant...
Tool: read_file / write_file / bash / search_code
  <full parameter schemas>
"""  # ~500 tokens, resent on every call — this is the prefix under test

Two caveats on that column. It’s a TTFT ratio, so anything that moves TTFT moves it. And the two scripts differ: bench.py (all Qwen3-8B rows) pairs the same system prompt with a different user question each time, so after the first pair both calls are partial cache hits and the ratio partly reflects prompt-length variation. compare.py (Qwen3.5-9B rows) sends a genuinely identical prompt twice, which is the cleaner design. Compare cache numbers within a model, not across those two groups.

# single vllm config — memory figures come from the serve log
python bench.py --config "4bit-MLX" --runs 5 \
  --model-memory-gb 4.61 --kv-budget-gb 12.81 --max-tokens-cached 86880

# cross-engine at matched quant
python compare.py --engine vllm   --model /path/to/model --config "qwen3.5-9b-vllm"
python compare.py --engine ollama --model qwen3.5:9b     --config "qwen3.5-9b-ollama"

Memory is passed in from the serve log rather than sampled, deliberately: model_memory, kv_budget and max_tokens_cached are what vLLM actually decided to allocate, which is far more meaningful than process RSS on a unified-memory system where “GPU memory” versus “process memory” is mostly fictional. Every results JSON carries them alongside the timings and the engine version.

Results

Memory, per config

Speed and quality

All runs: M4 24 GB, prefix caching enabled, concurrency 1. TTFT and throughput are medians over five timed runs; where the first request against a freshly started vLLM server carried multi-second engine warm-up, it’s dropped as an outlier. Per-run timings for every config are in the results JSONs if you want to check my medians.

Quantization buys throughput, not just memory. BF16 → 8-bit → 4-bit runs 10.8 → 17.8 → 25.0 tok/s, a clean 2.3× top to bottom. That isn’t an implementation accident: single-stream decoding on Apple Silicon is memory-bandwidth-bound, and every token requires streaming the full weight set through the memory system. Quarter the weights, roughly double the tokens. Quantizing is the highest-leverage thing you can do on this hardware, and it improves memory and speed simultaneously — unusual enough to be worth saying out loud.

AWQ is the wrong tool here, and it isn’t close. 7.4 tok/s — 3.4× slower than MLX 4-bit at the same bit width, and slower than BF16. Two mechanisms. Those 147 bfloat16 tensors don’t participate in the fast int4 matmul path, so the Metal kernel eats a mixed-precision dispatch every forward pass. And AWQ’s group layout (128-element groups, interleaved zero-points) was designed for CUDA integer dot-product; mlx-lm’s repack makes it load, not makes it native to MLX’s contiguous-block memory model. This is a format/hardware mismatch, not a quality-for-speed trade — which means the usual “AWQ is the better algorithm” advice simply doesn’t apply on Metal, however true it is on an A100.

Throughput: vLLM wins, decisively. Same Qwen3.5-9B, same 4-bit width: 29.6 vs 16.5 tok/s, nearly 1.8×. At Qwen3-8B the gap narrows to 25.0 vs 21.3, so it isn’t constant, but the direction held in every pairing. The MLX backend is doing genuinely hardware-tuned matmuls; llama.cpp’s Metal backend isn’t as far along.

TTFT: Ollama wins, decisively. 95 ms vs 348 ms on Qwen3-8B; 158 ms vs 554 ms on Qwen3.5-9B. That’s request-path overhead rather than compute — vLLM’s TTFT is dominated by KV allocation and chunked-prefill scheduling (max_num_batched_tokens=2048), machinery built to keep latency bounded under batching that you pay for and don’t use at concurrency 1.

The prefix cache is real but modest at concurrency 1. BF16’s 1.43× is the cleanest signal in the set (per-pair ratios 1.37–1.46, tight enough to trust), and Qwen3.5-9B on vLLM shows 1.53× under the stricter identical-prompt test. Ollama is inconsistent: 1.31× on the 9B but 0.67× on Qwen3-8B, where resending the prompt actively hurt latency. The sub-1.0 numbers in the MLX 4-bit and AWQ rows are mostly the methodology artifact flagged above rather than a broken cache — which is precisely why I’d rather say so than sell a clean-looking finding. Honest summary: at single-user concurrency, prefix caching is worth ~1.3–1.5× on TTFT when it works, and it is not the dominant term.

Quality: HumanEval via evalplus

Throughput is half the question. The premise of 4-bit is that you’re trading accuracy for memory, and if you don’t measure the accuracy side you’re just admiring a fast wrong answer.

Don’t write your own harness for this. Mine scored Qwen3.5-9B at 20%, and four of five failures were IndentationError — the model was emitting a correct function body that my code spliced onto the prompt’s signature at the wrong indent level. I was measuring my own string handling. The tell is exactly that: syntax errors rather than assertion failures. eval_quality.py is still in the repo as a reference, but use evalplus instead.

evalplus is the reference implementation for HumanEval and its extended HumanEval+ suite — roughly 80× more test cases, which catches solutions that pass the original tests by coincidence. It talks to anything with an OpenAI-compatible endpoint, which is both engines here.

python -m evalplus.evaluate --model "/path/to/model" --dataset humaneval \
  --backend openai --base-url http://localhost:8000/v1 --greedy

mkdir -p ~/Library/Caches/evalplus
gunzip -c HumanEvalPlus.jsonl.gz > ~/Library/Caches/evalplus/HumanEvalPlus-v0.1.10.jsonl

Qwen3.5-9B, all 164 problems, greedy decoding:

This is the finding I didn’t expect, and it complicates everything above: 4-bit is not 4-bit. Same model, same nominal bit width, 4.3 points apart on base tests and 4.9 on HumanEval+ — seven problems that one quantization solves and the other doesn’t. The mechanism is the algorithm. Q4_K_M’s k-quant grouping minimizes error across groups with mixed per-group precision; MLX rounds each group independently with no error feedback. That 15-second conversion isn’t free after all — it costs about 4 points of HumanEval.

Worth noting what isn’t alarming: the HumanEval → HumanEval+ drop is only 3.7–4.3 pp for both engines, against a more typical 10–15 pp. Models that pass the base tests by pattern-matching fall much harder on the extended suite. These solutions are mostly actually correct.

Conclusion: Which Engine Should You Use?

The raw numbers split cleanly — Ollama owns latency, vLLM-Metal owns throughput — so the answer depends on what a local LLM workload actually looks like. In my experience it’s one to five concurrent sessions doing medium-complexity work: a few agent turns, a few hundred to a few thousand output tokens each, occasionally two things at once. In that regime throughput is the metric that matters. A 500-token completion takes ~17 s on vLLM-Metal against ~31 s on Ollama; the 400 ms of extra TTFT disappears into that. Ollama’s latency advantage only really pays off for 20–30 token inline suggestions, and that’s not where most of the time goes.

So on the numbers, vLLM-Metal is the better engine for this workload. But the numbers aren’t the whole cost. Getting vLLM-Metal to a working config took real input from me: reasoning about metal_limit versus memory fraction versus context length to find a KV budget that fits, discovering that AWQ needs 0.90 where MLX tolerates 0.95, patching a missing chat_template out of a sibling tokenizer config, working around a CDN that corrupts shards. None of that is hard, but all of it assumes you know what a KV cache is and are willing to read a startup log. Ollama is ollama pull and it works.

There’s a second, narrower cost: model coverage. Ollama serves essentially anything with a GGUF build. vllm-metal’s support matrix is a specific list, and features vary within it — as we saw, Qwen3.5 is supported but its prefix caching is still experimental. That list is growing fast, but on any given day your preferred model might not be on it, or might be on it with the one feature you care about marked 🔵.

The honest recommendation, then, is a trade-off rather than a winner. If you’re technically comfortable and your model is on the supported list, use vLLM-Metal — you get ~1.8× the throughput, a prefix cache designed for exactly the system-prompt-resending pattern coding agents have, and headroom to grow into. If you want a local model to just work, use Ollama — you give up throughput and get back setup time, broader model coverage, much better idle latency, and, as it happens, better 4-bit weights.

For my own daily driver: Qwen3.5-9B at 4-bit on vLLM-Metal, --max-model-len 32768, ~6 GB of weights, editor and browser still open. Skip AWQ on Metal entirely, skip BF16 at this memory size, and skip MLX 8-bit unless you have a specific reason — it costs 4 GB of weights and 7 tok/s against 4-bit, and I have no quality measurement showing it buys anything back.

Going Further

This post stops at the serving layer: one model, one engine, an OpenAI-compatible endpoint. Getting an actual Claude-Code-like experience on top — tool calling, file edits, a multi-turn agent loop — is a layer above, and harnesses like DeepSeek’s agent harness (and similar open tool-calling scaffolds) will drive a local endpoint that way. That’s genuinely worth building on these numbers, but it’s beyond the scope of this work; what I set out to establish here is which endpoint you’d want underneath it.

Reproducing This — On Your Own Model and Mac

The benchmark isn’t specific to Qwen or to a 24 GB M4. Here’s how to point it at your own model on your own hardware, in the order I’d do it.

1. Check your model is supported, and how. Look it up in docs/supported_models.md. You want ✅ in both the support and the Automatic Prefix Cache column — a 🔵 in the cache column means your cache numbers are indicative, not conclusive, which is exactly the caveat I carry for Qwen3.5-9B above.

2. Do the memory arithmetic before you download anything. Two estimates decide whether a config is viable:

weights    ≈ params × bits/8          # 8B at 4-bit ≈ 4.5 GB; at 8-bit ≈ 8.7 GB
kv_budget  ≈ metal_limit × fraction − weights − overhead

metal_limit is roughly your total RAM minus what macOS and your apps are holding (~5 GB for me with a normal working set). overhead ran 0.66–1.45 GB across my configs, scaling with --max-model-len. Then convert the budget into context, which is exact and worth computing yourself:

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × 2 bytes

For Qwen3-8B that’s 2 × 36 × 8 × 128 × 2 = 147,456 bytes ≈ 144 KiB per token, so a 12.81 GB budget is 86,880 tokens — matching the serve log exactly. Note this is independent of weight quantization: the KV cache stays 16-bit however hard you squeeze the weights. If your target context needs more than the budget you computed, quantize harder or shrink --max-model-len before you spend an hour downloading.

3. Get the weights. download_model.py pulls from ModelScope (useful when huggingface.co is blocked), with a browser User-Agent, parallel shards, resume, and retry-on-disconnect — that CDN drops connections mid-transfer routinely.

RuntimeError: [load_safetensors] Tensor '...embed_tokens.biases'
invalid data offsets (5016729984, 5048514944) exceeding the size of the file.

"As of transformers v4.44, default chat template is no longer allowed"

dst['chat_template'] = src['chat_template']  # e.g. from Qwen--Qwen3-8B/tokenizer_config.json

4. Quantize from one source checkpoint. If you’re comparing algorithms, they all have to start from the same weights or you’re not measuring the algorithm. python quantize.py --method mlx4|mlx8|awq4.

5. Serve, and read the startup log. The Paged attention memory breakdown line tells you whether your arithmetic from step 2 was right, and gives you the three numbers to pass into the benchmark:

VLLM_METAL_MEMORY_FRACTION=0.95 vllm serve ~/models/<your-model> \
  --max-model-len 32768 --enable-prefix-caching

6. Benchmark, then measure quality. Edit CODING_QUESTIONS and SYSTEM_PROMPT in bench.py to look like your actual workload — the prefix-cache result in particular is only meaningful if the system prompt is the length yours really is.

python bench.py --config "<label>" --runs 5 \
  --model-memory-gb <from log> --kv-budget-gb <from log> --max-tokens-cached <from log>
python compare.py --engine ollama --model <tag> --config "<label>-ollama"
python -m evalplus.evaluate --model "<tag>" --dataset humaneval \
  --backend openai --base-url http://localhost:11434/v1 --greedy

Every run writes a JSON to results/ carrying the engine version, model path, memory breakdown and every individual timing rather than just the median — so when a future vllm-metal release moves these numbers, it’ll be obvious which part moved. Given how fast this stack is turning over, I’d expect that within a couple of releases.

Full setup instructions, script reference and raw data: github.com/shalabhsingh/local-bench-mac.

If you’re running models locally, working on inference optimization, or you get different numbers on different Apple Silicon — I’d like to hear about it. Connect with me on LinkedIn.