LLM on a Budget series
- Part I — The hardware plan
- Interlude — Gemma 4 on laptop hardware
- Part II — The build
- Part III — Software, the 4am saga, and benchmarks
- Companion — Model comparison deep dive
- Part IV — GPU upgrade, six models, 16GB VRAM
The swap
The original build used a secondhand RTX 2060 Super with 8GB of VRAM. Part III established the baseline: gemma4:e4b at ~55 tokens per second for generation, the larger models either impractically slow or partially running on CPU rather than GPU. The headline finding of the companion model comparison was that both gemma4 models correctly followed the brief's architectural constraints, while every other model tested failed the same test regardless of size or "coding" specialisation.
The RTX 2060 Super has since been replaced with an RTX 5060 Ti Twin X2 16GB — double the VRAM, a newer architecture, and a GPU that can actually hold the 30B+ class models in VRAM without spilling most of them into system RAM.
With Ollama reinstalled and the new driver confirmed working, the same benchmark suite ran across six models. Here's what changed.
Test setup
All tests run on the same machine described in Part II: Xeon E5-2640 v3, 128GB ECC RAM, Ubuntu, Ollama with a 32,768-token context window. The only change is the GPU: RTX 2060 Super (8GB) → RTX 5060 Ti (16GB, Driver 595.71.05, CUDA 13.2).
The benchmark prompt is the same six-section multi-tenant SaaS specification used in the model comparison: a demanding, multi-section technical specification request that tests both throughput (can it produce a large response quickly?) and correctness (does it follow stated architectural constraints?). All figures below are warm-run results — model already loaded, no cold-start overhead included.
A note on the Ollama CPU fallback issue: after installing the 5060 Ti and updating the driver, Ollama initially fell back to CPU inference despite the GPU being detected. The fix was reinstalling Ollama via the standard install script, which updated its bundled CUDA libraries to match the new driver. The first sign of the problem was a source=types.go:50 line in the logs (CPU path) instead of types.go:32 (GPU path). Worth knowing if you're upgrading to a newer-generation GPU.
The headline number: prompt evaluation
Generation speed gets most of the attention in GPU benchmarks because it's the number people feel — it's what determines how long you wait between sending a prompt and seeing the first token appear. But the number that changed most dramatically in this upgrade is prompt evaluation speed: how quickly the model processes your input before it starts generating a response.
On the RTX 2060 Super, prompt evaluation ran at roughly the same speed as CPU-only inference for most models — because most models were being partially evaluated on CPU due to VRAM constraints. On the RTX 5060 Ti, with models fitting fully in VRAM, prompt eval speed increased by 35–400x depending on the model.
This matters more than it sounds. In a long iterative session — asking a model to build something, correcting an error, asking it to revise, building further — the accumulated conversation context grows with every exchange. On the 2060, re-evaluating 7,000 tokens of context could take minutes. On the 5060 Ti, the same 7,000 tokens process in under 2 seconds. That changes the feel of a working session entirely.
Full benchmark results
| Model | 2060 Super Prompt eval |
5060 Ti Prompt eval |
2060 Super Generation |
5060 Ti Generation |
VRAM (5060 Ti) |
|---|---|---|---|---|---|
| gemma4:e4b | ~41 t/s | ~4,600 t/s (112x) | ~55 t/s | 70–73 t/s (1.3x) | ~5GB |
| gemma4:31b | ~2 t/s | 223–229 t/s (115x) | ~1.9 t/s | 3.08–3.60 t/s (1.9x) | ~13.5GB |
| qwen3-coder:30b | ~17 t/s | ~500 t/s (29x) | ~17 t/s | 33–34 t/s (2x) | ~14.7GB |
| deepseek-r1:14b | ~5 t/s | ~1,959 t/s (390x) | ~6.35 t/s | ~40 t/s (6.3x) | ~14.7GB |
| llama3.1:8b | ~16 t/s | ~3,632 t/s (227x) | ~16.6 t/s | ~69 t/s (4.2x) | ~9GB |
| qwen2.5-coder:32b | ~5 t/s | ~260 t/s (52x) | ~2.2 t/s | ~4 t/s (1.8x) | ~14.5GB |
The prompt eval column tells the story of how constrained the 2060 actually was. Every model on the 2060 was evaluating prompts at speeds consistent with CPU inference or heavy CPU/GPU splitting — even gemma4:e4b, which fits comfortably in 8GB, was only achieving ~41 t/s prompt eval (comparable to the pure CPU baseline). On the 5060 Ti, with models properly resident in VRAM, the same work happens 30–400x faster.
Generation speed improvements are more modest: 1.3–6x depending on the model. This is expected — generation is inherently sequential (each token depends on the previous one) and is ultimately bounded by memory bandwidth rather than raw compute. The 5060 Ti has wider memory bandwidth than the 2060 Super, which explains the improvement, but there's no architectural trick that makes sequential generation dramatically faster.
Model by model
gemma4:e4b — the daily driver
5GB VRAM · 70–73 t/s generation · ~4,600 t/s prompt evalComfortably in VRAM on both cards, so the generation speed improvement is real but modest: 55 → 70 t/s. The bigger gain is prompt eval — 41 → 4,600 t/s — which means long multi-turn sessions are dramatically faster. The model still hits the ceiling on very long context (32K tokens) and still occasionally drops requirements mid-session on iterative tasks, but for fast, single-pass work it remains the right choice. Importantly it's the only sub-8GB model that consistently followed the architectural constraints in the benchmark prompt.
gemma4:31b — the best ideas, now more practical
13.5GB VRAM · 3.08–3.60 t/s generation · 223–229 t/s prompt evalThis is the model whose results changed most meaningfully with the upgrade. On the 2060, with only ~2GB of its 14GB actually fitting in VRAM, prompt evaluation ran at ~2 t/s and the entire experience was unusable for practical work. On the 5060 Ti, with 13.5GB resident in VRAM, prompt eval jumped to 229 t/s — a 115x improvement — and total response time for the full six-section specification dropped from 34 minutes to under 5 minutes.
Generation at 3 t/s is still slow, and it won't win a speed contest with e4b. But for a task where you're going to set it running while you do something else, 5 minutes is a usable wait. gemma4:31b remains the only model tested that solved the actual hard architectural problem in the benchmark prompt — the per-tenant database routing pattern — and that quality is now accessible at a practical speed.
qwen3-coder:30b — fast and comprehensive, same constraint failure
14.7GB VRAM · 33–34 t/s generation · ~500 t/s prompt evalThe 2060's ~17 t/s for this model was almost certainly heavily CPU-bound — the 2060 result is now confirmed to have been a hardware constraint rather than a model ceiling. On the 5060 Ti at 14.7GB VRAM it doubles to 34 t/s, and it produced 13,933 output tokens for the benchmark prompt — by far the largest output of any model tested, at a speed that makes it genuinely usable. The Docker Compose constraint violation remains; the volume and speed of the output is impressive, but the same architectural mistake is still there.
deepseek-r1:14b — the biggest wall-clock improvement
14.7GB VRAM · ~40 t/s generation · ~1,959 t/s prompt evalThe single most dramatic improvement in the test suite. On the 2060, total response time was 6m 51s including 127 seconds of visible reasoning. On the 5060 Ti, total time dropped to ~12 seconds — a 34x improvement. The reasoning step still happens (the model thinks before answering), but the thinking itself now processes at GPU speed rather than CPU speed. Generation speed improved from 6.35 t/s to 40 t/s. The architectural constraint violation remained — the visible reasoning process didn't prevent the same PostgreSQL-in-Docker mistake, despite the thinking block explicitly covering Docker services.
llama3.1:8b — hardware ceiling, not model ceiling
~9GB VRAM · ~69 t/s generation · ~3,632 t/s prompt evalThe 2060's ~16 t/s was a hardware constraint, not the model's limit — it's now running at 69 t/s on the 5060 Ti. The 9GB VRAM figure is higher than you'd expect for an 8B model; Ollama is allocating a larger KV cache given the available headroom. The content quality issue is unchanged: fast generation, thin output, same Docker Compose architectural violation, and the same "What a behemoth of a request!" opening. This model's ceiling is the model, not the hardware.
qwen2.5-coder:32b — still VRAM-limited, but less so
14.5GB VRAM · ~4 t/s generation · ~260 t/s prompt evalAt Q4 quantisation, this model's weights exceed 16GB, so even the 5060 Ti can't keep it fully in VRAM. VRAM usage peaks at 14.8GB on load then settles to 14.5GB during inference, with some layers still spilling to system RAM. Generation speed at ~4 t/s is marginally better than the 2060's 2.2 t/s, but not meaningfully so for practical use. For this model to become truly fast, you'd need 20GB+ VRAM — a 3090 or better. On this hardware it remains the slowest option, and the same code correctness issues (the broken dependency injection, the Pydantic/ORM confusion) persist regardless of how the hardware is configured.
The modelfile trap: don't carry old tunings to new hardware
One finding worth flagging explicitly: qwen2.5-coder:32b was initially tested using a -cuda tagged model created specifically for the RTX 2060, with a forced num_gpu 16 layer offload setting and a reduced context window of 8,192 tokens. On the 5060 Ti, those settings artificially constrained the model — limiting it to 5.8GB VRAM active and effectively recreating the 2060's constraint on hardware that could handle significantly more.
Creating a fresh modelfile without the forced layer settings allowed Ollama to automatically allocate 14.5GB VRAM, which improved generation speed from 2.5 t/s to 4 t/s and doubled prompt eval throughput. The moral: custom modelfiles tuned for a specific GPU's limitations become a liability when the hardware changes. Check your Ollama model list for any tagged variants before benchmarking on new hardware.
What hardware can't fix
The benchmark prompt includes an explicit architectural constraint stated in plain language: "Everything runs in Docker except PostgreSQL, which runs in dedicated VMs — one per tenant for isolation." A correct response to section four therefore omits PostgreSQL from the Docker Compose file entirely. An incorrect response includes it.
On the RTX 2060 Super, the results were: gemma4:e4b correct, gemma4:31b correct, every other model tested incorrect. On the RTX 5060 Ti, across all six models, the result is identical. Hardware doesn't change model reasoning; faster inference of the same incorrect logic is still incorrect logic.
| Model | PostgreSQL constraint followed | Changed with new GPU? |
|---|---|---|
| gemma4:e4b | ✓ Yes | No — still correct |
| gemma4:31b | ✓ Yes | No — still correct |
| qwen3-coder:30b | ✗ No | No — still fails |
| deepseek-r1:14b | ✗ No | No — still fails |
| llama3.1:8b | ✗ No | No — still fails |
| qwen2.5-coder:32b | ✗ No | No — still fails |
Both gemma4 models — the general-purpose family — have been the only models to follow this constraint across every test. Both qwen2.5-coder models, qwen3-coder:30b, deepseek-r1:14b, and llama3.1:8b all include PostgreSQL as a Docker service regardless of how much GPU resource they have available. This appears to be a model-family characteristic rather than a resource or speed issue.
The practical picture: what actually changed
For the original goal of this build — a local LLM for day-to-day development assistance, without sending sensitive work to a cloud API — the upgrade materially changed the following:
- Long session performance. The 4,600 t/s prompt eval on gemma4:e4b means a 10,000-token accumulated context is processed in ~2 seconds rather than ~4 minutes. Sessions that would have slowed to a crawl on the 2060 stay responsive throughout.
- Access to the 31b model. gemma4:31b was effectively unusable on 8GB VRAM — 34-minute responses make iterative work impossible. At 3–4 t/s on 16GB it's still not fast, but it's no longer unusable. For a complex one-shot task where quality matters more than speed, it's now a practical option.
- deepseek-r1:14b becoming usable. 12 seconds vs 7 minutes is the difference between a tool you'll reach for and one you won't. The reasoning architecture's visible thinking step is now fast enough to be a feature rather than a tax.
What didn't change: the model quality findings. The models that reasoned correctly about the architecture did so because of how they were trained, not because of the hardware they ran on. gemma4 remains the family to reach for when correctness matters. Faster incorrect output is still incorrect output.
Is 16GB enough?
For everything up to 30B class models: yes, comfortably. gemma4:e4b (8GB), gemma4:31b (13.5GB), qwen3-coder:30b (14.7GB), and deepseek-r1:14b (14.7GB) all fit with workable performance. The 5060 Ti sits at 97% VRAM utilisation when qwen3-coder:30b is running, which leaves essentially no headroom for larger models.
qwen2.5-coder:32b at Q4 is the outlier — it exceeds 16GB and will always spill layers to system RAM on this card. For that model to run at full speed you'd need a 3090 (24GB) or similar. For the rest of the local LLM ecosystem at reasonable quantisation levels, 16GB is currently a good fit.