All blogs

My RTX 3050 had four gigs and my llama had eyes

I wanted faster text from Ollama. I gave three vision models a haircut, timed every change, and found out why my first theory about swap was wrong.

A freshly shorn llama on roller skates races across a circuit board, leaving its fleece and a pillow behind.

Five tokens a second is a lot of time to think

I asked Bonsai to explain how a computer executes a program. Somewhere around the first paragraph I started wondering if I should bring a pillow. At five tokens a second, even the 128-token sample I used for testing took about 25 seconds just to print. My laptop was giving me plenty of time to come up with my own answer.

My first suspect was swap. I’d seen RAM usage around 60% while swap looked full. Surely Ollama could use that spare RAM? I checked which processes owned the swapped pages. Most of the roughly 3.4 GiB belonged to other apps. Then I checked the model runner: only 26 of Bonsai’s 65 layers had landed on the GPU.

I was running Ollama 0.32.0 on an i7-12700H with about 15 GiB of usable RAM and an RTX 3050 Laptop GPU with 4 GiB of VRAM. Free system RAM could hold weights, but my CPU still had to do the work for layers placed there. And all three models I cared about could process images. I had sent them exactly zero pictures.

Bonsai taught me to change one thing at a time

I started with the original Bonsai 27B at a 4,096-token context. With the default six CPU threads, the two baseline runs gave me 5.21 and 5.08 tokens/s. I tried 48 layers on the GPU and got 8.34 and 8.42. At 56 layers I was already around ten.

Then I held the layer count at 56 and tried ten threads. The saved settings gave me 12.05 and 11.87 tokens/s with vision still attached. That run changed how I think about this experiment. If I’d timed only the first and last configurations, I’d have given the llama haircut credit for almost the entire speedup.

For my text copy, I used the same Q1_0 language weights and template and left out the separate vision projector. With a smaller processing batch of 128, I could fit 58 layers. I tried 60. CUDA ran out of memory. Back at 58, the final confirmation runs landed at 11.87 and 12.03 tokens/s. My 128-token wait had dropped to about eleven seconds.

Two ways to give a model a haircut

I figured the vision part could wait until I sent a picture. Ollama 0.32.0 makes that placement decision when it starts the model. When GPU placement is enabled for the vision projector, the runner budgets VRAM for its weights and buffers before fitting the text layers, even if every prompt I send is plain text. On my 4 GiB card, that leaves less room for the layers generating the answer I’m actually waiting for. That’s why I wanted a text-only copy. Ollama’s projector placement code.

Ornith 1.5 9B was the easy one. Its manifest listed a language-model blob and a separate 922 MB vision projector. I made a text-only tag that referenced the same Q4_K_M decoder and omitted the projector. Ollama’s content-addressed storage let the two tags share the decoder bytes. In my short tests, GPU placement went from 10 of 33 layers to 19 of 33.

Qwen 3.5 9B made me open the file. Its main GGUF contained both the language tensors and the vision encoder and merger. There was no separate projector entry to drop. I inventoried this particular file and checked the expected counts before writing a new GGUF. Here’s the guard from my script:

vision=[t for t in r.tensors if t.name.startswith('v.')]
keep=[t for t in r.tensors if not t.name.startswith('v.')]
assert len(vision)==441 and len(keep)==442

I removed the vision metadata, corrected the parameter count and copied the retained tensors in their original quantization. Then I reopened the output and compared all 442 tensors individually: names, shapes, types, byte lengths and SHA-256 hashes. Every retained tensor matched, including the draft-prediction tensors.

The blob went from 6.594 GB to 5.682 GB. That’s 912.8 MB of vision tensor data I could leave behind for text chat. I kept the tokenizer and the normal sampling settings, checked that the text copy could answer a chat request, and archived the vision-capable originals for when I actually want to send a picture.

What the stopwatch said

I asked each model the same text question about program execution, capped the output at 128 tokens, and set temperature to zero and seed to 42 for the benchmark requests. I calculated generation speed as eval_count × 10⁹ / eval_duration because Ollama returned that duration in nanoseconds. Loading and prompt processing have their own timings.

Short-prompt generation on my RTX 3050, in tokens per second
ModelContextOriginalTuned
Bonsai 27B4K5.1511.95
Ornith 1.5 9B4K8.7913.07
Ornith 1.5 9B8K9.2614.26
Qwen 3.5 9B4K9.5613.79
Qwen 3.5 9B8K9.9814.01

These are means from two runs per cell, except tuned Ornith at 8K, where I had three. Bonsai’s 4K row uses the fixed 58-layer profile. Ornith and Qwen used automatic placement, ten CPU threads and batch 256 for the tuned results. At 8K, Ornith gained 54% and Qwen gained 40% on this laptop.

I had to throw out some early Qwen runs. A permissions issue on a shared license blob kept the intended settings from being saved. The requests still produced perfectly plausible numbers. After fixing it, I checked the allocated context after generation and kept the failed-settings trials out of the averages. That check was worth more than another decimal place.

The configuration I actually kept

Fixed placement made a good 4K benchmark. I also wanted room for a conversation. A larger context needs a larger cache, which competes with the weights for VRAM. I settled on an 8K default, batch 256, ten CPU threads and automatic GPU placement, with a 256 MiB fitting margin on the server.

With that margin, Bonsai fit 49 of 65 layers at 8K and generated 9.97 and 10.06 tokens/s. Raising the batch to 512 left room for only 47 layers and came in slower. At 16K, automatic fitting dropped to 42 layers and I measured 8.70 tokens/s. Placement was chosen when the model loaded for the requested context. The margin gave the runner some headroom rather than enforcing a hard VRAM cap.

An 8K window with a tiny question is one thing. I also tried filling more of it:

  • Bonsai: 5,522 input tokens, 229.8 prompt tokens/s, then 8.79 generated tokens/s.
  • Ornith: 2,002 input tokens, 433.1 prompt tokens/s, then 13.80 generated tokens/s.
  • Qwen: 2,002 input tokens, 494.9 prompt tokens/s, then 13.97 generated tokens/s.

I’ll take Bonsai at roughly ten tokens a second with space for another question. I have the faster 4K measurements when I want a short answer, and the original models archived when I need eyes. The next time my laptop takes forever to answer, I’m checking layer placement before I blame swap. The pillow can stay on the bed.