The joy of running a 35B on My Existing Machine without having to invest in my server GPU’s.
Welcome to part three of my unexpected serries. Part one was getting a 9B running with llama.cpp and fixing the sampling. Part two was the evening I lost to a 401 that turned out to be an environment variable. Same advice both times: the model is usually fine, the settings are the problem.
Part 1: I Spent weeks Fighting My Local AI Coding Agent — Here’s Everything I Learned
Part 2: The Model Was Fine: What I Actually Did to Make Ornith Usable
For a year my plan for running something bigger was two used Tesla P40s in the Dell R720 I use as a backup server. I priced and they all were out of my reach as priority was on family and living then wants.
Then stumbled upon PolyStrata and installed on my gaming computer with the RTX 4070 that was already under my desk and had a 35B answering at 50 to 80 tokens per second that evening. Same card. No new power supply, no rack, no noise.
What PolyStrata is
The line on the repository is the pitch: run large AI models on your own PC.
Twelve models, two engines. Qwen3.8-Flash-Next runs on the Strata project’s engine, on a 12 GB NVIDIA or AMD card. The other eleven run on an engine written for PolyStrata: GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Flint, Qwen3.6, Clover, Ornith 1.5, Gemma 4, Tiel Coder, MiniCPM-V and Kolibri 1.
One-click install on Windows and Linux. OpenAI-compatible and Anthropic-compatible APIs on localhost, so Claude Code, Pi and everything else connect the way they would to a paid endpoint, and nothing leaves the machine.
Why I tried it
Two reasons, The installer meant finding out whether a 35B ran on my card cost an hour instead of a weekend and a power supply. And it ran that 35B on one 12 GB card rather than the two I had just priced.
If it had needed a day of flags I would have read the README and closed the tab, the way I have with a dozen other projects. Everything after that was a bonus: a generated config that keeps the old copy as .bak, a web UI, and an MCP server that lets an agent start it.
Why a 35B fits in 12 GB now
Nothing about my card changed. Three things changed around it.
A big model does not have to work all at once. Ornith 1.5, and Tiel Coder built on it, is a 35B with about 3 billion parameters active per token, choosing 8 experts out of 256 plus a shared one. The card never pushes 35 billion numbers through the math for one word. The Hugging Face explainer covers the idea.
What does not fit can live in RAM. My pack is 17 GB against 12 GB of VRAM, so part of it sits in the 32 GB of system RAM and gets fed over PCIe. That is the wall the P40s would have knocked down with money, knocked down here with memory I already owned and paid for in speed. --ram-experts-mib sets how much RAM the engine holds for the experts outside VRAM; you write a few percent slower for it, and the project measured that trade instead of guessing at it.
Conversations got cheaper too. Ornith interleaves three linear-attention layers with one full-attention layer, which keeps the cache small and is why 131072 tokens of context is affordable here. Quantised 35B files land between 14 and 24 GB.
The project’s developers numbers show the direction: Ornith 1.5 writes at 76 to 82 tokens per second on a laptop 4070 with 8 GB of VRAM, Tiel at 83, Qwen3.6 at 94 to 100. Those were workstation numbers a year ago.
Why Tiel Coder
I started on Ornith 1.5 and moved to Tiel Coder, which the repository describes in six words: Ornith 1.5 quantized for coding agents.
Same architecture, same offload arithmetic, 14 to 23 GB, vision still there. Nothing about my machine changes and neither does the speed: 83 tokens per second on the project’s 8 GB laptop card, against 76 to 82 for Ornith.
What changes is where the tuning went. Almost everything I ask it is agent work: read these files, call this tool, write the diff, keep the schema. A model tuned for that is a better use of the same weights than one that also has to be good at the weather.
Moving cost about 3 hours of me fiddling with powershell and nothing else. I only slightly needed to update pi’s agent config to point at the new model. I named the family and the quant, setup resolved and verified the files, wrote a new config, and I have not gone back to Ornith since. I decided to first test it with doing final touches on my blog cms platform I had been developing. I noticed a drastic difference. Where my orntith 9b model would spend a long period of time deciding back and forth.. Even with decent tokens per second. It would take much of a day to iterate trough features. Sometimes I would have to break out another model using freebuff to get it past the humps.
One hour of working with the 35b model had turned my world around.. It is so much more usable… With Ploystrata normal users are no more limited to very small models that struggle with agentic coding tasks.
On my machine
- UD-Q3_K_XL, 17 GB on disk, 12 GB of VRAM and 32 GB of system RAM.
- 50 to 80 tokens per second writing. Short prompt and a fresh session at the top of that range, long context at the bottom.
- 131072 tokens of context, a 2048-token reasoning budget, vision through a separate projector.
- 30 to 60 seconds of cold start while 17 GB faults in from disk.
What actually changed
The 9B was fast and it fit entirely in VRAM, but it could not hold a plan. Forty-three tools from three MCP servers in the prompt, and it made reasonable-looking decisions that ignored something I said four calls earlier. I would notice twenty minutes later and undo a file or two.
The 35B does that less. And at 50 to 80 tokens per second the reply appears while I am still reading my own prompt, so an agent runs several tool calls without me wondering whether it is stuck. Four tokens per second is a demo; this is a tool.
Two things I have done with it. I rewrote the body handling in the script that generates my other blog with the whole file in context. And I am working on a program to pass files between machines like Erugo, where the model has to remember a schema, an upload path and a permissions decision for twenty turns. The 9B kept two of the three.
Before you install
- Repository README for install and the speed tables.
- docs/BACKENDS.md for which quant fits which card and how much RAM sits beside it.
- docs/models/tiel.md for what is inside the model you are downloading.
- docs/FAQ.md for the two-engine question and client setup.