How I built a coding agent that runs entirely on my own hardware — no API, no cloud, no monthly bill — and the build steps that actually work.
A while back I decided I wanted a coding agent I could leave running without paying for it. Not a toy, not a trial. Something that could work on a project, use tools, and not call home every time it typed. The recipe sounds simple: a capable model, a serving layer that translates its output correctly, and a frontend to talk to it.
Getting there took a lot of trial and error — VRAM arithmetic, a serving stack that kept crashing, a Docker Desktop disk full of itself, and a few rules I had to write to keep a small model from looping on itself. The end result is a stack that works and is cheap to run. I’ll walk you through it, the way it actually happened.
Warning: Use at Your Own Risk
This is an agentic coding agent — it writes, edits, and runs things on its own. That’s the point. Run it in an isolated environment and point it at scratch work, not production data. An LLM can look certain and still make a destructive mistake, and an agentic loop that rewrites files can spend a while getting there before it stops. There’s no safety net in a bare-metal box.
A few rules I follow:
- Keep it off the production path. Run this on a throwaway project, a fork, a spare VM, or a container with a read-only or thin overlay of your real code. Whatever it touches, you can afford to throw away.
- Never let it reach system-level or privileged targets. No
sudo, no~/.piedits, no changes to your host config, no writes outside the sandboxed project. System-level work is where an agentic model’s confidence and your filesystem’s fragility meet. - Give it a hard stop. The
APPEND_SYSTEM.mdpolicy caps how long it can deliberate and lists exactly what it’s allowed to do. Read it before you trust the agent with anything you don’t want to retype. - It’s cheap, but not free. The model is free, but a bad run costs your time. I spent more hours watching it fail than coding, and I’d rather you not start there.
This build works on my hardware. If you run it somewhere else, expect the disk, the drive letters, and the IP addresses to be different. Adapt at your own risk.
This guide is written from what I did on my own machine, not from a documented-from-scratch deployment. I haven’t rebuilt it from a clean install since, so treat it as guidance, not a script to copy verbatim. Drive letters, paths, and the exact order of steps may differ when you start fresh. The steps are right enough to follow; the exact bytes aren’t guaranteed to match.
The Stack
To run a local agentic coding agent, you need five moving pieces:
Model — the brain. Ornith-1.5-9B, a 9B-parameter reasoning model. Small enough to fit on a modest GPU with room for a long context, big enough to reason and call tools.
Serving layer — llama.cpp, the open-source inference engine. It loads the model and exposes it over HTTP. I ran the CUDA build so it used my GPU directly.
Frontend — Pi, an open-source coding agent. It connects to the serving layer, drives the agent, and calls tools.
Web frontend — Pi Web, a browser UI for Pi. Same config, same sessions, from any laptop on the network.
Policy layer — a set of rules I added to keep the agent honest. This is the part most guides leave out, but it’s the part that actually made it usable.
The Setup: Two Machines
This stack runs across two machines, which is the part most guides don’t draw.
The Windows desktop is the model layer. It has an NVIDIA 4070 (12 GB) and runs Docker Desktop with the llama.cpp CUDA server. The model and its weights live here, on the GPU, where they belong — VRAM is the constraint, so the compute stays on the card.
The Ubuntu 24.04 AI VM is the agent layer. It connects to the Windows desktop over the LAN and talks to the llama.cpp endpoint. On this VM I run Pi and Pi Web — the agent, the browser UI, and the web tools. It’s a throwaway-enough box that it’s fine to point it at real work, but keep the same rules: it should never have write access to the Windows host’s production files.
The split matters because it changes where you install things:
- The model and Docker stay on Windows, on the GPU. That’s where the weight download and the CUDA context cache live.
- Pi, Pi Web, and the agent work run on the Ubuntu VM. Pi is installed here, not on Windows — the VM reaches the model over the network instead of the GPU.
- Pi and Pi Web bind to the LAN. Pi points at the Windows host’s IP (
192.168.2.248:8000), and Pi Web binds to all interfaces so it’s reachable on the network. Both machines are on the same subnet, so it’s just a normal LAN connection.
The cost of this split is a hop: every request goes from the VM to the Windows host over the network. On a LAN it’s negligible. The upside is separation — the agent can’t touch the model’s files, and the GPU host stays clean of agent tooling. That separation is also why the isolated-environment warning matters: the VM is your sandbox, and the Windows desktop is your real machine.
Before You Start: VRAM Math
The first decision is the model, and the model is decided by your GPU. A 30B model doesn’t fit on a small card, and a 30B model that fits leaves no room for a long context. The honest question to ask is “what fits,” and the honest answer comes from nvidia-smi.
Here’s the rough arithmetic for a 9B model:
- Model weights (Q4 quantized): roughly 5.7 GB.
- KV cache for a long context (131,072 tokens, fp8 cache): roughly 0.5–0.7 GB.
- Everything else (scratch, activation): a little more.
That’s about 6–7 GB in use at full utilization. So an 8 GB GPU is workable, a 12 GB GPU is comfortable. If nvidia-smi doesn’t have that much free memory, don’t start with a bigger model — you’ll spend more time fighting VRAM than coding.
The lesson from the beginning: pick the GPU first, and the model almost picks itself.
Ornith-1.5-9B is the choice for a reason. It’s a dense ~9B model built on Qwen3.5, and it’s the strongest 9B-class coding model available — it beats bigger models on the benchmarks that matter. On SWE-bench Verified it scores 70.6, above both the 31B Gemma-4 (52) and the 9B Qwen3.5 (53.2) it’s derived from. Terminal-Bench 2.1 lands at 46.2 (Terminus) and 47 (Claude Code). It’s a reasoning model, so it emits a thinking step before answering, and it ships an OpenAI-compatible endpoint with tool calling that works out of the box. That combination — a strong small model, a known-good template, native tool calling — is exactly what I need. There’s no reason to run something bigger and slower.”
The First Attempt: Building a Custom Model with a Modelfile
Before the serving-layer mess, I tried the obvious thing: take the model and constrain it. Ollama lets you build a custom model on top of a base model with a Modelfile — a text file that says what to change. I wrote one for Ornith.
mkdir "C:\Users\Brian\Downloads\"
@"
FROM ornith-1.5:9b
PARAMETER temperature 0.1
SYSTEM "You are an automated coding engine. When writing or updating files, output the raw code content ONLY. Do not include thinking tags, markdown code blocks (no ```), or explanatory chatter inside tool arguments."
"@ | Out-File -Encoding ascii Modelfile
ollama create ornith-clean:latest -f ModelfileThree directives, and each one was a deliberate choice:
FROM ornith-1.5:9b — the base model. The custom model inherits the base weights but overrides what the base does. I wasn’t retraining anything; I was dressing up the existing model with a narrower personality.
PARAMETER temperature 0.1 — the sampling temperature. Low temperature means deterministic output; the model picks the most likely next token instead of exploring. For coding, variance is a liability. I wanted predictable code, not creative variation. 0.1 is near the floor.
SYSTEM "..." — the system prompt override. The intent was to turn the model into a focused coding engine: output raw code only, no prose, no markdown fences, no thinking tags. The idea was that if I told it hard enough what to do, it would behave.
ollama run ornith-clean:latestIt ran. On simple tasks it was clean — exactly what the model was trained to do, dialed down. But under agentic work it fell apart, and the failure mode was deceptive. The system prompt could tell the model what to output, but it couldn’t stop the model from emitting thinking tags it was trained to emit. It couldn’t fix the tool-call format mismatch between Ollama and the model’s native output. And it couldn’t make the model trust tool results or stop looping.
The lesson: you can tune a model’s personality, but you can’t tune its serving layer. The next step was to give it a serving layer that matched how it actually talks — which led straight into the nightmare.
The Nightmare: Why Ollama and vLLM Corrupted Everything
The first thing I tried — Ollama — corrupted my work for weeks. Not a small glitch. Full loops, repeated destructive edits, output that made no sense. I spent more time watching it fail than coding. I’ll explain what actually happened, because the fix depends on understanding the failure.
How the model talks
Ornith doesn’t just write text. When it needs to do something — run a command, edit a file — it emits a structured tool call. A markup tag wrapping the action and its arguments. The serving layer’s job is to recognize that tag, extract the action, and trigger the tool. If the recognition fails, the tool never runs — but the model keeps talking as if it did.
That mismatch is the whole problem. The model was trained to emit calls in one format. The serving layer expects a different format. Most single calls still translated, because they fit loosely. But under agentic work — several rapid tool calls, multi-step reasoning with calls — the translation broke.
The Ollama failure
Under Ollama, the tool-call format and the server’s parser didn’t line up. The result looked exactly like context overflow: garbled output, repetition, the model confusing itself. But it scaled with tool-call frequency, not conversation length — the tell that pointed at the parsing layer, not the context.
Worse, it looped. The model would fire a tool call, the call wouldn’t register, and it would fire the same call again — and again — each time adding to the context. A second identical action is almost always the bug, not the fix. The model has no incentive to stop, and each failed cycle bloats the context until the whole thing degrades. That’s the full corruption looping that took weeks to stabilize.
The vLLM escalation
I moved to vLLM hoping for better parser support, and it made things worse before it made them better. vLLM needs explicit parser flags, and Ornith’s output format didn’t map cleanly to any of them. I went through a dozen combinations:
docker run -d --gpus all --name vllm-ornith ...
--tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-model-len 131072 --gpu-memory-utilization 0.95
# still crashed
--max-model-len 16384 --gpu-memory-utilization 0.80 --dtype half
# bumped utilization, fp8 KV cache, still crashed
--max-model-len 16384 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 --enforce-eager
# W4A16-g128 quant, 32k context
quocbao747/Ornith-1.5-9B-AWQ-W4A16-g128 --max-model-len 32768 --gpu-memory-utilization 0.92
# more precise, more memory-efficient, but its own quirks
--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-model-len 16384 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 --enforce-eager
# the model that actually ran
--tool-call-parser hermes --reasoning-parser qwen3 --max-model-len 131072 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 --enforce-eagerEach one was a gamble. Some recognized the call format; most didn’t. And every promising run crashed first — vLLM’s memory management couldn’t hold the model plus a long context plus fp8 KV cache at full utilization. I was tuning three knobs at once — --max-model-len, --gpu-memory-utilization, and the quant/dtype — all of which interact non-linearly. Raise the context and you need more KV-cache memory; raise utilization past the fit point and vLLM dies. It worked sometimes, which made it maddening. And while I was fighting vLLM, Docker Desktop’s disk filled C: and broke in ways that were hard to diagnose, until I moved the whole Docker data to a larger drive and reinstalled.
I was deep in quantization and memory archaeology for a problem that was really about how to parse the output — not what bits to store it in.
The fix: llama.cpp
I ran the simplest experiment I could have imagined at the start: run llama.cpp’s server directly, with a known-good chat template.
powershellCopy
docker run -d --gpus all --name llamacpp-ornith --restart unless-stopped `
-v B:\models\ornith-gguf:/models -p 8000:8000 `
ghcr.io/ggml-org/llama.cpp:server-cuda `
-m /models/Ornith-1.5-9B-Q4_K_M.gguf `
--alias ornith-1.5-9b --host 0.0.0.0 --port 8000 `
-c 131072 -ngl 99 --jinja `
-ctk q8_0 -ctv q8_0 --reasoning-budget 4096Two things mattered.
The Jinja chat template. llama.cpp lets you specify the exact chat template the model was trained to emit. For a model that emits XML-style tool calls, this means the server recognizes the start and end of a tool call at the grammar level — it never leaks into the visible text, and the tool always runs. No compatibility shim, no parser gamble. The parsing problem disappeared almost entirely.
Direct visibility. With llama.cpp, I could see exactly how each request was parsed and served. No black box. And --reasoning-budget 4096 caps how much the model can deliberate before it must answer — the server-side version of bound deliberation, which stops the looping.
The corruption stopped. The loops stopped. I could leave it running and trust it.
Step 1: Docker Desktop
llama.cpp ships a CUDA image, so Docker makes the GPU accessible without installing the CUDA toolkit on the host.
- Download Docker Desktop for Windows from the Docker site.
- Install it. It pulls in WSL 2 and the Docker Engine.
- On first launch, it configures WSL 2 as the backend. Keep the default options.
- Restart your machine so the WSL integration finishes.
Verify Docker is running and can see the GPU:
docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smiIf nvidia-smi prints GPU usage stats, Docker sees the card. If not, the WSL backend needs a restart.
Store the Docker data on a drive with plenty of space. The default lives on C:, and a serving stack fills it faster than you’d like. Point it at a larger drive from Docker’s settings if you have a bigger disk.
Step 2: Start the llama.cpp Server
Pull the CUDA server image:
docker pull ghcr.io/ggml-org/llama.cpp:server-cudaThen run the server. The flags matter, so here’s the full command with what each one does:
docker run -d --gpus all --name llamacpp-ornith --restart unless-stopped `
-v B:\models\ornith-gguf:/models -p 8000:8000 `
ghcr.io/ggml-org/llama.cpp:server-cuda `
-m /models/Ornith-1.5-9B-Q4_K_M.gguf `
--alias ornith-1.5-9b --host 0.0.0.0 --port 8000 `
-c 131072 -np 1 -ngl 99 --jinja `
-ctk q8_0 -ctv q8_0 --reasoning-budget 4096The flags:
-m /models/Ornith-1.5-9B-Q4_K_M.gguf— the model file. TheQ4_K_Mquant is the choice: roughly 5.4 GB on disk, a good balance between quality and size. Higher fidelity quants exist (Q8_0 ~9.1 GB, BF16 ~17 GB), but Q4_K_M is precise enough for coding and small enough to leave room for the 128k context.--alias ornith-1.5-9b— a name to identify the model.--host 0.0.0.0 --port 8000— bind to all interfaces so Pi can reach it.-c 131072— context length, 128k tokens.-ngl 99— put the layers in GPU memory.--jinja— use the Jinja2 chat template to parse tool calls. This is the key flag for agentic work.-ctk q8_0 -ctv q8_0— quantize the context tokens and values for lower memory.--reasoning-budget 4096— cap how many tokens the model can spend reasoning before it must answer. More on this later.
The model file needs to exist at that path, so get it first.
Get the Model
The model comes as a GGUF file. I pulled it directly from Hugging Face with the Python client:
mkdir "B:\models\ornith-gguf" -Force
pip install huggingface_hub
python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='ornith-ai/Ornith-1.5-9B-GGUF', filename='Ornith-1.5-9B-Q4_K_M.gguf', local_dir='B:\models\ornith-gguf')"The repo. The GGUF builds are published at ornith-ai/Ornith-1.5-9B-GGUF. It has several quantizations — Q4_K_M, Q5_K_M, Q6_K, Q8_0, and BF16. I took the Q4_K_M build: about 5.4 GB, the sweet spot between quality and size.
The file. Ornith-1.5-9B-Q4_K_M.gguf, stored in B:\models\ornith-gguf\. A larger drive helps — the GGUF plus the quantized context cache at 128k context comes to roughly 6–7 GB in use, so keep it off C: if you can.
The mount. The file is on the host and mounted into the container with -v B:\models\ornith-gguf:/models. The server loads it with -m /models/Ornith-1.5-9B-Q4_K_M.gguf — the path inside the container, not the host path.
If you’d rather not install Python, download Ornith-1.5-9B-Q4_K_M.gguf from the repo and place it in B:\models\ornith-gguf\.
Verify the server is running:
docker logs -f llamacpp-ornithYou should see the model loaded, context size, and a message like Listening on 0.0.0.0:8000. Note the host’s IP address — you’ll need it to reach the server from Pi.
Step 3: Install Pi
Pi is the agent. It connects to any OpenAI-compatible server, so the llama.cpp server is just a provider to it.
Pi runs on the Ubuntu VM, not Windows. Install it there with Node.js 22.19 or newer (the installer handles that):
curl -fsSL https://pi.dev/install.sh | sudo bashOr install it directly with npm:
npm install -g --ignore-scripts @earendil-works/pi-coding-agentVerify it’s installed:
pi -vStart it:
piPi runs interactively. Pick a project directory, and it loads. You’ll want to set the reasoning level — more on that in the next step.
Step 4: Point Pi at the Server
This is where the pieces connect. Pi runs on the Ubuntu VM and reaches the model on the Windows host over the LAN. Pi reads its config from ~/.pi/agent/models.json on the VM, and the provider there points at the Windows host’s llama.cpp endpoint (http://192.168.2.248:8000/v1). That IP is the Windows desktop’s address on the local network — you’ll see it from ipconfig on Windows or ip addr on the VM. Both machines are on the same subnet, so the connection is just a normal LAN call; the model weights never leave the GPU host.
Create or edit ~/.pi/agent/models.json:
{
"$schema": "https://pi.dev/schemas/models.schema.json",
"providers": {
"ornith-llamacpp": {
"name": "Ornith (local llama.cpp)",
"baseUrl": "http://192.168.2.248:8000/v1",
"api": "openai-completions",
"apiKey": "ollama",
"models": [
{
"id": "ornith-1.5-9b",
"name": "Ornith 1.5 9B (llama.cpp 128k)",
"input": ["text"],
"contextWindow": 131072,
"maxTokens": 8192,
"cost": {
"input": 0.000000431,
"output": 0.00000112,
"cacheRead": 0.0000000862
},
"samplingParams": {
"reasoning_budget_tokens": 4096,
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20,
"min_p": 0.0,
"repeat_penalty": 1.0
}
}
]
}
}
}A few notes on the fields:
baseUrl— the server. Replace192.168.2.248with your host’s IP. The trailing/v1is the OpenAI-compatible path.api—openai-completions. Pi sends the model ID as the model name.apiKey— the dummy valueollamaworks. The llama.cpp server ignores it.contextWindow— set to 131072 to match the server. Pi won’t let you use more than this.samplingParams— the coding settings.temperature 0.6gives the precise, less chatty output the model expects. Pi forwards these straight into the request, so no proxy is needed.
Reload the config in Pi:Reload the config in Pi:
/reloadThis reloads Pi’s context files — including the models.json provider you just edited — plus keybindings, skills, themes, and other settings. Run it after any config change so the UI reflects the new provider.
Step 5: Coding Sampling and Cost
The sampling block above is what makes the model behave the way it’s supposed to. A reasoning model at high temperature wanders. At temperature 0.6 it’s tighter and more deterministic, which matters when it’s editing code.
Pi’s client forwards these values directly to the server, so the model gets exactly the behavior it was trained to expect. There’s no need for a proxy or extra setup — the values live in the config.
For cost, the cost block lets Pi show you how much each message would cost. The rates are per-token estimates (input, output, and cached reads). The model itself is free, but tracking tokens is useful — you can see whether a session burns a lot or stays cheap. Pi shows the estimate next to each message in the web UI.
Step 6: Reasoning Level
Ornith is a reasoning model, so it uses a thinking step before answering. Pi has a setting for how much thinking to allow. The levels are off, minimal, low, medium, high, xhigh, and max.
Set it inside Pi with the command:
/thinking offor type /thinking and pick the level from the list. Or set it as a launch environment variable:
$env:PI_REASONING_LEVEL="off"The llama.cpp server emits the thinking block on its own, regardless of Pi’s setting. Pi controls whether that block is shown. Leaving the level off gives a clean conversation without the raw thinking text, while the server still reasons. Either way, the server handles the actual reasoning budget — Pi only decides what to display.
The value is confirmed from the shell tools; you can print it with:
echo $env:PI_REASONING_LEVELStep 7: Context7 and Web Search
A coding agent is only useful if it can look things up. Two MCP servers make that possible, and both are self-hosted so nothing leaves your network.
Context7 gives live documentation for libraries and APIs, so the agent reads real docs instead of guessing. SearXNG + Crawl4AI is a self-hosted search setup — SearXNG aggregates search results, and Crawl4AI fetches and reads the pages. Neither needs to touch your network in a way that leaves data behind, since SearXNG is self-hosted.
Both are configured in ~/.pi/agent/mcp.json as MCP servers. Pi reads that file at session start, so the agent sees them as tools automatically. The entry looks like this:
{
"mcpServers": {
"context7": {
"command": "node",
"args": ["...context7 server entrypoint..."],
"exposure": "direct"
},
"search": {
"command": "npx",
"args": ["-y", "searxng-crawl4ai@latest"],
"description": "Self-hosted web search and page extraction"
}
}
}You can also add or replace a server from the command line with:
pi mcp add <server> -- <command> [args...]Pi loads them with the session, so the agent can call web search and read documentation as tools.
Step 8: Pi Web
Pi Web is a browser frontend for the same Pi agent. It runs on the Ubuntu VM, using the same config and session files, so you can browse conversations, run agent turns, and inspect files from a browser.
Install it on the VM:
npm install -g @agegr/pi-web@latestRun it:
pi-webIt opens a browser at a local port. By default it binds to 127.0.0.1. To reach it from another machine on the network, bind to all interfaces with a password:
PI_WEB_PASSWORD='a-long-random-password' pi-web --hostname 0.0.0.0The --hostname 0.0.0.0 is the important flag: it makes Pi Web reachable on the LAN instead of only on the VM’s own loopback. Open it from any machine on the same subnet — a laptop on the network, or the Windows host itself — and you can watch the agent work or run turns without touching the VM. The browser just needs to be on the same network as the VM.
The agent connects to the same llama.cpp server through the same models.json config, so Pi Web and Pi see the same model. You can resume a session started in the terminal, or start one from the browser.
On a trusted LAN, a password is enough. Don’t expose Pi Web over plain HTTP to the internet — use HTTPS through a reverse proxy.
What Pi Web gives you
Pi Web is what makes the stack worth running on your own hardware, because it removes the worst part of working with a local model: the terminal.
- Talk to it from any browser. No more squinting at a 120-line PowerShell command or a terminal you can’t resize. A normal window, normal history.
- Browse sessions and files. Open a past conversation, read what the agent wrote, jump into a file — the browser is a second terminal that sees the same sessions and config.
- Share it. On a LAN, hand anyone the URL and they can watch or run turns without touching Docker or the host.
- Resume anywhere. Start a turn in the terminal, pick it back up in the browser hours later, on a laptop.
- Same agent, different surface. It’s not a separate frontend with its own model. It connects to the same llama.cpp server through the same
models.json, so Pi Web and the CLI drive the exact same model and costs.
The tradeoff is real, and it’s the reason the CLI is still the primary interface for most of the work:
Copying text to the remote shell is slow. The biggest annoyance isn’t the model — it’s getting a command from the guide onto the remote box. It does not work like going trough putty..
Step 9: The Policy Layer
This is the part most guides skip, but it’s what makes a small model safe to leave running. The model reasons, and under agentic work it can loop — it retries a destructive action it hadn’t confirmed, keeps “fixing” a state that was never really in that state, and bloats the context with each cycle.
The strongest guardrail isn’t a smaller context. It’s explicit rules about trust and termination, plus a hard cap on how long it can deliberate. I put these in ~/.pi/agent/APPEND_SYSTEM.md, which Pi loads into the system prompt.
- Never emit raw tool-call markup as text. Invoke tools only.
- If a tool call did not clearly execute, check the actual result before proceeding.
- A second identical destructive action is almost always the bug, not the fix.
- Make one atomic edit, then read it back — don't compose multi-file rewrites in your head.
- Bound deliberation: answer instead of looping over reasoning.
- Read after every write. Never assume a file is in the state you expect.Two layers together: the serving layer parses tool calls correctly, and the policy layer stops the model from looping. Change either one and the other matters less.
What I Kept
Four things carried through the whole process:
- Diagnose the layer you’re changing. Parsing failures look like context overflow. Watch what the symptom scales with — here, tool-call frequency, not conversation length.
- Frontend and serving layer are separate. Tune Pi’s config and the llama.cpp flags independently.
- A model is willing, not forced, to loop. The guardrail is explicit rules about trust and termination, plus a hard deliberation budget.
- Simplicity is a feature. The most elaborate serving stack I built was also the most fragile. A 9B model run directly with a known-good template beat three layers of abstraction I couldn’t fully map.
If you’re running a tool-use-heavy agent locally and seeing garbled output or reasoning loops that look like context problems, check the parsing and policy layers first — not the context size. It cost me real debugging time to chase the wrong layer before I found the right one.