Brief #14 · AI & LLM
Build Your Own Agent Harness
Everything around the model that turns an LLM into a working coding agent: the tool loop, permissions, context management, a sandbox, and evals.
Outlines are free · building with tracking needs Pro
Everyone on our team uses an AI coding agent, but nobody can explain what happens between typing a request and a file changing on disk. When it deletes the wrong thing or loops forever, we're helpless. Build a small agent harness of your own, a terminal agent that can read, search, edit and run code in a repository, so the team understands every moving part.
What you'll build
- 1The core loop: send the conversation and tool schemas to an LLM API, run the tool calls it asks for, append the results, and repeat until it answers, with a step limit.
- 2Five tools with JSON schemas: read_file, list_files, search (ripgrep), edit_file (exact string replace) and run_command.
- 3A permission layer: reads are allowed, edits and commands ask the user first, and a deny list blocks dangerous commands outright.
- 4Run commands in a sandbox (a Docker container with the repo mounted, no network, time and memory limits) and stream their output.
- 5Context management: count tokens, truncate huge tool outputs, and summarise old turns when the window fills up.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- Given a small repo with a failing test, the agent finds the bug, fixes it and makes the test pass on its own.
- No edit or command runs without approval, and
rm -rf /never reaches the shell, even if the model asks. - A loop that makes no progress stops at the step limit with a clear message instead of burning tokens.
- Every run logs each step (tool, input, output size, tokens, cost) to a file you can replay.
Stretch goals
- An eval set of 10 small tasks with a success rate you track across prompt and model changes.
- Expose your tools over MCP so other agents can use them.