Brief #13 · AI & LLM
LLM Evaluation Pipeline
Medium⏱ 10-14 hSoon
A test harness that scores an AI feature on every pull request, so a prompt tweak can't quietly make it worse.
Outlines are free · building with tracking needs Pro
LThe briefFrom the team
Our support bot classifies tickets and drafts replies. Someone 'improved' the prompt last week, and refund requests started being tagged as spam. Nobody noticed for four days. Build an evaluation harness that runs in CI and blocks changes that make the bot worse.
What you'll build
- 1A golden dataset of 50 to 100 labelled examples, including tricky edge cases.
- 2Exact checks for structured output: valid JSON, matching schema, correct label.
- 3An LLM-as-judge rubric for free-text replies, with the judge prompt versioned.
- 4Store results per run (SQLite or Postgres) with prompt version, model, score, latency and cost.
- 5A CI job that fails the PR if accuracy drops more than 2 points.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- Re-introducing last week's bad prompt fails CI with a clear report.
- Runs are reproducible: same inputs, same scores (cache or fixed seeds).
- The report shows which examples got worse, not just the average.
- Cost per run is shown and kept under a budget you set.
Stretch goals
- Compare two models side by side on the same set.
- Spot-check the judge against your own labels and report agreement.