Brief #15 · AI & LLM
Fine-Tune a Tiny LLM
Fine-tune a small open model on your own support tickets with LoRA, and prove with an eval set whether it actually beats good prompting.
Outlines are free · building with tracking needs Pro
We pay a large model to sort and draft replies for 40,000 support tickets a month, and the bill keeps growing. Someone suggested fine-tuning a small model we can run ourselves. Before we commit, find out honestly: is a tuned small model as good as the big one with a good prompt, how much cheaper is it, and where does it fail?
What you'll build
- 1Build a labelled dataset of 2,000 tickets (category plus a good reply), and split it into train, validation and a held-out test set nobody touches until the end.
- 2Measure two baselines on the test set first: the big model with a careful prompt, and the small model with the same prompt.
- 3Fine-tune the small model with LoRA, tracking training and validation loss, and stop before it overfits.
- 4Score all three on accuracy for the category and a rubric-based judge for replies, and look at 30 failures by hand.
- 5Serve the tuned model behind an API and measure latency and cost per 1,000 tickets against the big model.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- A results table: accuracy, reply score, latency and cost for all three, on the same held-out test set.
- Training and validation loss curves, and the point where you stopped and why.
- A written recommendation: switch, don't switch, or switch only for some ticket types.
- No test-set ticket was ever used for training or prompt tuning.
Stretch goals
- Quantize the tuned model to 4-bit and measure what it costs in quality.
- Mix Bangla and English tickets and check whether quality holds for both.