Brief #08 · DevOps & SRE
Production Incident Debugging
Hard⏱ 12-16 hSoon
Break a running system in realistic ways, find each cause from the evidence, fix it, and write a blameless postmortem.
Outlines are free · building with tracking needs Pro
PThe briefFrom the team
Interviewers love asking 'tell me about an outage you handled'. Most juniors have never handled one. So build a small system, then stage outages a teammate (or a script) triggers without telling you: a full disk, a memory leak, a bad config deploy, a dead database. Your job is to recover each one and leave a paper trail.
What you'll build
- 1A small app with a database and the observability stack from the 'glass boxes' project (or similar).
- 2A chaos script that randomly triggers one of at least four failures.
- 3For each incident: detect, mitigate, find the root cause, then fix it properly.
- 4A runbook per failure type: symptoms, checks, commands, rollback.
- 5One full postmortem: timeline, impact, root cause, what went well, action items.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- You recovered every staged failure and recorded time-to-detect and time-to-recover.
- Each root cause is backed by a log line, metric or command output.
- The postmortem blames systems, not people.
- At least one action item is implemented (an alert, a limit, a check).
Stretch goals
- Run a drill with a friend acting as incident commander.
- Automate one mitigation (for example, log cleanup when the disk hits 90%).