Observability & Incident Response Platform
On-call lab platform that turns host pressure into a defensible investigation—multi-host metrics with history, alerts fired on deliberate injects, safe snapshots, honest trend ordering, and structured incident reports.
StackPython 3 · SQLite · JSONL alerts · HTML dashboard
Signalsmetrics history · proven alerts · do-no-harm snapshot · correlate · PIR report
Requirements outline
Scope this page covers. Each row is an acceptance item the automation targets; delivery is what ships in this revision.
| ID | Requirement | Delivery |
|---|---|---|
| FR1 | Metrics collection and storage with history | collect → SQLite samples; prune; multi-host |
| FR2 | Investigative dashboards on real data | ops-health.html — exhaustion/co-movement questions |
| FR3 | Alerts on deliberately caused conditions | lab inject + evidence/04-alerts-fired.json |
| FR4 | Automated incident snapshot | snapshot CLI; do-no-harm budget; pre-evaporation capture |
| FR5 | Trend correlation capability | correlate CLI — order + r + honesty block |
| FR6 | Incident report generator | report → state/incidents/*.md JSON; evidence copy |
Evidence
Terminal captures from the lab host. Click an image to expand fullscreen.
Problem
I needed tooling an engineer would actually reach for at 2 AM: not wallpaper graphs, but history, alerts proven on real triggered conditions, a safe freeze of host state, correlation that admits what it cannot prove, and a report fit for a post-incident review.
Approach
Python obs_platform with versioned config/observability.json. Collect low-footprint samples from a real local host plus synthetic agents into SQLite. Alerts require consecutive breaches. Lab injects mem/load on lab-agent-a and records actual fire events. Snapshot runs under a wall-clock do-no-harm budget. Correlate orders threshold crossings with Pearson r and explicit non-causation notes. Report emits Markdown+JSON; dashboard answers exhaustion and co-movement questions from real data.
Outcomes
- Multi-host metrics retained in SQLite (trend windows, not only latest)
- mem_high and load_elevated fired on deliberate lab-agent-a inject (evidence)
- Snapshot completed within budget with no mutations
- Correlation showed mem_pct before load1_per_cpu matching injection order
- Generated PIR-ready incident report + investigative HTML dashboard