circagraph
Sign inSign up

Incident 2026-03-14: orders API outage

Three hours of syslog from web-1, web-2, db-1 and backup-1, with the investigation on top: the cause, the outage and what the on-call engineer did. A made-up incident for the logs video.

2,297 moments · 2026-03-14 CE 01:30 UTC – 2026-03-14 CE 04:29:57.914461 UTC · by @athan · @athan

GET /api/orders 200 35ms

  • syslog
  • severity:info
  • facility:daemon
  • host:web-1
  • app:nginx

At 2026-03-14T01:30:40.616694 UTC, `nginx` on `web-1` emitted a `info` record in the synthetic orders-API incident. The access record describes `GET /api/orders` returning HTTP 200, a successful response, with a logged duration of 35 milliseconds. It identifies one request at one web host, rather than an aggregate rate or a measurement of all users’ experience. The record belongs to a deliberately fictional incident on 14 March 2026, created to demonstrate log-based investigation. The scenario attributes the storage problem to a nightly backup directed to `db-1:/var/lib/backup`, on the same volume as the database. This request falls before the backup begins. Comparing requests across `web-1` and `web-2` helps locate the observed change in service behavior. The 35-millisecond value belongs to this recorded request; it does not identify the database query time or the time needed to resolve the incident. **Recorded message:** ```text GET /api/orders 200 35ms ```

Open GET /api/orders 200 35ms in the timeline