circagraph
Sign inSign up

Incident 2026-03-14: orders API outage

Three hours of syslog from web-1, web-2, db-1 and backup-1, with the investigation on top: the cause, the outage and what the on-call engineer did. A made-up incident for the logs video.

2,297 moments · 2026-03-14 CE 01:30 UTC – 2026-03-14 CE 04:29:57.914461 UTC · by @athan · @athan

checkpoint complete

  • syslog
  • severity:info
  • facility:daemon
  • host:db-1
  • app:postgres

At 2026-03-14T01:32:43.399592 UTC, `postgres` on `db-1` emitted a `info` record in the synthetic orders-API incident. PostgreSQL reports completion of a checkpoint. In this exercise the message is routine database activity, providing a reference point for the host’s behavior among the web requests, disk-monitor records and later write failures. It records a completed checkpoint, not a successful orders-API request. The record belongs to a deliberately fictional incident on 14 March 2026, created to demonstrate log-based investigation. The scenario attributes the storage problem to a nightly backup directed to `db-1:/var/lib/backup`, on the same volume as the database. This checkpoint falls before the backup begins. Its timestamp can be compared with the storage-pressure and recovery milestones. The short message supplies no checkpoint duration, count of written pages or guarantee that every transaction has succeeded. **Recorded message:** ```text checkpoint complete ```

Open checkpoint complete in the timeline