circagraph
Sign inSign up

Incident 2026-03-14: orders API outage

Three hours of syslog from web-1, web-2, db-1 and backup-1, with the investigation on top: the cause, the outage and what the on-call engineer did. A made-up incident for the logs video.

2,297 moments · 2026-03-14 CE 01:30 UTC – 2026-03-14 CE 04:29:57.914461 UTC · by @athan · @athan

Accepted publickey for deploy

  • syslog
  • severity:notice
  • facility:auth
  • host:web-1
  • app:sshd

At 2026-03-14T01:30:34.813780 UTC, `sshd` on `web-1` emitted a `notice` record in the synthetic orders-API incident. The SSH daemon reports successful public-key authentication for the account `deploy`. This identifies an authenticated login event on the named host. It contains no command history and does not identify the human behind the account or the purpose of that session. The record belongs to a deliberately fictional incident on 14 March 2026, created to demonstrate log-based investigation. The scenario attributes the storage problem to a nightly backup directed to `db-1:/var/lib/backup`, on the same volume as the database. The login falls before the backup begins. It is useful background when correlating access with the response chronology, but timing alone does not prove that this session started the backup, diagnosed the problem or performed the cleanup. **Recorded message:** ```text Accepted publickey for deploy ```

Open Accepted publickey for deploy in the timeline