circagraph
Sign inSign up

openchronology/timeline-app on GitHub

The OpenChronology repository on GitHub, mirrored by och-import-github: issues and pull requests as durations (ongoing while open), what happened to them as moments, plus releases, milestones, commits and branches. The Branches view shows the most recently active branches each in a track of its own.

Source: https://github.com/openchronology/timeline-app

3,131 moments · 2026-10-05 CE 20:21:49 UTC – 2026-11-16 CE 12:10:45 UTC · by @athan · @athan

#159 Expire in small saves, finding what to delete through the trees

–

  • github
  • github/openchronology/timeline-app
  • github/openchronology/timeline-app#159
  • pull request
  • state:merged

Part of step 5 of #135: retention stops holding a timeline for minutes, and finds what to delete at a cost that follows what it deletes. ## Why - **Lock time.** Retention deleted up to 200,000 moments in one save. Since #158 that save is applied in slices, but it was still one transaction, holding the timeline's lock for minutes at a million moments. A log save arriving meanwhile waits for that lock, and a lock wait counts against the server's 30 s statement limit, so ingestion timed out and backed off during every retention pass. - **Finding what to delete.** `deleteWhere` found moments with a filtered, `LIMIT`ed scan of the timeline, since nothing indexes moments by time. It found durations with an unbounded scan. Both cost more as the timeline grows. ## What changed **Retention deletes in small saves** (`expire`): - Saves of at most 5,000 moments (`RETENTION_SAVE`), each a transaction of its own, so other saves wait for one of them at most. - A run deletes for up to ten minutes; the next run continues. - A save that lost a race with another one (409) re-reads the timeline and tries again, up to 10 times. **One statistics refresh per run.** Saves to forks of 1,000 changes or more refresh statistics before answering: about 10 s at a million moments. That's needed after one large write, but not after every save in a series. `transaction` and `save` now take an `analyses` set that collects the refreshes instead of running them, and retention runs one after its last save. Other saves meanwhile are unaffected: the set is per call. **Finding what to delete** (`deleteWhere`): - **Moments:** walk the moment tree between the range's bounds a page at a time (`oc_events`, which reads through for forks), oldest first, stopping at the limit or at `before`. The cost follows what's found. - **Durations** wholly inside the range: found through the `(timeline_id, first_key)` start index, bounded by the limit too. - **`more`** now accounts for durations as well as moments. **Delete-range API unchanged:** up to 200,000 moments per call, one revision, `more: true` for the rest. It uses the same tree walk. ## Measurements 1M benchmark timeline and its fork, deleting the oldest 50,000 moments, 8-core VM, PostgreSQL 17: | | Saves | Each | Whole pass | |---|--:|--:|--:| | Fork, before (refresh after every save) | 10 | 15–21 s (first 47 s) | 200 s | | Fork, after | 10 | 6.5–8.2 s (first 18 s), + one 13 s refresh | 96 s | | Timeline a fork reads (every deletion keeps a version) | 12 | 8.6–14 s, statements ≤ 3 s | | ## Tests `test/retention-postgres.mjs` adds: - **A long expiry:** 23 expired moments and a duration on a timeline and on its fork (which reads through), in saves of 5. That's five saves on each. - **A lost race:** another save gets in just before retention's first save; retention retries and the other save's moment stays. - **The cutoff:** a moment exactly at it stays. - **A range deletion past its limit:** the oldest 4 go, with `more: true`. - **Paging:** a deletion walking the moment tree 3 moments at a time finds the same: the oldest 10, and the duration before the cutoff. **Fault checks, each caught:** - an inclusive cutoff; - no retry after a lost race; - a walk cursor that doesn't advance. **Other checks:** - all 36 database suites pass locally, apart from the two browser suites this VM can't run; - 258 unit tests pass; - the build passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011fFY8eruEi1xAkkMBni13B https://github.com/openchronology/timeline-app/pull/159

Related

Open #159 Expire in small saves, finding what to delete through the trees in the timeline

#159 Expire in small saves, finding what to delete through…