circagraph
Sign inSign up

openchronology/timeline-app on GitHub

The OpenChronology repository on GitHub, mirrored by och-import-github: issues and pull requests as durations (ongoing while open), what happened to them as moments, plus releases, milestones, commits and branches. The Branches view shows the most recently active branches each in a track of its own.

Source: https://github.com/openchronology/timeline-app

3,131 moments · 2026-10-05 CE 20:21:49 UTC – 2026-11-16 CE 12:10:45 UTC · by @athan · @athan

#158 Apply large patches in place, a slice at a time

–

  • github
  • github/openchronology/timeline-app
  • github/openchronology/timeline-app#158
  • pull request
  • state:merged

Part of step 5 of #135: large patches stop rebuilding the whole index. ## Why A patch changing more than an eighth of a timeline was rebuilt from a whole document in Node (`incremental` in `server/incremental.mjs`). Through the API that rarely mattered, since `/changes` takes at most 5,000 entries. But the server makes large patches itself: - **Retention** and **delete-range** delete up to 200,000 moments per save. - **Merges and syncs** can carry many changes. On a log timeline of a million moments, a retention pass therefore rebuilt everything. Since #154 made in-place saves follow the nodes they touch, in place is cheaper: | | In place | Rebuilt | |---|--:|--:| | Delete 30,000 of 200,000 moments | 26 s, 155 MB | 103 s, 689 MB | | Rebuild of a 1M-moment timeline | | ~10 min, over 1 GB | ## What changed **Threshold.** A patch rebuilds only when it changes more than half of a timeline under 200,000 moments (`REBUILD_UNDER`, `REBUILD_SHARE`). Any other patch is applied in place, however large. In place costs about 1 ms per change and a rebuild about 0.5 ms per moment, so break-even is around half. Past 200,000 moments, building a whole document in memory isn't worth it at all. Plugin changes still rebuild; #157 is about removing that. **Slices.** A large in-place patch is applied 5,000 changes at a time (`slicePatch`, `saveIncrementally`), moments first, then durations (which may name them), then relationships (which may link them). - Each slice is applied to the timeline as the previous one left it: its row is written between slices. - It's still one revision, with one history record over everything the slices touched. - No statement grows with the patch. At a million moments, with a fork reading the timeline so every deleted row is kept as a version, deleting 200,000 moments in one piece ran a 29 s statement, one second under the server's 30 s statement limit. In slices, the longest statement was 10 s, peak memory 259 MB instead of 510 MB, and the whole pass about 5 minutes. 20,000-change slices took about the same time, so the smaller slices keep memory lower. **Log importer:** batches follow the new threshold: at most `max(64, moments ÷ 2)` entries, up to 5,000. ## Tests **`test/incremental-postgres.mjs`**, the oracle comparing in-place saves with rebuilds: - every other random patch is now applied in slices of three changes; - a new case grows a 10-moment timeline by 300 moments, 40 durations, 30 relationships, a deletion and a move, applied in place in slices of 7. That moves the trees' roots repeatedly. It must match a rebuild of the patched document (queries, search, links, arcs), keep valid AVL trees, and read back as one revision. **Fault checks, each caught:** - not updating the timeline between slices (the new case catches it; the random patches alone didn't); - recording history for only the last slice's keys. **Other checks:** - all 36 database suites pass locally, apart from the two browser suites this VM can't run; - 258 unit tests pass (the importer's batch limit updated); - the build passes. ## Not changed - **Delete-range's API contract stays:** up to 200,000 moments per call, with `more: true` for the rest. - **Retention's single save holds the timeline's lock** for those minutes; splitting retention into several saves is a possible follow-up. - **Whole-document saves and merges of revisions** from before history was kept as rows still rebuild, by nature. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011fFY8eruEi1xAkkMBni13B https://github.com/openchronology/timeline-app/pull/158

Related

Open #158 Apply large patches in place, a slice at a time in the timeline

#158 Apply large patches in place, a slice at a time