#140 Copy a fork's index in the database instead of rebuilding it in Node
–
- github
- github/openchronology/timeline-app
- github/openchronology/timeline-app#140
- issue
- state:closed
Found while measuring #138. ## Problem Since #138, forking or duplicating a timeline no longer copies its saved history. It still builds the copy's live index from scratch in Node (`Versioning.copy` in `server/versioning.mjs`): 1. It reads the source's head revision as a whole document (`revisionDocument`, which assembles every moment in memory). 2. It calls `store.replace`, which validates the document, builds the moment tree in memory (`indexedNodes`), and writes, batch by batch: - `oc_moments`, `oc_nodes`, `oc_duration_nodes`, `oc_edge_nodes`, `oc_relationships`; - the search rows (`oc_entity_search`) and the catalogue text. On the benchmark VM this takes about **2 minutes for 200,000 moments**, about the same as creating the timeline. It also holds the whole document in memory: about 1.3 GB at a million moments, per #135. In practice this makes forking the slowest operation on a large timeline. ## Proposal The copy is made from the source's current head, so its index is exactly the source's live rows. Copy them inside the database, one `INSERT … SELECT` per table (paged if needed): - `oc_moments`, `oc_nodes`, `oc_duration_nodes`, `oc_edge_nodes`, `oc_relationships`, `oc_entity_search`, keyed by the new timeline ID; - plus the timeline row's derived columns (`root`, `event_count`, `duration_root`, `edge_root`, `storage_bytes`, `event_text`, …). Node IDs only need to be unique within a timeline, so they can be copied as they are. (Check this for every tree table.) Then: - run `ANALYZE` on the copied tables (see #139); - record the first revision as #138 does (it shares the source's history). ## Points to check - Anything else `replace` derives that isn't stored in those tables, such as views (`oc_views`, which a copy starts without) and catalogue text staleness. - Quotas: the charge for the copy's `storage_bytes` must still be checked before the copy is made. - Browser forks (`browser-forks.mjs`) are a separate path and not affected. ## Done when - Forking a 200,000-moment timeline takes seconds, not minutes, without loading the document into Node. - The fork's index matches a rebuild from the same document. `test/incremental-postgres.mjs` already has an index-comparison helper (`observe`). Related: #135, #138, #139. https://github.com/openchronology/timeline-app/issues/140
Related
- #135 Scale saves and whole-timeline work toward very large timelines (1 billion moments)
- #138 Share saved history between timelines and their forks
- #139 Saves right after a large write get bad plans from stale statistics
- #144 Copy a fork's index in the database instead of rebuilding it
- #145 Share index rows between a timeline and its forks instead of copying them
- Copy a fork's index in the database instead of rebuilding it
Open #140 Copy a fork's index in the database instead of rebuilding it in Node in the timeline