#199 och-convert can't write large timelines within its CPU limit
–
- github
- github/openchronology/timeline-app
- github/openchronology/timeline-app#199
- issue
- state:closed
Found while fixing #193. `och-convert write` (server `.och` exports) runs under a 20 s CPU limit (`RLIMIT_CPU`, `native-store/src/bin/och-convert.rs`). Filling the overview index (`points`, the `rational_index` virtual table) costs about 0.35 ms per distinct moment time. That holds for one bulk insert and for one insert per time alike. So a conversion runs out of CPU at roughly 45–50k moments, well under the 200,000 the format allows. Measured on a release build (moments only): | Moments | Overview index | Result | | --- | --- | --- | | 10,000 | ~3.5 s | ok | | 30,000 | ~12 s | ok | | 200,000 | — | killed at the CPU limit | #193 fixed the earlier failure: the instruction budget was shared by the whole conversion and ran out near 9,000 moments. What's left is the index's own insert cost, or the limit chosen for it. Options: - a bulk-load path in sqlite-rational for sorted input, building the tree bottom-up; - a CPU limit that scales with the input's size. My (Athan) recommendation: Make the conversion a streaming process so the event loop isn't occupied for a long time. Additionally, make a size limit for how saves / exports should be allowed for all users except admin or privileged accounts. Its a threat vector that doesn't seem to currently scale well, even if there's a fleet of openchronology servers load balanced - unless the process is cacheable in redis or something. https://github.com/openchronology/timeline-app/issues/199
Related
- #193 Things to look at, found writing #177's comments
- #200 Fixes from the #193 triage list
- #201 Large timelines are written with the overview index loaded in bulk (#199)
- #202 och-convert's memory and file-size limits stop large timelines with real metadata
- #299 The file conversion test writes a large generated timeline (#296)
- Merge pull request #201 from openchronology/feature/bulk-index-load
- The file conversion test writes a large generated timeline (#296)
- Large timelines are written with the overview index loaded in bulk (#199)
Open #199 och-convert can't write large timelines within its CPU limit in the timeline