#202 och-convert's memory and file-size limits stop large timelines with real metadata
–
- github
- github/openchronology/timeline-app
- github/openchronology/timeline-app#202
- issue
- state:closed
Found while fixing #199 (PR #201). With the overview index loaded in bulk, `och-convert write` now writes 200,000 moments in about 9 s, well within its 20 s CPU limit. The other two limits it runs under (`resource_limits` in `native-store/src/bin/och-convert.rs`, Linux) are now the ones a large timeline reaches first: - **Address space, 512 MiB (`RLIMIT_AS`):** at 200,000 moments with a one-word title each, the process peaks at about 400 MB, roughly 1.8 KB per moment. Most of it is the parsed JSON document: `save()` holds the whole `Document`, every moment's metadata as a `serde_json::Value`, before writing anything. - **Output file, 32 MiB (`RLIMIT_FSIZE`):** the same timeline makes a 27.9 MB `.och` (from 13 MB of JSON). Larger metadata pushes it over. Measured on a release build: | Moments | JSON in | .och out | Peak memory | | --- | --- | --- | --- | | 30,000 | ~2 MB | — | 62 MB | | 200,000 | 13 MB | 27.9 MB | ~400 MB | So timelines below the format's 200,000-moment maximum fail once their moments carry real metadata: descriptions, tags, sources. Options: - Parse and insert moments as a stream (e.g. `serde_json`'s `StreamDeserializer` over `events`), so memory no longer grows with the document. - Size the file limit from the input, or from the format's own maximums, instead of a fixed 32 MiB. - Say which limit was hit. (Not checked yet: what message a conversion stopped by `RLIMIT_AS` or `RLIMIT_FSIZE` gives today.) https://github.com/openchronology/timeline-app/issues/202
Related
- #199 och-convert can't write large timelines within its CPU limit
- #201 Large timelines are written with the overview index loaded in bulk (#199)
- #203 .och conversions hold less memory, and their file limit is a setting (#202)
- #299 The file conversion test writes a large generated timeline (#296)
- .och conversions hold less memory, and their file limit is a setting (#202)
- Merge pull request #203 from openchronology/feature/file-limits