Import log files into timelines with och-import-logs
- github
- github/openchronology/timeline-app
- commit
- main
- github/athanclark
The syslog receiver handles live streams only; replaying files into it gave wrong times (ISO lines unread, classic lines in UTC and the year nearest now), lost levels, overflowed its queue and duplicated on re-runs. och-import-logs (server/import-logs.mjs) imports files, .gz and stdin, one moment per log entry: - Formats (log-formats.mjs, log-time.mjs): syslog (rsyslog ISO, classic, with priority), rfc5424, Apache common/combined and error, nginx combined and error, JSON lines and journald, plus custom regex or JSON formats with typed fields and multi-line joining. Times are exact Unix seconds; zones are fixed offsets or IANA names; classic years follow the file's modification time (or --year), stepping across New Year. - Moments (log-mapping.mjs): title, notes and tag templates, a typed JSON metadata template (plugin keys included), severity from a field or text rules, optional durations, every field under metadata. , and IDs from each entry's text and occurrence, so re-imports overwrite. - Routing (log-routes.mjs): the receiver's routes over any field, with numeric comparisons; --only, --since and --until filters. - Delivery (timeline-writer.mjs, shared with the receiver): batches of at most max(64, moments / 8) entries, which the server applies in place (larger saves snapshot the whole timeline), waiting for each save, with retries; a quota check first, from an estimate that follows the server's patches and snapshots (within 1% in the PostgreSQL test). --dry-run, a progress bar, exit codes and --help. The receiver gains SYSLOG_TIMEZONE and ISO lines. The server skips changes identical to what it stores, and GET /api/timelines/usage reports an API key owner's storage and quota. Packaged as an npm bin, in the web image and the .deb (/usr/bin/och-import-logs), documented in docs/log-import.md, and tested by unit tests and a PostgreSQL test that imports a mixed .gz set twice. Closes #65 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011fFY8eruEi1xAkkMBni13B https://github.com/openchronology/timeline-app/commit/478e0dc98d6853b3d2e20f48443c50f245a5696f
Related
Open Import log files into timelines with och-import-logs in the timeline