#65 Log file importer with pluggable formats and metadata mapping
–
- github
- github/openchronology/timeline-app
- github/openchronology/timeline-app#65
- issue
- state:closed
## Problem The syslog receiver (#64) handles live streams only. Importing historic logs by piping files into it gives wrong results: - **Modern timestamps aren't recognised.** rsyslog's ISO file format (`2026-10-09T08:15:01.123456+02:00 host app[pid]: …`) isn't parsed, so every line gets the import time. - **Classic lines get the wrong year.** `Mmm dd hh:mm:ss` lines take the year nearest to now (last December lands in the future) and are read as UTC instead of local time. - **Severity and facility are lost.** Files have no priority, so every line becomes `user.notice`. - **Large files lose data.** Replaying faster than saves complete overflows the 10,000-message queue, which drops the oldest. - **Re-importing duplicates.** IDs are assigned on receipt. Other common logs (Apache, nginx, application JSON logs) have no path in at all. ## Proposal Keep streaming as its own service, and add a separate import utility (for example `server/import-logs.mjs`, `npm run import:logs`). Move what they share into a common module: routing, moment building, and the batched API client with revision/409 handling. ```sh npm run import:logs -- --config import.json --format nginx-combined /var/log/nginx/access.log* npm run import:logs -- --format syslog --timezone +02:00 --dry-run /var/log/syslog.2.gz journalctl -o json --since yesterday | npm run import:logs -- --format journald - ``` ### Input - Plain files, `.gz` and stdin, read line by line in order. - Multi-line entries, such as stack traces, are joined with a continuation pattern. ### Formats Each format turns a line into named fields plus a time. Built-in formats: - `syslog`: the rsyslog ISO file format and the classic format. Classic lines get `--timezone` and a year at or before the file's modification time (or `--year`), stepping across New Year as lines are read. - `rfc5424`: lines with a priority; reuses `server/syslog.mjs`. - `apache-common`, `apache-combined`, `nginx-combined`: access logs. - `apache-error`, `nginx-error`: their level becomes severity. - `jsonl` with a field mapping, plus a `journald` preset for `journalctl -o json`. Custom formats in the config file: - a regex with named groups (or a JSON-lines mapping); - a time field and how to read it: ISO 8601, Apache `[10/Oct/2026:13:55:36 -0700]`, classic syslog, or epoch seconds/milliseconds; - the time zone and year rules. Times become exact Unix seconds, keeping fractions. ### Moments, including arbitrary metadata Timeline moments carry arbitrary JSON metadata, so the importer should treat the mapping into metadata as a first-class part of each format, not an afterthought. - **Title and notes:** templates over the fields, e.g. `"{method} {path} → {status}"`; the notes default to the whole line. - **Tags:** templates, e.g. `["status:{status_class}", "vhost:{vhost}"]`, sanitised to valid tags. - **Metadata:** a template that builds any JSON structure from the fields: - nested objects and arrays; - typed values (`int`, `number`, `bool`, `string`, `json` for embedded JSON); - constants; - dropping empty values. - The default stores every extracted field, typed by the format, under `metadata. `, as the receiver does under `metadata.syslog`. - The mapping can also fill keys that plugins read, such as sources, colors, icons or stack entries, so imported moments look right in the editor. - **Severity:** from a field, or from text rules (e.g. a message containing `error`) for logs without levels. - **Durations:** optionally, where a log records them. For example, an access log's request time can become a span from `time − request_time` to `time`, carrying the same metadata. ### Routing Generalise the receiver's routes to match any extracted field: - glob patterns; - numeric comparisons such as `status >= 500`; - severity thresholds; - first match wins, with `continue` and a default timeline. The receiver's host/app/severity rules become the syslog case. ### Delivery and safety - **No drops:** read, then……
Related
Open #65 Log file importer with pluggable formats and metadata mapping in the timeline