circagraph
Sign inSign up

openchronology/timeline-app on GitHub

The OpenChronology repository on GitHub, mirrored by och-import-github: issues and pull requests as durations (ongoing while open), what happened to them as moments, plus releases, milestones, commits and branches. The Branches view shows the most recently active branches each in a track of its own.

Source: https://github.com/openchronology/timeline-app

3,131 moments · 2026-10-05 CE 20:21:49 UTC – 2026-11-16 CE 12:10:45 UTC · by @athan · @athan

#65 Log file importer with pluggable formats and metadata mapping

–

  • github
  • github/openchronology/timeline-app
  • github/openchronology/timeline-app#65
  • issue
  • state:closed

## Problem The syslog receiver (#64) handles live streams only. Importing historic logs by piping files into it gives wrong results: - **Modern timestamps aren't recognised.** rsyslog's ISO file format (`2026-10-09T08:15:01.123456+02:00 host app[pid]: …`) isn't parsed, so every line gets the import time. - **Classic lines get the wrong year.** `Mmm dd hh:mm:ss` lines take the year nearest to now (last December lands in the future) and are read as UTC instead of local time. - **Severity and facility are lost.** Files have no priority, so every line becomes `user.notice`. - **Large files lose data.** Replaying faster than saves complete overflows the 10,000-message queue, which drops the oldest. - **Re-importing duplicates.** IDs are assigned on receipt. Other common logs (Apache, nginx, application JSON logs) have no path in at all. ## Proposal Keep streaming as its own service, and add a separate import utility (for example `server/import-logs.mjs`, `npm run import:logs`). Move what they share into a common module: routing, moment building, and the batched API client with revision/409 handling. ```sh npm run import:logs -- --config import.json --format nginx-combined /var/log/nginx/access.log* npm run import:logs -- --format syslog --timezone +02:00 --dry-run /var/log/syslog.2.gz journalctl -o json --since yesterday | npm run import:logs -- --format journald - ``` ### Input - Plain files, `.gz` and stdin, read line by line in order. - Multi-line entries, such as stack traces, are joined with a continuation pattern. ### Formats Each format turns a line into named fields plus a time. Built-in formats: - `syslog`: the rsyslog ISO file format and the classic format. Classic lines get `--timezone` and a year at or before the file's modification time (or `--year`), stepping across New Year as lines are read. - `rfc5424`: lines with a priority; reuses `server/syslog.mjs`. - `apache-common`, `apache-combined`, `nginx-combined`: access logs. - `apache-error`, `nginx-error`: their level becomes severity. - `jsonl` with a field mapping, plus a `journald` preset for `journalctl -o json`. Custom formats in the config file: - a regex with named groups (or a JSON-lines mapping); - a time field and how to read it: ISO 8601, Apache `[10/Oct/2026:13:55:36 -0700]`, classic syslog, or epoch seconds/milliseconds; - the time zone and year rules. Times become exact Unix seconds, keeping fractions. ### Moments, including arbitrary metadata Timeline moments carry arbitrary JSON metadata, so the importer should treat the mapping into metadata as a first-class part of each format, not an afterthought. - **Title and notes:** templates over the fields, e.g. `"{method} {path} → {status}"`; the notes default to the whole line. - **Tags:** templates, e.g. `["status:{status_class}", "vhost:{vhost}"]`, sanitised to valid tags. - **Metadata:** a template that builds any JSON structure from the fields: - nested objects and arrays; - typed values (`int`, `number`, `bool`, `string`, `json` for embedded JSON); - constants; - dropping empty values. - The default stores every extracted field, typed by the format, under `metadata. `, as the receiver does under `metadata.syslog`. - The mapping can also fill keys that plugins read, such as sources, colors, icons or stack entries, so imported moments look right in the editor. - **Severity:** from a field, or from text rules (e.g. a message containing `error`) for logs without levels. - **Durations:** optionally, where a log records them. For example, an access log's request time can become a span from `time − request_time` to `time`, carrying the same metadata. ### Routing Generalise the receiver's routes to match any extracted field: - glob patterns; - numeric comparisons such as `status >= 500`; - severity thresholds; - first match wins, with `continue` and a default timeline. The receiver's host/app/severity rules become the syslog case. ### Delivery and safety - **No drops:** read, then……

Related

Open #65 Log file importer with pluggable formats and metadata mapping in the timeline

#65 Log file importer with pluggable formats and metadata…