#367 Server package builds can hang for 90 minutes on a stalled mirror
–
- github
- github/openchronology/timeline-app
- github/openchronology/timeline-app#367
- issue
- state:closed
The CI run on the `v0.12.1` tag hung in **Server packages / Server / debian13 → Build the package** for over an hour (started 05:14 UTC, cancelled 06:27 UTC; run 40636197792). The same package built normally in the release workflow a few minutes earlier, so it's intermittent; the job's only limit is the 90-minute `timeout-minutes` on the whole job, so a hang holds a runner for 90 minutes before failing. The step runs `scripts/server-package.mjs debian13`, a Docker build of `deploy/debian-server/Dockerfile`, whose stages run `apt-get update && apt-get install` against Debian's mirrors with apt's default (no) retries and long timeouts, which is the likeliest place for a silent hang (as with the Ubuntu mirrors in #351). GitHub doesn't keep a log of a step that's cancelled, so this is not confirmed. **Proposed fix** - Give apt in every Dockerfile stage that installs packages (`deploy/debian-server`, `deploy/platform`, `deploy/Dockerfile.app`, `deploy/Dockerfile.postgres`) retries and a timeout: `Acquire::Retries "3"; Acquire::http::Timeout "30";`, and the same for dnf in `deploy/rpm-server` (`retries`, `timeout`). - Give the **Build the package** and **Install, run, upgrade and remove it** steps their own `timeout-minutes` (from what a normal build takes), so a hang fails in minutes and a rerun can follow. - Have the build print progress (Docker's `--progress=plain`), so the log shows where a hang happens. https://github.com/openchronology/timeline-app/issues/367
Related
- #351 CI: the WebKit job's Playwright install can hang past the 30-minute limit
- #368 Server package builds retry a stalled mirror and fail in minutes (#367)
- #369 Cache the Node.js tarball for the package builds
- #370 Cache the Node.js tarball for the package builds
- #571 Build Ubuntu server packages from the runners' own mirror
- The package builds take a cached Node.js tarball instead of downloading it in Docker
- The package builds' curl downloads retry and give up on a stall (#367)
- Server package builds retry a stalled mirror and fail in minutes (#367)
- Each curl attempt in the package builds ends after 3 minutes (#367)
- Merge pull request #368 from openchronology/fix/367-package-build-hangs
Open #367 Server package builds can hang for 90 minutes on a stalled mirror in the timeline