fix(ci): name the disk-exhaustion failure instead of crashing in the linker #500

Merged

longjacksonle merged 2 commits from fix/PMS-740-ci-disk-headroom into main 2026-08-07 02:40:50 +02:00

Conversation 0 Commits 2 Files changed 1 +52 -1

longjacksonle commented 2026-08-06 22:03:36 +02:00

Owner

Copy link

PMS-740. Makes the twice-seen linker crash describe itself, and captures the measurement that would have identified it the first time.

The failure this addresses

integration.yml has failed twice with every test binary dying at LINK time rather than any test failing (runs #3666 and #3719):

collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped

rust-lld writes its output through an mmap, so a filesystem that fills mid-write surfaces as SIGBUS inside the linker. It reads as a toolchain crash, and both occurrences were triaged by hand as a possible code defect before being ruled out.

Root cause is confirmed, and it is the host

Measured on the runner host while investigating this:

$ docker info --format '{{.DockerRootDir}}'   ->  /srv/d1/docker-daemon
$ df -h /srv/d1 /home
/dev/nvme0n1p4  875G  821G  9.8G  99% /srv/d1
/dev/nvme0n1p4  875G  821G  9.7G  99% /home

Docker's data root, every container volume, the runner's workspace and the developer's home directory are one 875G filesystem, currently under 10G free. 233G of that is reclaimable Docker volumes, almost entirely unused cargo target caches. Details and the reclamation caveat (a bare docker volume prune would also take dev-mokosh-postgres-data-*, which is real data) are on PMS-740.

Fixing the capacity is a host matter and is not what this PR does.

What this PR changes

Nothing about what is built or tested. Two additions to the job:

A precondition step fails the job when the workspace filesystem has under 25G free, naming runner capacity as the cause in one line rather than spending twenty minutes of compilation to reach a confusing crash. Verified against the current host state, where it correctly fires at 10G free. 25G is roughly twice the peak this suite's link stage has been seen to need; raise it if the suite grows rather than lowering it after a near miss.

A sampler records free space every 5 seconds for the life of the build and reports the low-water mark. This is the measurement both previous occurrences lacked: checking after a failed run always looked healthy (I measured ~130G free after run #3666) because the run's temporary output is released during cleanup, so the peak was never observed. When the build fails and that mark was under 5G, the log says so explicitly and points at the linker output.

Verification

The df/awk parsing and the integer comparison were run directly on the runner host; the guard evaluates correctly and fires at the current 10G. YAML parses. check-runner-labels, check-oci-build-cache, check-migration-prefixes and check-migration-immutability all pass.

Not run: the cargo gates. They need the very disk this PR is about, and the change is YAML-only with no Rust touched.

Note

The first CI run of this PR may well fail at the new guard rather than reaching the tests. That would be the guard working: the runner genuinely does not have the space today, and the point is that it now says so instead of crashing in the linker twenty minutes later.

PMS-740. Makes the twice-seen linker crash describe itself, and captures the measurement that would have identified it the first time. ## The failure this addresses `integration.yml` has failed twice with every test binary dying at LINK time rather than any test failing (runs #3666 and #3719): ``` collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped ``` `rust-lld` writes its output through an mmap, so a filesystem that fills mid-write surfaces as SIGBUS inside the linker. It reads as a toolchain crash, and both occurrences were triaged by hand as a possible code defect before being ruled out. ## Root cause is confirmed, and it is the host Measured on the runner host while investigating this: ``` $ docker info --format '{{.DockerRootDir}}' -> /srv/d1/docker-daemon $ df -h /srv/d1 /home /dev/nvme0n1p4 875G 821G 9.8G 99% /srv/d1 /dev/nvme0n1p4 875G 821G 9.7G 99% /home ``` Docker's data root, every container volume, the runner's workspace and the developer's home directory are one 875G filesystem, currently under 10G free. 233G of that is reclaimable Docker volumes, almost entirely unused cargo target caches. Details and the reclamation caveat (a bare `docker volume prune` would also take `dev-mokosh-postgres-data-*`, which is real data) are on PMS-740. Fixing the capacity is a host matter and is not what this PR does. ## What this PR changes Nothing about what is built or tested. Two additions to the job: **A precondition step** fails the job when the workspace filesystem has under 25G free, naming runner capacity as the cause in one line rather than spending twenty minutes of compilation to reach a confusing crash. Verified against the current host state, where it correctly fires at 10G free. 25G is roughly twice the peak this suite's link stage has been seen to need; raise it if the suite grows rather than lowering it after a near miss. **A sampler** records free space every 5 seconds for the life of the build and reports the low-water mark. This is the measurement both previous occurrences lacked: checking after a failed run always looked healthy (I measured ~130G free after run #3666) because the run's temporary output is released during cleanup, so the peak was never observed. When the build fails and that mark was under 5G, the log says so explicitly and points at the linker output. ## Verification The `df`/`awk` parsing and the integer comparison were run directly on the runner host; the guard evaluates correctly and fires at the current 10G. YAML parses. `check-runner-labels`, `check-oci-build-cache`, `check-migration-prefixes` and `check-migration-immutability` all pass. Not run: the cargo gates. They need the very disk this PR is about, and the change is YAML-only with no Rust touched. ## Note The first CI run of this PR may well fail at the new guard rather than reaching the tests. That would be the guard working: the runner genuinely does not have the space today, and the point is that it now says so instead of crashing in the linker twenty minutes later.

longjacksonle added 1 commit 2026-08-06 22:03:36 +02:00

fix(ci): name the disk-exhaustion failure instead of crashing in the linker ... a64de4455b

PMS-740. `integration.yml` has twice failed with every test binary dying at LINK time rather than any test failing (runs #3666 and #3719):

    collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped

`rust-lld` writes its output through an mmap, so the filesystem filling mid-write surfaces as SIGBUS inside the linker. It reads as a toolchain crash, and both occurrences were triaged as a possible code defect before being ruled out by hand.

Two changes, neither of which alters what is built or tested.

A precondition step fails the job when the workspace filesystem has under 25G free, naming the runner's capacity as the cause in one line rather than spending twenty minutes of compilation to reach a confusing crash. 25G is roughly twice the peak this suite's link stage has been seen to need; raise it if the suite grows rather than lowering it after a near miss.

A sampler records free space every 5 seconds for the life of the build and reports the lowest reading. That is the measurement the first two occurrences lacked: checking after a failed run always looked healthy, because the space is released when the run is cleaned up, so the peak was never observed. When the build fails and the low-water mark was under 5G, the log now says so explicitly and points at the linker output.

The underlying capacity problem is a host matter and is tracked in PMS-740; this only makes the symptom self-describing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011myibMMwyb6za3GVWJGkiX

longjacksonle added 1 commit 2026-08-06 22:09:31 +02:00

fix(ci): check disk headroom before restoring the cargo cache ... d2d836c156

The guard sat after the rust-cache restore, so a runner too full to link still spent time pulling down a multi-GB cargo cache before being told it had no room. Moving it directly after checkout fails first, skips that download, and leaves less on disk for whatever runs next.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011myibMMwyb6za3GVWJGkiX

longjacksonle merged commit 520f3334b6 into main 2026-08-07 02:40:50 +02:00

longjacksonle deleted branch fix/PMS-740-ci-disk-headroom 2026-08-07 02:40:51 +02:00

longjacksonle referenced this pull request from a commit 2026-08-07 02:40:52 +02:00

Merge pull request 'fix(ci): name the disk-exhaustion failure instead of crashing in the linker' (#500) from fix/PMS-740-ci-disk-headroom into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

psa-systems/mokosh-server!500

No description provided.