fix(kvm): gate DRM/KMS scanout capture off by default (VA-109) #150

Merged

longjacksonle merged 1 commit from fix/VA-109-gate-drm-scanout into main

2026-07-15 20:04:49 +02:00

longjacksonle commented

2026-07-15 20:00:03 +02:00

Owner

Addresses VA-109: a hard machine freeze (full seat lock, power-cycle to recover) seen during KVM desktop-tab testing on an amdgpu host, prime suspect the DRM/KMS scanout readback (VA-104).

Root cause (code-level)

The Wayland streaming capture wayland::capture_rgb (the desktop-tab tile source, 5-30 fps) tried DRM scanout first, on by default, with no opt-in. Per frame the DRM path (drm_kms::capture_rgb -> drm_gbm::read_scanout_rgb):

PRIME-exports the live primary scanout framebuffer;
gbm_create_device (fresh every frame), gbm_bo_import the live buffer, and gbm_bo_map(READ) - which has the driver blit the actively-scanned-out buffer out of VRAM - plus the same for the cursor plane;
tears it all back down.

Forcing a CPU-read map of a scanout buffer up to 30x/s, concurrent with active scanout, is a known GPU-hang / reset trigger on amdgpu (the host is AMD). A display-engine hang or failed reset presents as exactly this full-system freeze.

Caveat (honest): the link to this specific incident is circumstantial - the host that froze had latched DRM unavailable before the risky map that boot. But the path was default-on, ran the live-scanout blit every frame, and is unproven on the target GPUs, so it is a release-blocking freeze vector regardless. A hardware repro / GPU-log confirmation cannot be done in the CI sandbox (no GPU; deliberately wedging a display GPU is destructive) - it is left as an on-hardware step behind the opt-in.

Fix: opt-in, off by default

DRM/KMS scanout capture now engages only when VERVAIN_KVM_DRM_SCANOUT is truthy (1/true/yes/on), logging a one-time warning when enabled. Unset -> capture_rgb skips DRM and uses grim / xdg-portal, which never touch the raw scanout buffer.

Tradeoff (intentional, safety over capability): KWin/GNOME-Wayland hosts with no grim and no portal grant lose the consent-free capture path until an operator opts in on validated hardware.

Diagnostics

drm_gbm::map_bo logs (debug) immediately before and after the gbm_bo_map blit, so a post-freeze journal captured while validating the opt-in pinpoints whether it wedged inside the map (before-line present, after-line absent = froze in the blit).

Docs + tests

docs/va-109-drm-scanout-freeze.md: analysis, caveat, the gate, on-hardware validation steps, and follow-ups (per-frame gbm device churn; render-node import path). Unit test covers the opt-in parsing (off by default / off for any non-truthy value). Linux workspace clippy --deny warnings, fmt, and 94 kvm tests pass.

Acceptance criteria

DRM/gbm path root-caused at the code level and made safe (the freeze itself could not be hardware-reproduced in CI; documented as circumstantial with an on-hardware validation path).
DRM path gated off by default; opt-in per host after validation.
Repro steps and findings documented.

🤖 Generated with Claude Code

Addresses VA-109: a hard machine freeze (full seat lock, power-cycle to recover) seen during KVM desktop-tab testing on an amdgpu host, prime suspect the DRM/KMS scanout readback (VA-104). ## Root cause (code-level) The Wayland streaming capture `wayland::capture_rgb` (the desktop-tab tile source, 5-30 fps) tried DRM scanout **first, on by default, with no opt-in**. Per frame the DRM path (`drm_kms::capture_rgb` -> `drm_gbm::read_scanout_rgb`): - PRIME-exports the **live** primary scanout framebuffer; - `gbm_create_device` (fresh **every frame**), `gbm_bo_import` the live buffer, and `gbm_bo_map(READ)` - which has the driver **blit the actively-scanned-out buffer** out of VRAM - plus the same for the cursor plane; - tears it all back down. Forcing a CPU-read map of a scanout buffer up to 30x/s, concurrent with active scanout, is a known GPU-hang / reset trigger on amdgpu (the host is AMD). A display-engine hang or failed reset presents as exactly this full-system freeze. **Caveat (honest):** the link to this specific incident is circumstantial - the host that froze had latched DRM unavailable *before* the risky map that boot. But the path was default-on, ran the live-scanout blit every frame, and is unproven on the target GPUs, so it is a release-blocking freeze vector regardless. A hardware repro / GPU-log confirmation cannot be done in the CI sandbox (no GPU; deliberately wedging a display GPU is destructive) - it is left as an on-hardware step behind the opt-in. ## Fix: opt-in, off by default DRM/KMS scanout capture now engages only when `VERVAIN_KVM_DRM_SCANOUT` is truthy (`1`/`true`/`yes`/`on`), logging a one-time warning when enabled. Unset -> `capture_rgb` skips DRM and uses grim / xdg-portal, which never touch the raw scanout buffer. Tradeoff (intentional, safety over capability): KWin/GNOME-Wayland hosts with no grim and no portal grant lose the consent-free capture path until an operator opts in on validated hardware. ## Diagnostics `drm_gbm::map_bo` logs (debug) immediately before and after the `gbm_bo_map` blit, so a post-freeze journal captured while validating the opt-in pinpoints whether it wedged inside the map (before-line present, after-line absent = froze in the blit). ## Docs + tests `docs/va-109-drm-scanout-freeze.md`: analysis, caveat, the gate, on-hardware validation steps, and follow-ups (per-frame gbm device churn; render-node import path). Unit test covers the opt-in parsing (off by default / off for any non-truthy value). Linux workspace `clippy --deny warnings`, `fmt`, and 94 kvm tests pass. ## Acceptance criteria - [x] DRM/gbm path root-caused at the code level and made safe (the freeze itself could not be hardware-reproduced in CI; documented as circumstantial with an on-hardware validation path). - [x] DRM path gated off by default; opt-in per host after validation. - [x] Repro steps and findings documented. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

longjacksonle added 1 commit

2026-07-15 20:00:03 +02:00

fix(kvm): gate DRM/KMS scanout capture off by default (VA-109)

Check / fmt + clippy + tests (pull_request) Successful in 4m13s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

93b08bb4f3

A hard machine freeze (full seat lock, power-cycle to recover) was seen during KVM desktop-tab testing on an amdgpu host, with the DRM/KMS scanout readback (VA-104) as the prime suspect.

Root cause (code-level): the Wayland streaming capture (`wayland::capture_rgb`, the desktop-tab tile source at 5-30 fps) tried DRM scanout first, on by default with no opt-in. Per frame it PRIME-exports the live primary scanout framebuffer, creates a fresh gbm device, imports that live buffer, and `gbm_bo_map(READ)` has the driver blit the actively-scanned-out buffer out of VRAM - plus the same for the cursor plane - then tears it all down. Forcing a CPU-read map of a scanout buffer up to 30 times a second, concurrent with active scanout, is a known GPU-hang / reset trigger on amdgpu; a display-engine hang or failed reset presents as exactly this full-system freeze. The link to this specific incident is circumstantial (the host that froze had latched DRM unavailable before the map that boot), but the path was default-on, ran the risky live-scanout blit every frame, and is unproven on the target GPUs - a release-blocking freeze vector regardless.

Fix: make DRM/KMS scanout capture opt-in and OFF by default, engaged only when `VERVAIN_KVM_DRM_SCANOUT` is truthy (`1`/`true`/`yes`/`on`), with a one-time warning logged when enabled. With it unset, `capture_rgb` skips DRM and uses the grim / xdg-portal paths, which never touch the raw scanout buffer. The tradeoff (KWin/GNOME-Wayland hosts with no grim and no portal grant lose the consent-free path until an operator opts in on validated hardware) is intentional: safety over capability.

Diagnostics: `drm_gbm::map_bo` logs immediately before and after the `gbm_bo_map` blit (debug), so a post-freeze journal captured while validating the opt-in pinpoints whether it wedged inside the map.

Docs: `docs/va-109-drm-scanout-freeze.md` records the analysis, the circumstantial caveat, the gate, on-hardware validation steps, and follow-ups (per-frame gbm device churn; render-node import). A unit test covers the opt-in parsing (off by default / off for any non-truthy value). Linux workspace clippy --deny warnings, fmt, and 94 kvm tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYb1fkK32XQdwkGkStxXR9
Signed-off-by: longjacksonle <longjacksonle@gmail.com>