fix(kvm): route no-modifier DRM scanout through gbm, not raw mmap (VA-106) #110

Merged

David merged 1 commit from fix/VA-106-drm-scanout-modifier-none-tiled into main

2026-07-04 13:18:02 +02:00

David commented

2026-07-04 07:24:17 +02:00

Owner

Problem

With the VA-105 backpressure fix live (0.3.0-16-g16fa2bb), remote Wayland viewers now receive the keyframe over the data channel - but desktop-03 renders a horizontally-sheared / streaked image instead of the desktop, while desktop-02 renders correctly.

Root cause is the capture read-path selection, not the pixel math (convert_to_rgb is proven by desktop-02 + unit tests). is_linear(modifier) treated a missing modifier (None) the same as an explicit DrmModifier::Linear, so capture_rgb took the raw-mmap fast path for either. But drmModeGetFB2 only reports a modifier when the framebuffer was added with DRM_MODE_FB_MODIFIERS; many drivers scan out of an implicitly-tiled buffer and report None. Reading a tiled buffer row-by-row as if it were linear shears it into horizontal streaks. desktop-02 (AMD, explicit tiling modifier) already went through the gbm readback and rendered correctly; desktop-03 (no modifier reported) took the raw mmap and sheared.

Fix

Take the raw mmap (read_linear_rgb) only when the modifier is explicitly Linear (new is_explicit_linear). Route None and any tiled modifier through drm_gbm::read_scanout_rgb, which has the driver blit the tiled / VRAM scanout into a CPU-linear buffer. gbm handles genuinely-linear buffers too, so this is safe for the linear case.
Keep a raw-mmap fallback for a headless / VM host with no usable gbm whose scanout is nonetheless plain linear, surfacing the gbm error if that also fails.
Add log_scanout_once: every host emits its selected scanout width/height/fourcc/modifier/pitch/offset/explicit_linear once to the journal, so a distorted or black capture is diagnosable without a separate probe run.

Testing

just pre-commit (fmt + clippy --deny warnings + build + test --lib) green in Docker.
On-desktop retest needed: desktop-03 (and other Wayland hosts reporting no modifier) should render an undistorted desktop; desktop-02 must continue to render (no regression). The new DRM/KMS scanout selected ... journal line will confirm each host's actual modifier.

Not covered

desktop-01's black screen is the X11 greeter path (x11rb), not DRM - tracked separately.

Fixes VA-106.

## Problem With the VA-105 backpressure fix live (`0.3.0-16-g16fa2bb`), remote Wayland viewers now receive the keyframe over the data channel - but desktop-03 renders a **horizontally-sheared / streaked** image instead of the desktop, while desktop-02 renders correctly. Root cause is the capture read-path selection, not the pixel math (`convert_to_rgb` is proven by desktop-02 + unit tests). `is_linear(modifier)` treated a **missing** modifier (`None`) the same as an explicit `DrmModifier::Linear`, so `capture_rgb` took the raw-`mmap` fast path for either. But `drmModeGetFB2` only reports a modifier when the framebuffer was added with `DRM_MODE_FB_MODIFIERS`; many drivers scan out of an **implicitly-tiled** buffer and report `None`. Reading a tiled buffer row-by-row as if it were linear shears it into horizontal streaks. desktop-02 (AMD, explicit tiling modifier) already went through the gbm readback and rendered correctly; desktop-03 (no modifier reported) took the raw mmap and sheared. ## Fix - Take the raw `mmap` (`read_linear_rgb`) only when the modifier is **explicitly** `Linear` (new `is_explicit_linear`). Route `None` and any tiled modifier through `drm_gbm::read_scanout_rgb`, which has the driver blit the tiled / VRAM scanout into a CPU-linear buffer. gbm handles genuinely-linear buffers too, so this is safe for the linear case. - Keep a raw-`mmap` fallback for a headless / VM host with no usable gbm whose scanout is nonetheless plain linear, surfacing the gbm error if that also fails. - Add `log_scanout_once`: every host emits its selected scanout `width/height/fourcc/modifier/pitch/offset/explicit_linear` once to the journal, so a distorted or black capture is diagnosable without a separate probe run. ## Testing - `just pre-commit` (fmt + clippy `--deny warnings` + build + `test --lib`) green in Docker. - On-desktop retest needed: desktop-03 (and other Wayland hosts reporting no modifier) should render an undistorted desktop; desktop-02 must continue to render (no regression). The new `DRM/KMS scanout selected ...` journal line will confirm each host's actual modifier. ## Not covered desktop-01's black screen is the **X11 greeter** path (x11rb), not DRM - tracked separately. Fixes VA-106.

David added 1 commit

2026-07-04 07:24:17 +02:00

fix(kvm): route no-modifier DRM scanout through gbm, not raw mmap

Check / fmt + clippy + tests (pull_request) Successful in 13m8s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

6ad8677bf2

The Wayland DRM/KMS capture backend chose its read path from the scanout framebuffer's tiling modifier, and is_linear() treated a MISSING modifier (None) the same as an explicit DrmModifier::Linear, so capture_rgb took the raw-mmap fast path for either. But drmModeGetFB2 only reports a modifier when the framebuffer was added with DRM_MODE_FB_MODIFIERS; many drivers scan out of an implicitly-tiled buffer and report None. Reading such a buffer row-by-row as if it were linear shears the image into horizontal streaks. desktop-02 (AMD, explicit tiling modifier) went through the gbm readback and rendered correctly; desktop-03 (no modifier reported) took the raw mmap and rendered a sheared image.

Take the raw mmap only when the modifier is EXPLICITLY linear (new is_explicit_linear); route None and any tiled modifier through the gbm readback, which has the driver blit the tiled / VRAM scanout into a CPU-linear buffer (gbm handles genuinely-linear buffers too). Keep a raw-mmap fallback for a headless / VM host with no usable gbm whose scanout is plain linear, surfacing the gbm error if that also fails. Add log_scanout_once so every host emits its selected scanout width/height/fourcc/modifier/pitch/offset once to the journal, making a distorted or black capture diagnosable without a separate probe run.

#VA-106

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>