VA-11: Windows KVM via DXGI Desktop Duplication + SendInput #57

Merged

nrupard merged 3 commits from feat/VA-11-windows-kvm into main

2026-06-13 00:33:02 +02:00

nrupard commented

2026-06-13 00:02:09 +02:00

Owner

Summary

Phase-6 Windows parity for KVM (VA-11). Adds a Windows backend to vervain-agent-kvm behind cfg(target_os = "windows"), reusing the existing tile-diff streaming pipeline rather than forking it.

What changed

Capture - DXGI Desktop Duplication (src/windows/capture.rs): a D3D11 device duplicates the primary output; each frame is copied to a CPU-readable staging texture and read back as packed BGRA, which maps directly onto the pipeline's PixelLayout::Bgrx. Reacquires transparently on DXGI_ERROR_ACCESS_LOST, falls back to the WARP software device when there's no GPU, and persists the duplication across frames. stream.rs::capture_raw_frame and the one-shot capture_primary/capture_display get Windows arms.
Input - SendInput (src/windows/input.rs): absolute mouse moves (normalized over the primary monitor), buttons + wheel, and key_event mapping the SPA's X11 keysym names to virtual keys (named control keys) or injecting printable characters as Unicode (KEYEVENTF_UNICODE).
Multi-display (src/windows/display.rs): EnumDisplayMonitors + GetMonitorInfoW, same DisplayInfo shape as the Linux xrandr path (AC#3).
Active-session helper (src/windows/session.rs): WTSQueryUserToken + CreateProcessAsUserW to run KVM in the console user's session, because a Session 0 service has no interactive desktop. Detection + spawn primitive land here; the service<->helper frame/input relay is the agent-side follow-up.
Pipeline refactor: the tile-diff + JPEG encode in stream.rs is now platform-agnostic (capture backend hands in a RawFrame), and jpeg-encoder moved to a cross-platform dependency.

The existing run_desktop tunnel calls capture_changed_tiles / mouse_move / key_event, so interactive-session Windows KVM works with no agent change.

Decisions

windows crate (not windows-sys) since DXGI/D3D11 are COM; active-session helper (chosen over in-process-only) for Session 0; DXGI-only (Win10/11 both support Desktop Duplication).

Validation

Typecheck-only interim (agreed): oci-build/check.Dockerfile passes - fmt, clippy --deny warnings, build, the full test suite, and the Windows cross-check compiling the DXGI/SendInput/WTS backend. The runtime ACs (capture on Win10/11, input in both session contexts, multi-display parity) are validated on a real Windows host.

Follow-up

Session-0 service<->helper frame/input relay (IPC); per-output DXGI capture selection by name.

Refs VA-11.

## Summary Phase-6 Windows parity for KVM (VA-11). Adds a Windows backend to `vervain-agent-kvm` behind `cfg(target_os = "windows")`, reusing the existing tile-diff streaming pipeline rather than forking it. ## What changed - **Capture - DXGI Desktop Duplication** (`src/windows/capture.rs`): a D3D11 device duplicates the primary output; each frame is copied to a CPU-readable staging texture and read back as packed BGRA, which maps directly onto the pipeline's `PixelLayout::Bgrx`. Reacquires transparently on `DXGI_ERROR_ACCESS_LOST`, falls back to the WARP software device when there's no GPU, and persists the duplication across frames. `stream.rs::capture_raw_frame` and the one-shot `capture_primary`/`capture_display` get Windows arms. - **Input - SendInput** (`src/windows/input.rs`): absolute mouse moves (normalized over the primary monitor), buttons + wheel, and `key_event` mapping the SPA's X11 keysym names to virtual keys (named control keys) or injecting printable characters as Unicode (`KEYEVENTF_UNICODE`). - **Multi-display** (`src/windows/display.rs`): `EnumDisplayMonitors` + `GetMonitorInfoW`, same `DisplayInfo` shape as the Linux xrandr path (AC#3). - **Active-session helper** (`src/windows/session.rs`): `WTSQueryUserToken` + `CreateProcessAsUserW` to run KVM in the console user's session, because a Session 0 service has no interactive desktop. Detection + spawn primitive land here; the service<->helper frame/input relay is the agent-side follow-up. - **Pipeline refactor**: the tile-diff + JPEG encode in `stream.rs` is now platform-agnostic (capture backend hands in a `RawFrame`), and `jpeg-encoder` moved to a cross-platform dependency. The existing `run_desktop` tunnel calls `capture_changed_tiles` / `mouse_move` / `key_event`, so interactive-session Windows KVM works with no agent change. ## Decisions `windows` crate (not `windows-sys`) since DXGI/D3D11 are COM; active-session helper (chosen over in-process-only) for Session 0; DXGI-only (Win10/11 both support Desktop Duplication). ## Validation Typecheck-only interim (agreed): `oci-build/check.Dockerfile` passes - fmt, clippy --deny warnings, build, the full test suite, and the Windows cross-check compiling the DXGI/SendInput/WTS backend. The runtime ACs (capture on Win10/11, input in both session contexts, multi-display parity) are validated on a real Windows host. ## Follow-up Session-0 service<->helper frame/input relay (IPC); per-output DXGI capture selection by name. Refs VA-11.

nrupard added 2 commits

2026-06-13 00:02:09 +02:00

feat(kvm): Windows KVM via DXGI Desktop Duplication + SendInput (VA-11) 181b3bdea9

Phase-6 Windows parity for the KVM stack. Adds a Windows backend to `vervain-agent-kvm` behind `cfg(target_os = "windows")`, reusing the existing tile-diff streaming pipeline rather than forking it.

- Capture: DXGI Desktop Duplication (`src/windows/capture.rs`). A D3D11 device duplicates the primary output; each frame is copied to a CPU-readable staging texture and read back as packed BGRA, which maps directly onto the pipeline's `PixelLayout::Bgrx`. The duplication is reacquired transparently on `DXGI_ERROR_ACCESS_LOST`, falls back to the WARP software device when there's no GPU, and persists across frames via a thread-local on the blocking pool. The streaming path (`stream.rs::capture_raw_frame`) and the one-shot `capture_primary`/`capture_display` now have Windows arms.
- Input: `SendInput` (`src/windows/input.rs`). Absolute mouse moves normalized over the primary monitor, buttons + wheel, and `key_event` mapping the SPA's X11 keysym names to virtual keys (named control keys) or injecting printable characters as Unicode code points (`KEYEVENTF_UNICODE`).
- Multi-display: `EnumDisplayMonitors` + `GetMonitorInfoW` (`src/windows/display.rs`) producing the same `DisplayInfo` shape as the Linux xrandr path (AC#3).
- Active-session helper (`src/windows/session.rs`): `WTSQueryUserToken` + `CreateProcessAsUserW` to launch a helper in the console user's session, because a Session 0 service has no interactive desktop to capture or inject into. Detection (`in_session_0`) + spawn primitive are here; the service<->helper frame/input relay is the agent-side follow-up.
- Pipeline refactor: the tile-diff + JPEG encode in `stream.rs` is now platform-agnostic (the capture backend hands in a `RawFrame`), and `jpeg-encoder` moved to a cross-platform dependency.

The existing `run_desktop` tunnel uses `capture_changed_tiles` / `mouse_move` / `key_event`, so the interactive-session KVM works on Windows with no agent change; the Session-0 relay is the remaining integration.

Bindings: the `windows` crate (not `windows-sys`) because DXGI/D3D11 are COM interfaces; it cross-compiles clean for `x86_64-pc-windows-gnu`.

Validation: typecheck-only interim (agreed) - `oci-build/check.Dockerfile` passes (fmt, clippy --deny warnings, build, tests, and the Windows cross-check compiling the DXGI/SendInput backend). The runtime ACs (capture on Win10/11, input in both session contexts, multi-display parity) are validated on a real Windows host.

#VA-11

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs(roadmap): mark VA-11 Windows KVM implemented

Check / clippy + fmt + tests (pull_request) Successful in 1m54s

Details

960fb51c19

#VA-11

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

nrupard referenced this pull request from a commit

2026-06-13 00:15:23 +02:00

fix(kvm): address VA-11 Windows review findings

nrupard added 1 commit

2026-06-13 00:15:23 +02:00

fix(kvm): address VA-11 Windows review findings

Check / clippy + fmt + tests (pull_request) Successful in 2m34s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

f736d37f70

Follow-up to the code review on PR #57.

- DXGI single-duplication (the blocker): capture used a per-thread (thread_local) Duplicator, but DXGI allows only one duplication of an output per process and a duplication has thread affinity. Run on tokio's blocking pool, a second pool thread's DuplicateOutput would fail with DXGI_ERROR_NOT_CURRENTLY_AVAILABLE and never recover, so streaming capture broke as soon as the task migrated threads. Replaced with a single dedicated capture thread that owns the one Duplicator; callers send a request and block on the reply. The thread also caches the last frame so a one-shot screenshot on an idle desktop still returns an image (streaming passes want_cached=false to skip idle frames).
- HDR/10-bit guard: warn once when the duplicated output is not 8-bit BGRA (e.g. R10G10B10A2), since the BGRA->RGB swizzle would otherwise mis-render the colors (the row copy stays in-bounds either way). Full HDR support is a follow-up.
- Active-session helper: dropped CREATE_UNICODE_ENVIRONMENT, which is meaningless without an explicit environment block; documented that building the user's block via CreateEnvironmentBlock is a follow-up.
- Tidied the fully-qualified MOUSE_EVENT_FLAGS into an import.

Verified via `oci-build/check.Dockerfile`: fmt, clippy --deny warnings, build, full test suite, and the Windows cross-check compiling the reworked DXGI/SendInput backend.

#VA-11

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

nrupard merged commit 0cd08d7490 into main

2026-06-13 00:33:02 +02:00

nrupard deleted branch feat/VA-11-windows-kvm

2026-06-13 00:33:02 +02:00

nrupard referenced this pull request from a commit

2026-06-13 00:33:04 +02:00

Merge pull request 'VA-11: Windows KVM via DXGI Desktop Duplication + SendInput' (#57) from feat/VA-11-windows-kvm into main