feat(clips): async video clips with transcription [operator-action] (LC-496) #494

Merged

longjacksonle merged 6 commits from feat/LC-496-video-clips into main

2026-06-29 20:08:26 +02:00

longjacksonle commented

2026-06-29 20:02:41 +02:00

Owner

LC-496: Async video clips (Loom-style) with transcription [operator-action]

Record a short camera or screen clip in the browser, post it as a message, and have its audio auto-transcribed. Built to match the app's "browser captures, server only stores/tokenizes" posture: no server-side ffmpeg or native media deps.

What a user gets

A new "Record video clip" button in the composer opens a camera-or-screen chooser, records with MediaRecorder (5-minute cap) with a live preview, and on stop shows an inline playback preview with Re-record / Remove.
The clip posts as a normal attachment and renders inline as a <video controls> player in the timeline.
When STT is configured, the clip's audio is transcribed server-side and the transcript appears under the player, re-rendered live (same path + VoiceTranscribed event as voice messages).

How it fits the codebase (reuse-first)

Uploads (routes::uploads): new kind=clip path - sniff the container, canonicalize to a video/* mime, store content-addressed like any non-image upload. Dedicated 64 MiB clip cap (CLIP_MAX_BYTES); the per-user storage quota still applies. The serve route now honors single-range Range requests (206 + Accept-Ranges) so inline <video> seeking works (Safari MP4 needs it; WebM benefits too).
No schema change: the attachment loader already reads file_uploads.transcript for every attachment, so clips reuse the existing mime_type + transcript columns. Attachment::is_video() drives the new render branch.
Transcription (maybe_transcribe_voice_message, generalized): also transcribes video/* clips by forwarding the recorded container to the STT endpoint as-is. OpenAI-compatible engines demux the audio track themselves - hence no server ffmpeg. The STT filename mapping learns video containers (webm/mp4/mov).
Recorder JS (composer): a self-contained IIFE modeled on the LC-483 voice recorder; reuses the shared attachment slot (#lc-staged, hidden file_id, __lcClearAttach/__lcUploadBusy/__lcRefreshSend/__lcS), so clips are mutually exclusive with voice + file attachments and ride the normal form submit. Streams are released on stop and on htmx teardown. CSP already allows media-src 'self' blob: - no change needed.

Operator note

[operator-action]: when LETS_CHAT_STT_* is configured, recorded video clips are now ALSO sent to the STT endpoint (extra load/cost for metered or third-party engines). An engine that rejects video containers simply yields no clip transcript; the clip still posts and plays. Documented in the STT env row in CLAUDE.md.

Scope / follow-ups

v1 ships record + post + inline play + auto-transcribe (the core Loom value). LLM summary of a clip transcript is a natural additive follow-up (the existing summarize path targets call transcripts; pointing it at upload transcripts is a small wire-up). No camera/mic device picker for clips yet (uses defaults), matching the current voice-message behavior.

Tests

just check (clippy standalone + saas + desktop + fmt), just test, just test-saas all pass. New: parse_range + clip-mime-allowlist unit tests, and video_clip_is_transcribed_and_attached (a video/* upload with no waveform is transcribed and rendered as a clip).

Browser capture can't be exercised by the Rust tests. Worth a quick check in two browsers: record a camera clip and a screen clip, confirm inline playback + seeking, and (with an STT endpoint configured) confirm the transcript appears.

## LC-496: Async video clips (Loom-style) with transcription [operator-action] Record a short camera or screen clip in the browser, post it as a message, and have its audio auto-transcribed. Built to match the app's "browser captures, server only stores/tokenizes" posture: no server-side ffmpeg or native media deps. ### What a user gets - A new "Record video clip" button in the composer opens a camera-or-screen chooser, records with MediaRecorder (5-minute cap) with a live preview, and on stop shows an inline playback preview with Re-record / Remove. - The clip posts as a normal attachment and renders inline as a `<video controls>` player in the timeline. - When STT is configured, the clip's audio is transcribed server-side and the transcript appears under the player, re-rendered live (same path + `VoiceTranscribed` event as voice messages). ### How it fits the codebase (reuse-first) - **Uploads** (`routes::uploads`): new `kind=clip` path - sniff the container, canonicalize to a `video/*` mime, store content-addressed like any non-image upload. Dedicated 64 MiB clip cap (`CLIP_MAX_BYTES`); the per-user storage quota still applies. The serve route now honors single-range `Range` requests (206 + `Accept-Ranges`) so inline `<video>` seeking works (Safari MP4 needs it; WebM benefits too). - **No schema change:** the attachment loader already reads `file_uploads.transcript` for every attachment, so clips reuse the existing `mime_type` + `transcript` columns. `Attachment::is_video()` drives the new render branch. - **Transcription** (`maybe_transcribe_voice_message`, generalized): also transcribes `video/*` clips by forwarding the recorded container to the STT endpoint as-is. OpenAI-compatible engines demux the audio track themselves - hence no server ffmpeg. The STT filename mapping learns video containers (webm/mp4/mov). - **Recorder JS** (composer): a self-contained IIFE modeled on the LC-483 voice recorder; reuses the shared attachment slot (`#lc-staged`, hidden `file_id`, `__lcClearAttach`/`__lcUploadBusy`/`__lcRefreshSend`/`__lcS`), so clips are mutually exclusive with voice + file attachments and ride the normal form submit. Streams are released on stop and on htmx teardown. CSP already allows `media-src 'self' blob:` - no change needed. ### Operator note `[operator-action]`: when `LETS_CHAT_STT_*` is configured, recorded video clips are now ALSO sent to the STT endpoint (extra load/cost for metered or third-party engines). An engine that rejects video containers simply yields no clip transcript; the clip still posts and plays. Documented in the STT env row in CLAUDE.md. ### Scope / follow-ups v1 ships record + post + inline play + auto-transcribe (the core Loom value). LLM summary of a clip transcript is a natural additive follow-up (the existing summarize path targets call transcripts; pointing it at upload transcripts is a small wire-up). No camera/mic device picker for clips yet (uses defaults), matching the current voice-message behavior. ### Tests `just check` (clippy standalone + saas + desktop + fmt), `just test`, `just test-saas` all pass. New: `parse_range` + clip-mime-allowlist unit tests, and `video_clip_is_transcribed_and_attached` (a `video/*` upload with no waveform is transcribed and rendered as a clip). ### Manual smoke test (recommend before merge) Browser capture can't be exercised by the Rust tests. Worth a quick check in two browsers: record a camera clip and a screen clip, confirm inline playback + seeking, and (with an STT endpoint configured) confirm the transcript appears.

longjacksonle added 6 commits

2026-06-29 20:02:41 +02:00

feat(clips): accept video uploads + Range serving (LC-496) 3c76e299e3

The uploads endpoint gains a kind=clip path: video containers (webm/mp4/quicktime) are sniffed, canonicalized to a video/* mime, and stored content-addressed like any non-image upload, with a dedicated 64 MiB clip cap (the per-user quota still applies). The serve route now honors single-range Range requests with a 206 and advertises Accept-Ranges, so inline <video> seeking works (Safari MP4 requires it; WebM benefits too). parse_range + the clip allowlist are unit-tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(clips): inline <video> rendering for clip attachments (LC-496) 804e68654b

Attachment::is_video() drives a new branch in the attachment partial: a controls <video preload=metadata> player, with the server transcript rendered beneath it (same markup + WS re-render as voice messages).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(clips): transcribe video clips via the STT endpoint [operator-action] (LC-496) 68138e38b5

Generalize maybe_transcribe_voice_message: it now also transcribes video/* clips (a clip carries no waveform), forwarding the recorded container to the STT endpoint as-is. OpenAI-compatible engines demux the audio track themselves, so there is no server-side ffmpeg dependency. The STT filename mapping learns video containers (webm/mp4/mov) so name-sniffing engines route correctly. The same VoiceTranscribed event re-renders the clip with its transcript.

Operator-Action: when LETS_CHAT_STT_* is configured, recorded video clips are now also sent to the STT endpoint (extra load/cost for metered or third-party engines); an engine that rejects video containers simply yields no clip transcript and the clip still posts and plays.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(clips): in-composer camera/screen recorder (LC-496) 981364f5f5

A "Record video clip" composer button opens a camera-or-screen chooser, records the stream with MediaRecorder (5-minute cap, live preview), and on stop shows an inline playback preview with Re-record / Remove. The clip uploads through /api/upload (kind=clip) and stashes the file_id in the shared hidden input, so it rides the normal form submit and is mutually exclusive with the voice + file attachment slots. Releases the camera/screen stream on stop and on htmx teardown. JS strings wired through the __lcI18n map.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

i18n(clips): en + es strings for the clip recorder (LC-496) 592e5eedc3

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test+docs(clips): clip transcription coverage + STT env note (LC-496)

check-secrets / Kingfisher (push) Successful in 13s

Details

check-secrets / TruffleHog (push) Successful in 13s

Details

check-secrets / Nosey parker (push) Successful in 9s

Details

check-secrets / TruffleHog (pull_request) Successful in 11s

Details

check-secrets / Nosey parker (pull_request) Successful in 9s

Details

check-secrets / Kingfisher (pull_request) Successful in 12s

Details

Check / clippy + fmt + tests (pull_request) Successful in 3m8s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

7c045e95d5

Adds video_clip_is_transcribed_and_attached (a video/* upload with no waveform is transcribed and rendered as a clip) and documents in the STT env row that enabling STT now also forwards clip audio to the endpoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>