STT language + prompt hinting and real segment timestamps (LC-591) #576
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/lc591-stt-accuracy"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes LC-591, the transcription "Accuracy" child of LC-589. This is also the prerequisite you asked me to land before LC-593 (provider flexibility), since LC-593 normalizes every provider onto the
SttResultstruct this introduces.What was missing
The STT request sent only
model+file, and the parser read only{"text"}. So the engine autodetected language with no hint, an operator couldn't bias spelling of names/jargon, and WebVTT cue times were synthetic monotonic second-resolution guesses.The changes
SttClient::transcribenow returns a normalizedSttResult { text, segments }and takes alanguagehint.ReqwestSttClient:response_format=verbose_json, parsessegments[]{start,end,text}, and falls back to the plain{text}shape (empty segments) for engines that ignore it.parse_openai_resulthandles both and is unit-tested.languagefrom the speaker's/uploader's preferred locale when set, else omits it (autodetect). Rooms carry no locale, so the user's is the signal - and it's the accurate one, since the clip is that user's own mic.LETS_CHAT_STT_PROMPT.Real timestamps
One additive column,
transcript_segments.duration_ms(migration 0088): the clip's spoken span from the engine's segment timings.build_vttuses it for the cue length when > 0, and falls back to the pre-LC-591 synthetic length when 0 - which is the browser Web Speech path and any non-verbose_json engine - so both paths stay valid and non-overlapping.Scope note I want to be explicit about: live-caption granularity is unchanged (one caption per clip), and genuine sub-clip absolute placement is bounded by the capture model - the browser POSTs independent ~5s clips that the server stamps at second resolution. Real per-clip durations are a clear improvement over synthetic 2-3s guesses; finer sub-clip timing would require restructuring capture and is a deliberate non-goal here.
Tests
stt.rsunit tests: verbose_json parse (with trimming + duration), plain-json fallback, malformed-body-to-empty (matching the old behaviour), andLETS_CHAT_STT_PROMPTfrom env.duration_ms= 2500 -> a real00:00:00.000 --> 00:00:02.500VTT cue. Proven load-bearing by forcing the synthetic branch (the cue assertion then fails).MockSttClientgainedcanned_segments+ a::textconstructor; the four existing call sites migrated to::text.Full suite green (184 test binaries),
just checkclean. Provider-seam docs (LETS_CHAT_STT_PROMPT, verbose_json, language hint) added to.env.standalone,.env.saas, andREADME.md.Next
LC-593 lands on top:
LETS_CHAT_STT_PROVIDERselecting a request builder / response parser behindSttClient, with Deepgram normalized onto thisSttResult.🤖 Generated with Claude Code
https://claude.ai/code/session_01FhDVMViNHqUbwmfe2aXTv5
The transcription "Accuracy" child of LC-589, and the foundation LC-593 (provider flexibility) builds on. The STT request sent only `model` + `file` and the parser read only `{"text"}`, so language was left to autodetect and WebVTT cue times were synthetic monotonic second-resolution guesses. `SttClient::transcribe` now returns a normalized `SttResult { text, segments }` and takes a `language` hint. `ReqwestSttClient`: - requests `response_format=verbose_json` and parses `segments[]{start,end,text}` into the result, falling back to the plain `{text}` shape (empty segments) for engines that ignore it - `parse_openai_result` handles both and is unit-tested; - sends `language` when the speaker's/uploader's preferred locale is set (else omits it, letting the engine autodetect). Rooms carry no locale, so the user's locale is the signal, which is also the most accurate one - the clip is that user's own mic; - sends an optional operator glossary via `LETS_CHAT_STT_PROMPT` to bias spelling of names and jargon. Real timestamps reach the export via one additive column: `transcript_segments.duration_ms` (migration 0088), the clip's spoken span from the engine's segment timings. `build_vtt` uses it for the cue length when > 0, and falls back to the pre-LC-591 synthetic "until the next cue" length when 0 (the browser Web Speech path, or a non-verbose_json engine), so both paths stay valid and non-overlapping. Live-caption granularity is unchanged (one caption per clip); genuine sub-clip absolute placement is bounded by the clip-based capture model (independent ~5s clips the server stamps at second resolution) and is a deliberate non-goal here. `MockSttClient` gains `canned_segments` (and a `::text` constructor for the common no-segments case) so the real-timestamp path is testable. Tests: verbose_json parse, plain-json fallback, malformed-body-to-empty, and prompt-from-env as stt.rs unit tests; a `verbose_json engine -> stored duration_ms -> real VTT cue` integration test, proven load-bearing by forcing the synthetic branch. Full suite green (184 test binaries). Provider seam docs (`LETS_CHAT_STT_PROMPT`) in `.env.standalone`, `.env.saas`, and `README.md`. Unblocks LC-593: providers now normalize onto `SttResult`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhDVMViNHqUbwmfe2aXTv5