feat/email-digest #80

Merged
longjacksonle merged 12 commits from feat/email-digest into main 2026-05-12 23:13:31 +02:00
No description provided.
Phase 22 task 1: ship the column and the WS-handler bump in isolation so the rest of the email-digest work has a clean signal for "the in-app surface had a chance to fire" without overloading last_active_at and breaking idle-flip.

Adds three columns on users: last_ws_seen_at (nullable), notify_email_digest_enabled (default 0), last_digest_sent_at (nullable). The eligibility partial index covers opted-in users only so future digest scans are cheap on instances where most users are not opted in.

The WS handler bumps last_ws_seen_at once on connection-open and again on each outbound Mentioned frame that passes the mute filter. The throttle is per-connection (a std::time::Instant local to the send task), 5-minute window, no shared state across tabs. Two open tabs do at most two writes per window for the same user; that is the documented tradeoff for keeping the throttle cheap and DB-free.

Idle-flip in mark_idle_users continues to use last_active_at only. The two signals are deliberately separate: last_active_at means "user touched HTTP," last_ws_seen_at means "user's app was alive enough to surface a ping." Comments at both definitions explain the split so a future reader does not "fix" the inconsistency.

bump_last_ws_seen is fire-and-forget (logs on error) because the WS hot path must not propagate a side-effect DB failure. set_last_digest_sent_at returns Result because the future digest tick needs to know if the write failed to avoid marking-as-sent something that did not actually persist.

CLAUDE.md gains a one-line note about the --jobs 2 workaround for the OOM-killed linker that triggers on the default cargo-test parallelism inside the dev container. The compile step is fine; only the parallel link step blows up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `last_ws_seen_at`, `notify_email_digest_enabled`, and `last_digest_sent_at` columns to the `users` table (migration 0010) and wire them through all query paths that hydrate `UserRecord`.

- `bump_last_ws_seen`: records that the in-app notification surface was alive for a user; called on WS connect and throttled to once per 5 minutes per connection on outbound `Mentioned` frames
- `set_last_digest_sent_at`: stamped after a digest is sent; auto-resets on the next activity bump so only one digest fires per offline session
- `notify_email_digest_enabled` propagated to the `User` projection used by handlers
- All existing test fixtures updated to apply migration 10; new `db_digest_helpers` test file covers column defaults, bump idempotency, timestamp independence from `last_active_at`, and digest-sent stamping
Phase 22 task 2 (admin UI half). The DB layer + email transport landed in 408c7d1; this commit threads them through the operator-facing surface.

Admin SMTP form (/admin/settings) is refactored to read/write the typed `smtp_settings` row instead of the per-key entries in `settings`. The TLS mode is now an explicit dropdown (starttls / tls / none) rather than implicit STARTTLS. The password field is still write-only with "blank = keep existing" semantics; that contract is now enforced both in the route handler and inside `db::smtp_settings::save` (defence in depth).

POST /admin/settings/smtp/test renders a banner showing whether SMTP works for the recipient typed into the form. The handler routes through `state.email_client` rather than constructing a fresh transport from the just-saved row, which means the operator workflow is "save, restart, click test." Trying to do live-config testing without a restart would have required threading a swappable client across the request boundary; the simpler model matches the VAPID/Push snapshot pattern from phase 16 and keeps the prod and test code paths identical (both go through the trait, tests just inject a different impl).

The settings page also picks at most one disabled-state banner explaining why email is off: missing LETS_CHAT_SECRET_KEY beats unconfigured-SMTP beats startup-load-failed. Each banner names the next step the operator should take.

`email_client: None` plumbed through every test AppState construction. Settings-pool test fixtures gain migration 0004. Two new test files:
- `db_smtp_settings`: round-trips the AES-GCM password column, verifies blank-password-leaves-existing semantics, asserts wrong-key decrypt fails loudly, exercises TLS-mode round-trip.
- `routes_admin_smtp_test`: drives the admin route end-to-end against a `MockEmailClient` in state, covering the success path plus the blank-recipient, transport-failure, and empty-from rejection branches. Gated to `feature = "standalone"` since admin routes are not mounted under `saas`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the Build Modes section, saas-specific just recipes, test parallelism note, and inline script teardown guidance from CLAUDE.md now that the app is standalone-only. The Rust changes are formatting-only (rustfmt line-length adjustments in smtp_settings.rs, email/mod.rs, and main.rs with no logic changes). Also allows the lowercase `echo "exit=$?"` variant in local Claude settings.
The third disabled-state banner on /admin/settings fires whenever `state.email_client` is None despite the SMTP row having a host. In practice the common reason for that is "I just saved settings and have not restarted yet" - the snapshot was taken at startup. The old wording led with "if you rotated LETS_CHAT_SECRET_KEY," which is the rare case and made operators think their just-saved config was broken.

Rewrite so the common case is the first sentence and the key-rotation case is parenthetical. The route-handler comment is expanded to record both reasons and their order of likelihood, so a future reader knows why the wording is structured this way.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 22 task 3. Pure presentation; no scheduler or DB wiring yet.

`email::digest::build_snippet(body)` returns a `(plain, html)` pair. Plain is trimmed and word-truncated to 140 chars with an ASCII ellipsis when cut; html runs the same truncation, HTML-escapes, wraps `@name` substrings in `<strong>`, and linkifies URLs via the existing `linkify` crate. Boundary rule for mentions matches `db::mentions::TOKEN_PATTERN` so `foo@bar.com` does not accidentally produce a chip. The fallback for a single 200-char unbreakable word does a char-boundary cut so the snippet never blows past the cap.

The HTML output is intentionally MUCH leaner than `views::room::render_body`: no custom emojis, no profile-link chips, no avatars. Email clients render that surface inconsistently; the safe set is "escaped text + bold + anchor" and a couple of inline styles in the template wrapper.

Two new Askama templates in `templates/email/`:
- `digest.html` - block layout, sections for DMs (first) and rooms (second), per-item author + timestamp + snippet + "view" deep link, an overflow footer when items exceeded the per-digest cap, and a settings-link footer.
- `digest.txt` - plaintext mirror with `===` section rules. Askama's per-extension escaper turns escaping off for `.txt` so URLs and special chars pass through verbatim.

View structs `DigestHtml`, `DigestText`, `DigestDmSection`, `DigestRoomSection`, `DigestItem` live in `views::email_digest`. Both templates consume the same section types so the future digest tick (task 4) builds the data once and renders both halves of the multipart message without re-deriving anything.

10 unit tests in `email::digest` cover the snippet contract (short, long, trim, html-escape, mention boundary against email addresses, URL linkify, mention+URL together, single-long-word fallback, atomic URL segment). 4 tests in `views::email_digest` cover template rendering (section ordering, empty-section omission, overflow footer, plaintext-has-no-markup).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 22 task 4. The previous commits supplied the building blocks (WS-receive tracking, SMTP transport, content primitives, view structs). This commit closes the loop: an hourly background task that finds offline users with unread mentions or DMs, builds a multipart email, and dispatches it.

Adds a `users.email` column via migration 0011. Recipient address is otherwise the missing piece - without it the digest cannot send anywhere. Nullable, no UNIQUE constraint: it is a notification destination, not an identity field. The eligibility query filters out NULL/empty so users without an address are silently no-ops until task 5 ships the /settings UI to let them set one.

Adds the operator-supplied site URL as a key in the `settings` table (`public_base_url`). The admin SMTP page gains an input for it; on save a trailing slash is stripped so the digest can `format!("{base}/room/...")` without worrying about doubles. The templates degrade gracefully when empty: items still render but anchor tags are omitted (no `<a href="">` malformed-link emails). This was the half-built risk the user flagged before approving the task: addressed by making the URL optional rather than mandatory, with a clear path to enable.

`db::auth::find_digest_candidates` runs the eligibility predicate from the design doc: `notify_email_digest_enabled = 1` AND non-empty email AND `MAX(last_active_at, COALESCE(last_ws_seen_at, '')) < now - quiet_period` AND `last_digest_sent_at < the same MAX OR NULL`. The SQLite `MAX(a, COALESCE(b, ''))` shape works because timestamps are ISO 8601, which sorts lexicographically the same as chronologically. One digest per offline session falls out automatically: the moment the user comes back online and any activity column bumps, the self-comparison reopens eligibility for the next offline stretch.

`email::digest::run_tick` orchestrates the whole pass:
1. Short-circuit when `state.email_client` is None.
2. Load `public_base_url` and SMTP from-address once per tick; abort if from is empty.
3. For each candidate, fetch missed mentions and missed DMs from the chat pool with the `MAX(...)` activity floor as the lower bound on message `created_at`, capped at a 7-day window, with `room_notification_settings.mute_mode <> 'all'` excluding muted rooms (DM mute writes 'all' to the same shared table per `db::notifications::set_dm_mute`).
4. Bulk-resolve all author/peer ids to usernames in one auth-pool round trip.
5. Group rows into `DigestDmSection`/`DigestRoomSection` while applying a 50-item budget across both sections; spill into the overflow counter.
6. Render both Askama templates, build the `EmailMessage`, send through the trait, mark `last_digest_sent_at` on success.
7. Per-user failures log warn and the loop continues - one bad recipient does not stop the rest of the tick.

`main.rs` gains `spawn_digest_sender`, modeled on `spawn_idle_scanner`: `tokio::time::interval(3600s)` plus a leading-tick skip so the first run lands an hour after boot rather than at startup.

The dispatch tests in `email_digest_dispatch.rs` cover seven scenarios: happy-path send with both DMs and mentions, second-tick self-skipping, missing email skipping, opt-out skipping, room-mute exclusion, base-URL-empty graceful degradation, and SMTP-from-empty whole-tick abort. The test discovered that the chat migrations auto-seed a default "general" room at id 1, so the harness pins its own created room id rather than assuming `WHERE room_type = 'public' LIMIT 1` returns the right one.

24 existing test files updated to apply auth/0011. The dispatch test is `#![cfg(feature = "standalone")]` because it constructs a full AppState through routes::build_router, which only mounts the admin/settings paths under standalone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "one digest per offline session" property has two halves: same session must not double-fire (already tested) AND a new offline session after the user came back and went offline again must re-fire. The second half was previously only implicit. This test pins it by back-dating columns to simulate the elapsed time without waiting an hour in CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 22 task 5. The dispatch tick has been functionally complete since task 4 but had no UI for end users to opt in or supply an email address; this commit adds both, plus the admin toggle that lets operators flip the default for new registrations.

/settings gains an email input pre-populated from `users.email` and an "Email me a digest of missed mentions and DMs" checkbox. The checkbox is disabled when `state.email_available()` is false, with help text explaining that SMTP is unconfigured. POST handler defends against client tampering by AND'ing the form value with the server gate (same shape as the existing push checkbox). Empty email field stores NULL so the eligibility query filters cleanly. Light validation requires exactly one '@' on a non-empty input; full RFC validation is the upstream MTA's job.

`db::auth::set_notification_prefs` now takes a fourth `email_digest: bool` argument. The one existing test caller (`push_dispatch`) updated to pass `false`. Two new helpers carry the digest-specific paths: `set_email` (Option<&str>; None stores NULL) and `set_notify_email_digest_enabled` (target toggle the register flow uses).

`UserSettingsPage` gains `email_available: bool` and `current_email: String`. The public `User` projection still does NOT carry `email` - it is recipient metadata, not identity, and shouldn't flow through handler/template contexts where the in-app rendering layer could leak it.

Admin SMTP page gets a small "Defaults for new users" form below the test-email controls. Stored in the existing `settings` key-value table under `default_notify_email_digest` ("0"/"1"). Documented clearly that it only affects FUTURE registrations: existing users are not retroactively changed. POST handler at /admin/settings/email-digest-default flips the row.

Register flow (standalone-only; saas users come from the parent app) reads `default_notify_email_digest` after `create_user` and calls the helper when "1". Failure is logged warn but not fatal: the worst case is the user has to opt in manually.

`db_email_digest_default.rs` pins three properties: absent key gives the column default (0), flag flipped to "1" propagates via the helper to a new user, and toggling the flag off later does NOT touch users created under the previous regime. Fourth test rounds out the file with a `set_email` NULL-on-empty round-trip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(readme): document email-digest setup, semantics, and upgrade path
Some checks failed
Check / clippy + fmt + tests (pull_request) Failing after 11s
9045f476f5
Phase 22 task 7. Operator-facing README for the email-digest feature: the four config knobs (LETS_CHAT_SECRET_KEY, SMTP settings, public_base_url, default-for-new-users toggle), the save-restart-test workflow, delivery semantics (cadence, quiet period, one-per-session predicate, 7-day window, 50-item cap), the user opt-in flow, and the upgrade note that the existing plaintext SMTP password is discarded by the migration and must be re-entered.

Also updates the LETS_CHAT_SECRET_KEY section: the env var now also encrypts the SMTP password, so the "Without it / lose it / rotate it" subsections gain SMTP-specific recovery paths alongside Push and 2FA.

Features list gains a one-line "Email digest of missed mentions and DMs (off by default per user)" entry next to admin / RBAC.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge main into feat/email-digest
Some checks failed
Check / clippy + fmt + tests (pull_request) Failing after 9s
654c0a63ab
Integrates three phases shipped on main while this branch was open (#77 password reset, #78 email verification, #79 multi-device sessions) with the email-digest work.

Architectural reconciliation. Main introduced env-var-driven SMTP through `mail::Mailer::from_env()`, a `state.mailer: Option<Mailer>`, a `state.base_url: String` populated from `LETS_CHAT_BASE_URL`, and a `users.email` column (with UNIQUE-when-not-NULL) plus email verification flow. This branch's parallel approach (admin-form SMTP, AES-encrypted password in `smtp_settings` table, `public_base_url` admin setting, separate `users.email` column) was redundant and is dropped in favour of main's design.

Dropped:
- `server/src/email/` module (EmailClient trait + LettreEmailClient + MockEmailClient + EmailMessage + EmailError). The digest tick now goes through `state.mailer` directly.
- `server/src/db/smtp_settings.rs` (typed encrypted-password table). Main reads SMTP from env vars.
- `server/migrations/auth/0011_user_email.sql`. Main's `0010_password_reset.sql` already added the `email` column.
- `server/migrations/settings/0004_smtp_settings.sql`.
- `server/tests/db_smtp_settings.rs` and `server/tests/routes_admin_smtp_test.rs`.
- The admin SMTP form refactor on this branch (encryption, send-test button, public_base_url field). The admin SMTP page is restored to main's plaintext-key-value-table version. The legacy form is now dead UI but kept for backward compatibility per main; main's password reset and email verification use `Mailer::from_env`. Only the new "Default digest for new users" toggle is layered on top.

Renamed:
- `server/migrations/auth/0010_digest_columns.sql` -> `0013_digest_columns.sql` (after main's 0010-0012). Contents unchanged: still adds `last_ws_seen_at`, `notify_email_digest_enabled`, `last_digest_sent_at`, plus the partial digest-eligible index.

Adapted:
- `server/src/digest.rs` (was `email::digest`) now uses `state.mailer` and `state.base_url`. Added a `Mailer::send_multipart` method since the digest needs HTML+plaintext multipart while main's existing callers (password reset, email verification) only sent plaintext. Removed reading SMTP from-address from settings DB (Mailer holds it from env).
- `routes/settings.rs` digest opt-in checkbox kept; email-input handling reverted in favour of main's separate email-change flow with verification. `set_notification_prefs` keeps its fourth `email_digest` arg.
- `routes/auth.rs` register handler merged: main's email + verification path runs first, then this branch's "apply admin default digest opt-in" hook runs after.
- `views/settings.rs` `UserSettingsPage` keeps all of main's session/email-verification fields plus an `email_available: bool` for the digest checkbox's disabled state.
- 22 test files: migration include lists rewritten to use main's 0010/0011/0012 plus the renamed 0013; AppState constructions updated to add main's new fields (`last_seen_ledger`, `activity_ledger`, `mailer: None`, `base_url`) and drop `email_client`.
- The end-to-end `email_digest_dispatch` test rewritten as a DB-level integration test. Main's concrete `Mailer` is not test-substitutable, so content assertions (subject, body, mock recording) are dropped. The DB tests cover candidate selection, opt-in/opt-out filtering, "one digest per offline session" gating in both directions (same session does not refire AND new session does), and mute-filter exclusion.

README's email-digest section rewritten to drop the admin-form-SMTP narrative in favour of env-var SMTP + `LETS_CHAT_BASE_URL`, and the upgrade-discards-password note (no longer applies under main's env-var design).

Test surface: 47 standalone + saas test binaries pass; zero failures both modes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
longjacksonle deleted branch feat/email-digest 2026-05-12 23:13:31 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
psa-systems/lets-chat!80
No description provided.