feat(email-ingress): IMAP poll + resolution + synthetic-actor send (LC-77) #196

Merged

longjacksonle merged 6 commits from feat/lc-77-imap-poll into main 2026-05-25 19:24:59 +02:00

Conversation 0 Commits 6 Files changed 28 +2347 -177

longjacksonle commented 2026-05-25 19:16:15 +02:00

Owner

Copy link

Summary

The runtime body for LC-77 email-ingress. Builds on PR #194 (MessageActor refactor) and PR #195 (schema). Adds the third synthetic-actor variant end-to-end: an operator-configured IMAP mailbox is polled every 5 minutes; mail addressed to <token>@<ingress-domain> posts to its room as the MessageActor::EmailInbox actor.

Six commits, each independently reviewable:

1. refactor(messages): add EmailInbox actor variant + AuthorMeta.actor — behavior-preserving plumbing for the third actor type. Webhook fixtures stay byte-identical; two new EmailInbox fixtures pin the new arm. resolve_msg_author now also takes email_inbox_id; every caller updated.
2. feat(email-ingress): db layer + rate-limit kind — db::email_inbox extended with insert/find_by_secret_hash/list_for_room/revoke/touch_last_used. db::imap_config (new) reads + writes the singleton sealed-creds row via crypto::seal/open. db::chat::insert_email_inbox_message mirrors insert_webhook_message. RateLimitKind::EmailInbox added.
3. feat(email-ingress): email_ingress module core + finalize send path — deps land here (mail-parser =0.10.2, async-imap =0.10.4, rustls 0.23 with ring, tokio-rustls 0.26, webpki-roots 0.26). Each pinned per LC-59 discipline with a re-spike-on-bump comment. New module: resolve.rs (header precedence), parse.rs (subject + text/plain body extraction, 64 KiB cap with truncate-with-marker), actor.rs (post orchestration), mod.rs (DropReason taxonomy). routes/room.rs gains finalize_email_inbox_message_send mirroring the webhook path. Named decision (commented in the code): the link filter is INTENTIONALLY skipped for email-ingress messages, mirroring LC-74; the secret-gates-it trust model is consistent across both synthetic-actor surfaces.
4. feat(email-ingress): IMAP poll loop + spawn_email_poll — poll.rs (spawn gate, poll_once tick driver, process_polled_message per-message handler, detect_loop heuristic). main.rs gains the rustls::crypto:💍:default_provider().install_default() startup call + spawn_email_poll in the spawn list. MAX_RAW_MESSAGE_BYTES = 5 MiB closes the LC-77 spike-A finding that mail-parser does not impose its own header-size limit.
5. test(email-ingress): per-message pipeline integration tests — 14 tests against the pure-async process_polled_message surface (no IMAP transport involved). Happy path, all four header positions, unknown secret, unknown domain, revoked inbox, Auto-Submitted (both auto-replied and the "no" negative), Precedence: bulk, List-Id (detail explicitly names it), rate limit, empty parse, forged From threat-model negative test (forged admin@... still posts as the inbox; identity is the secret, not From).
6. style: cargo fmt sweep across the LC-77 surface + LC-59 carryover — pure whitespace. Closes the LC-77-FMT-CARRYOVER followup inline.

Always-Seen failure log

The sole diagnostic channel for dropped mail (no bounces, no dead-letter folder in v1). Every dropped message logs at WARN with target: "email_ingress::drop" carrying uid, reason (from the exhaustive DropReason enum: parse_fail / address_no_match / revoked_inbox / loop_detected / rate_limited / internal_error), and a free-form detail string that the support guide will key off (e.g. "tried: lc_xxx@..." for AddressNoMatch, "Auto-Submitted present" for LoopDetected, "inbox 7 exceeded 60/min cap" for RateLimited).

The unconditional STORE +Seen happens AFTER the attempt, success OR fail. A poison message that never processes still gets Seen-d so it cannot trigger retry-forever; this is exactly the trap the brainstorm called out.

Threat model verifications pinned in tests

• Identity is the secret, not From: forged_from_still_posts_identity_is_secret_not_from posts an email with From: admin@lets-chat-deployment.test (a clearly forged sender) and confirms the message attaches to the inbox synthetic actor, not the From sender. Stored body never echoes the From verbatim.
• Domain match required: unknown_domain_drops_even_if_local_part_matches confirms the right token at the wrong domain drops with AddressNoMatch.
• Revoked inboxes drop silently: revoked_inbox_drops_with_revoked_inbox_reason — no bounce, no error to the sender, just a log line.
• Rate limit caps a burst: rate_limit_blocks_burst_above_cap — 60 posted, 5 dropped with RateLimited.

What is NOT in this PR (deferred to subsequent LC-77 commits)

• Admin lifecycle UI (commit 4 / next PR). v1 of this PR has no way to CREATE an inbox from the web UI; tests insert rows directly. The schema and the db helpers are ready; the admin UI just wires them.
• MIME body + attachments (commit 5 / following PR). parse.rs is minimal: subject + first text/plain part with 64 KiB cap. HTML-stripped fallback, attachment extraction, signature stripping all land in commit 5. The poll loop is functional today for plain-text-only senders; a sender who emails HTML-only will currently drop with ParseFail (or post a sparse body if mail-parser's text fallback returns anything).
• Docs + threat-model integration tests (commit 6). docs/email-ingress.md lands there, including the operator deployment guide that pins the Delivered-To / X-Original-To / To / Cc header precedence as a deployment requirement.

Test plan

• All 6 commits compile cleanly individually (cargo check at each).
• Full server test suite passes under both --features standalone (default) and --no-default-features --features saas. 52 binaries, 0 failures.
• cargo clippy --tests -- -D warnings clean across the workspace.
• cargo fmt --check clean.
• LC-77 spike A (mail-parser hostile corpus) PASS evidence already posted; the EXACT-pinned mail-parser = =0.10.2 in this PR matches the spiked version.
• LC-77 spike B (async-imap against greenmail) PASS evidence already posted; the EXACT-pinned async-imap = =0.10.4 matches.

Followups (not in scope, separate tickets)

• LC-77-SMTP-SEAL: migrate SMTP password from plaintext to the same VAPID-sealed pattern this PR uses for IMAP.
• LC-77-REPLY: reply-by-email, depends on per-message notification email surface first.
• LC-77-MID-DEDUP: exactly-once dedup via Message-ID table, defer until duplicates observed.
• LC-77-DEAD-LETTER: optional dead-letter IMAP folder for poison messages, defer until operators ask for recoverability.

LC-77-FMT-CARRYOVER closed inline in commit 6 of this PR.

## Summary The runtime body for LC-77 email-ingress. Builds on PR #194 (MessageActor refactor) and PR #195 (schema). Adds the third synthetic-actor variant end-to-end: an operator-configured IMAP mailbox is polled every 5 minutes; mail addressed to `<token>@<ingress-domain>` posts to its room as the `MessageActor::EmailInbox` actor. Six commits, each independently reviewable: 1. **refactor(messages): add EmailInbox actor variant + AuthorMeta.actor** — behavior-preserving plumbing for the third actor type. Webhook fixtures stay byte-identical; two new EmailInbox fixtures pin the new arm. resolve_msg_author now also takes email_inbox_id; every caller updated. 2. **feat(email-ingress): db layer + rate-limit kind** — db::email_inbox extended with insert/find_by_secret_hash/list_for_room/revoke/touch_last_used. db::imap_config (new) reads + writes the singleton sealed-creds row via crypto::seal/open. db::chat::insert_email_inbox_message mirrors insert_webhook_message. RateLimitKind::EmailInbox added. 3. **feat(email-ingress): email_ingress module core + finalize send path** — deps land here (mail-parser =0.10.2, async-imap =0.10.4, rustls 0.23 with ring, tokio-rustls 0.26, webpki-roots 0.26). Each pinned per LC-59 discipline with a re-spike-on-bump comment. New module: resolve.rs (header precedence), parse.rs (subject + text/plain body extraction, 64 KiB cap with truncate-with-marker), actor.rs (post orchestration), mod.rs (DropReason taxonomy). routes/room.rs gains finalize_email_inbox_message_send mirroring the webhook path. **Named decision** (commented in the code): the link filter is INTENTIONALLY skipped for email-ingress messages, mirroring LC-74; the secret-gates-it trust model is consistent across both synthetic-actor surfaces. 4. **feat(email-ingress): IMAP poll loop + spawn_email_poll** — poll.rs (spawn gate, poll_once tick driver, process_polled_message per-message handler, detect_loop heuristic). main.rs gains the rustls::crypto::ring::default_provider().install_default() startup call + spawn_email_poll in the spawn list. MAX_RAW_MESSAGE_BYTES = 5 MiB closes the LC-77 spike-A finding that mail-parser does not impose its own header-size limit. 5. **test(email-ingress): per-message pipeline integration tests** — 14 tests against the pure-async process_polled_message surface (no IMAP transport involved). Happy path, all four header positions, unknown secret, unknown domain, revoked inbox, Auto-Submitted (both auto-replied and the "no" negative), Precedence: bulk, List-Id (detail explicitly names it), rate limit, empty parse, **forged From threat-model negative test** (forged admin@... still posts as the inbox; identity is the secret, not From). 6. **style: cargo fmt sweep across the LC-77 surface + LC-59 carryover** — pure whitespace. Closes the LC-77-FMT-CARRYOVER followup inline. ## Always-Seen failure log The sole diagnostic channel for dropped mail (no bounces, no dead-letter folder in v1). Every dropped message logs at WARN with `target: "email_ingress::drop"` carrying `uid`, `reason` (from the exhaustive `DropReason` enum: parse_fail / address_no_match / revoked_inbox / loop_detected / rate_limited / internal_error), and a free-form `detail` string that the support guide will key off (e.g. "tried: lc_xxx@..." for AddressNoMatch, "Auto-Submitted present" for LoopDetected, "inbox 7 exceeded 60/min cap" for RateLimited). The unconditional STORE +Seen happens AFTER the attempt, success OR fail. A poison message that never processes still gets Seen-d so it cannot trigger retry-forever; this is exactly the trap the brainstorm called out. ## Threat model verifications pinned in tests - **Identity is the secret, not From**: `forged_from_still_posts_identity_is_secret_not_from` posts an email with `From: admin@lets-chat-deployment.test` (a clearly forged sender) and confirms the message attaches to the inbox synthetic actor, not the From sender. Stored body never echoes the From verbatim. - **Domain match required**: `unknown_domain_drops_even_if_local_part_matches` confirms the right token at the wrong domain drops with `AddressNoMatch`. - **Revoked inboxes drop silently**: `revoked_inbox_drops_with_revoked_inbox_reason` — no bounce, no error to the sender, just a log line. - **Rate limit caps a burst**: `rate_limit_blocks_burst_above_cap` — 60 posted, 5 dropped with RateLimited. ## What is NOT in this PR (deferred to subsequent LC-77 commits) - **Admin lifecycle UI** (commit 4 / next PR). v1 of this PR has no way to CREATE an inbox from the web UI; tests insert rows directly. The schema and the db helpers are ready; the admin UI just wires them. - **MIME body + attachments** (commit 5 / following PR). parse.rs is minimal: subject + first text/plain part with 64 KiB cap. HTML-stripped fallback, attachment extraction, signature stripping all land in commit 5. The poll loop is functional today for plain-text-only senders; a sender who emails HTML-only will currently drop with `ParseFail` (or post a sparse body if mail-parser's text fallback returns anything). - **Docs + threat-model integration tests** (commit 6). `docs/email-ingress.md` lands there, including the operator deployment guide that pins the Delivered-To / X-Original-To / To / Cc header precedence as a deployment requirement. ## Test plan - [x] All 6 commits compile cleanly individually (`cargo check` at each). - [x] Full server test suite passes under both `--features standalone` (default) and `--no-default-features --features saas`. 52 binaries, 0 failures. - [x] `cargo clippy --tests -- -D warnings` clean across the workspace. - [x] `cargo fmt --check` clean. - [x] LC-77 spike A (mail-parser hostile corpus) PASS evidence already posted; the EXACT-pinned `mail-parser = =0.10.2` in this PR matches the spiked version. - [x] LC-77 spike B (async-imap against greenmail) PASS evidence already posted; the EXACT-pinned `async-imap = =0.10.4` matches. ## Followups (not in scope, separate tickets) - LC-77-SMTP-SEAL: migrate SMTP password from plaintext to the same VAPID-sealed pattern this PR uses for IMAP. - LC-77-REPLY: reply-by-email, depends on per-message notification email surface first. - LC-77-MID-DEDUP: exactly-once dedup via Message-ID table, defer until duplicates observed. - LC-77-DEAD-LETTER: optional dead-letter IMAP folder for poison messages, defer until operators ask for recoverability. LC-77-FMT-CARRYOVER closed inline in commit 6 of this PR.

longjacksonle added 6 commits 2026-05-25 19:16:15 +02:00

refactor(messages): add EmailInbox actor variant + AuthorMeta.actor (LC-77) ... 28326023d9

Behavior-preserving for LC-1 users and LC-74 webhooks. Adds the third actor type to the MessageActor enum and replaces AuthorMeta's is_webhook + avatar_url scalar pair with an embedded actor: MessageActor field. resolve_msg_author now also takes email_inbox_id; every caller (9 sites) passes m.email_inbox_id alongside m.webhook_id. The route handlers that build MessageViews (11 sites) now read meta.actor.clone() instead of reconstructing the enum from flags.

Verification: the webhook fixtures (tests/fixtures/lc77_webhook_render_*.html) are byte-identical to PR #194's pinned baseline. Two new fixtures (tests/fixtures/lc77_email_inbox_render_*.html) pin the rendering of the new EmailInbox arm so a future change cannot silently break it. All 51 lets-chat-server test binaries pass under both default and saas feature sets.

Email-ingress messages cannot be CONSTRUCTED yet (no insert path, no poll loop, no admin UI). This commit is the data-model + render plumbing only; the runtime hookup lands in the subsequent commits on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat(email-ingress): db layer + rate-limit kind (LC-77) ... f3a0d4cba4

Extends the LC-77 schema (PR #195) with the Rust-side helpers commit 3c onward will call. No public route is wired yet; this is intermediate plumbing.

- db::email_inbox: insert, find_by_secret_hash, list_for_room, revoke, touch_last_used + EmailInboxRow / EmailInboxAuth types. Mirrors db::webhooks function-for-function so the admin UI and the poll loop can mount on a familiar shape.
- db::chat::insert_email_inbox_message: parallel to insert_webhook_message; stores empty user_id + email_inbox_id, NULL webhook_id.
- db::imap_config (new): read / write the singleton imap_inbox_config row. AES-256-GCM seals the password under the process secret key (same crypto::seal/open path VAPID uses). Returns a typed ImapConfig with the decoded password ready to hand to async-imap.
- rate_limit::RateLimitKind::EmailInbox: new variant + tag string. Keyed by inbox row id; commit 3d's poll loop calls it with the 60/min cap matching WEBHOOK_RATE_PER_MIN.

Verification: cargo check clean on workspace + saas. No new tests yet; commit 3c lands the email_ingress core where these helpers get exercised.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat(email-ingress): email_ingress module core + finalize send path (LC-77) ... eecbdf9c26

Dependencies (BOTH after passing spike A and spike B):
- mail-parser =0.10.2 EXACT-pinned, with re-spike-on-bump comment.
- async-imap =0.10.4 EXACT-pinned with runtime-tokio feature only.
- rustls 0.23 ring crypto provider (no aws-lc-rs, no OpenSSL).
- tokio-rustls 0.26.

New module: server/src/email_ingress/
- mod.rs: shared DropReason taxonomy (parse_fail, address_no_match, revoked_inbox, loop_detected, rate_limited, internal_error). DropReason is the failure log axis; new variants require a docs update.
- resolve.rs: address-to-inbox resolution via Delivered-To / X-Original-To / To / Cc header precedence. Domain match is case-insensitive. Local part hashed via auth::hash_api_token. ResolveOutcome distinguishes Match / Revoked / NotFound; NotFound carries tried_addresses for the failure-log detail field.
- parse.rs: minimal v1 body extraction. Subject prefixes the body as a Markdown-bold first line; body is mail-parser's first text/plain part. 64 KiB cap with truncate-with-marker. Commit 5 will replace this with the full MIME walk (HTML strip fallback, attachments, signature stripping); for now it is just enough to integration-test the send path.
- actor.rs: post_email_message orchestrates db::chat::insert_email_inbox_message -> routes::room::finalize_email_inbox_message_send -> db::email_inbox::touch_last_used. PostOutcome::Dropped carries the DropReason + detail for the caller's failure log.

routes/room.rs:
- finalize_email_inbox_message_send: email-shaped sibling of finalize_webhook_message_send. Same NewMessage broadcast + mention reconcile path. Named-decision comment: the link filter is INTENTIONALLY skipped for email-ingress messages, mirroring the LC-74 posture. The secret-gates-it trust model is consistent across both synthetic-actor surfaces; this is a deliberate choice here, not drift.

No poll loop wired yet. The commit-3d work adds spawn_email_poll + poll_once + rustls install_default on top of this foundation.

Verification: cargo check and cargo check --tests clean. No new integration tests yet (commit 3e batches the end-to-end coverage).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat(email-ingress): IMAP poll loop + spawn_email_poll (LC-77) ... 71095abc54

The runtime body for the email-ingress feature. Builds on commits 3a-3c (data-model + db layer + module core).

poll.rs:
- spawn_email_poll: startup-gated background task. Gates on LETS_CHAT_SECRET_KEY configured, imap_inbox_config row present, enabled = 1, and ingress_domain set. All four checks happen at startup (not per-tick), matching the LETS_CHAT_RETENTION_SWEEP_ENABLED + retention-sweeper precedent. Flipping enabled requires a server restart.
- poll_once: drives one full IMAP tick. TLS connect (tokio-rustls + ring) -> LOGIN -> SELECT folder -> UID SEARCH UNSEEN -> per-UID: FETCH BODY[] -> process -> STORE +Seen ALWAYS (success OR fail). The unconditional Seen-after-attempt is non-negotiable: a poison message that never processes still gets Seen-d so it cannot trigger retry-forever.
- process_polled_message: pure async function from raw RFC 822 bytes through parse -> loop-detect -> resolve -> rate-limit -> body extract -> actor post. No transport involved. Integration tests in commit 3e feed this directly.
- detect_loop: header heuristic. Drops on Auto-Submitted (any non-"no" value), Precedence: bulk/list/junk, X-Autoreply, X-Autorespond, List-Id. List-Id drop is conservative; the docs (commit 6) will explicitly call out "mail with a List-Id header is dropped, including legitimate automated senders that set it" so an operator hitting reason=loop_detected detail="List-Id present" can diagnose from logs alone.
- MAX_RAW_MESSAGE_BYTES = 5 MiB: upstream bound at the FETCH boundary. Closes the LC-77 spike A finding that mail-parser does not impose its own header-size limit.
- POLL_INTERVAL_SECS = 300 (5 min), POLL_RATE_LIMIT_PER_MIN = 60.

main.rs:
- rustls::crypto:💍:default_provider().install_default() at startup. Returns Err if already installed (reqwest may install transitively); we treat as benign no-op since we just need SOMETHING installed.
- spawn_email_poll added to the startup spawn list.

Cargo.toml:
- webpki-roots = "0.26" added. Mozilla root CA list for TLS verification; pinned by range (not exact) because the root list is data not code and rotation is routine.

Verification: cargo check + cargo check --tests + cargo clippy -- -D warnings all clean. End-to-end integration tests land in commit 3e.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

test(email-ingress): per-message pipeline integration tests (LC-77) ... d2d24097bd

14 tests against the email_ingress::poll::process_polled_message pure-async surface (no IMAP transport involved). Covers:

- Happy path: well-formed message with a valid token in `To` posts to its room as the EmailInbox synthetic actor; user_id is empty, webhook_id is NULL, email_inbox_id matches, body carries the subject as a Markdown-bold prefix.
- Header precedence: Delivered-To, X-Original-To, To, Cc each succeed on their own.
- Unknown secret -> AddressNoMatch (detail lists the tried address).
- Unknown domain (right token, wrong domain) -> AddressNoMatch.
- Revoked inbox -> RevokedInbox.
- Loop detection: Auto-Submitted: auto-replied -> LoopDetected; Auto-Submitted: no does NOT drop; Precedence: bulk -> LoopDetected; List-Id present -> LoopDetected (detail explicitly names List-Id).
- Per-inbox rate limit: 60 messages posted, attempts 61-65 dropped with RateLimited.
- Empty body + missing subject -> ParseFail.
- Threat model: a forged `From` (e.g. admin@lets-chat-deployment.test) does NOT prevent posting; identity attaches via the inbox secret, not From. Stored body never echoes the From verbatim.

One tiny fix to poll.rs: extract_body's empty check now uses .trim().is_empty() so a body that's just CRLF whitespace (the empty-body+no-subject test case) drops with ParseFail instead of posting.

Verification: cargo test under both default and saas feature sets passes all binaries. cargo clippy --tests -- -D warnings clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

style: cargo fmt sweep across the LC-77 surface + LC-59 carryover ... 8891bbdeb8

Pure whitespace. cargo fmt --check is now clean on the repo from this commit forward. The new LC-77 files (db/imap_config.rs, db/email_inbox.rs, email_ingress/*) and every LC-77-touched route file (routes/{dm,mod,room,ws}.rs) get the wrap rules applied. The math.rs / markdown.rs deltas are the LC-59 carryover that has been failing fmt on main since the LC-59 PR landed; folded in here because it is mechanical, contributes zero semantic change, and the LC-77 PRs already established the pattern of running cargo fmt as part of pre-commit. Closes LC-77-FMT-CARRYOVER inline.

Verification: cargo fmt --all -- --check returns zero diffs after this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

longjacksonle merged commit b2b24b082c into main 2026-05-25 19:24:59 +02:00

longjacksonle referenced this pull request from a commit 2026-05-25 19:25:00 +02:00

Merge pull request 'feat(email-ingress): IMAP poll + resolution + synthetic-actor send (LC-77)' (#196) from feat/lc-77-imap-poll into main

longjacksonle deleted branch feat/lc-77-imap-poll 2026-05-25 19:25:00 +02:00

longjacksonle referenced this pull request from a commit 2026-05-25 20:33:38 +02:00

docs(email-ingress): operator guide + threat-model integration tests (LC-77)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

psa-systems/lets-chat!196

No description provided.