LC-207-OBSERVABILITY: surface email-ingress drops, retention-sweep runs, and IMAP poll health to operators #278

New issue

Closed

opened 2026-05-31 03:13:41 +02:00 by longjacksonle · 0 comments

longjacksonle commented

2026-05-31 03:13:41 +02:00

Owner

Problem

Three background subsystems emit a tracing log line as their sole operator-facing diagnostic. An operator without container-log access cannot self-serve answer "why did X happen / did it run / is it alive?". LC-207 set the precedent that closing this kind of gap (the bridge-avatar admin diagnostic page) is worth doing; this ticket extends the same pattern to three more subsystems.

The three gaps

Email-ingress drops are log-only. Drops emit WARN target=email_ingress::drop (server/src/email_ingress/poll.rs:210/227/252); the admin IMAP page (routes::email_inboxes, templates/admin/settings.html) is config-only. An operator asking "why didn't my email post?" must read container logs. docs/email-ingress.md itself names the log as the only channel.
Retention sweep has no run observability. spawn_message_retention_sweeper (server/src/main.rs:490) logs rooms_touched / messages_deleted to tracing only (main.rs:505-509); nothing persisted, nothing surfaced. An operator who flipped the destructive LETS_CHAT_RETENTION_SWEEP_ENABLED cannot confirm it ran or how many rows it deleted. Highest-stakes of the three (destructive + opt-in).
IMAP poll-loop health is log-only. spawn_email_poll (poll.rs:100) logs per-tick results but persists no last_poll_at / consecutive-failure / status; "is the poll loop alive and when did it last connect?" needs logs. The imap_inbox_config row carries config only.

Failure mode / cost

Operators running ingress + retention in production cannot diagnose silent drops, confirm a destructive sweep behaved, or notice a wedged poll loop, without shell access to the container. For the retention sweep specifically, "did my data get deleted as configured?" has no self-serve answer at all.

Proposed scope (one ticket, cheap, mirrors LC-207)

Persist a small amount of state: last-poll timestamp + last-error + consecutive-failure count for the IMAP loop; last-sweep timestamp + rooms/messages-deleted counts for the retention sweep; a rolling drop log (or counters by DropReason) for ingress.
Surface them as read-only rows on the existing admin settings / IMAP page, the same shape as the LC-207 bridge-avatars page (stats header + recent-failures list).
Ingress drop log: store the structured drop reason + timestamp + matched inbox (no plaintext body), bounded/swept like the LC-77 dedup table.

Out of scope

Push-notification delivery failures (push/mod.rs) are also log-only but self-heal (410-Gone subscriptions auto-prune), so they are intentionally excluded; revisit only if a real complaint surfaces.

Acceptance

An operator can answer, from the admin UI without logs: did the last IMAP poll succeed and when; did the retention sweep run and how much did it delete; what was recently dropped at ingress and why.

Anchored to LC-207 (admin diagnostic surfaces). Severity: operability.

## Problem Three background subsystems emit a `tracing` log line as their **sole** operator-facing diagnostic. An operator without container-log access cannot self-serve answer "why did X happen / did it run / is it alive?". LC-207 set the precedent that closing this kind of gap (the bridge-avatar admin diagnostic page) is worth doing; this ticket extends the same pattern to three more subsystems. ## The three gaps 1. **Email-ingress drops are log-only.** Drops emit `WARN target=email_ingress::drop` (`server/src/email_ingress/poll.rs:210/227/252`); the admin IMAP page (`routes::email_inboxes`, `templates/admin/settings.html`) is config-only. An operator asking "why didn't my email post?" must read container logs. `docs/email-ingress.md` itself names the log as the only channel. 2. **Retention sweep has no run observability.** `spawn_message_retention_sweeper` (`server/src/main.rs:490`) logs `rooms_touched` / `messages_deleted` to tracing only (`main.rs:505-509`); nothing persisted, nothing surfaced. An operator who flipped the **destructive** `LETS_CHAT_RETENTION_SWEEP_ENABLED` cannot confirm it ran or how many rows it deleted. Highest-stakes of the three (destructive + opt-in). 3. **IMAP poll-loop health is log-only.** `spawn_email_poll` (`poll.rs:100`) logs per-tick results but persists no `last_poll_at` / consecutive-failure / status; "is the poll loop alive and when did it last connect?" needs logs. The `imap_inbox_config` row carries config only. ## Failure mode / cost Operators running ingress + retention in production cannot diagnose silent drops, confirm a destructive sweep behaved, or notice a wedged poll loop, without shell access to the container. For the retention sweep specifically, "did my data get deleted as configured?" has no self-serve answer at all. ## Proposed scope (one ticket, cheap, mirrors LC-207) - Persist a small amount of state: last-poll timestamp + last-error + consecutive-failure count for the IMAP loop; last-sweep timestamp + rooms/messages-deleted counts for the retention sweep; a rolling drop log (or counters by `DropReason`) for ingress. - Surface them as read-only rows on the existing admin settings / IMAP page, the same shape as the LC-207 bridge-avatars page (stats header + recent-failures list). - Ingress drop log: store the structured drop reason + timestamp + matched inbox (no plaintext body), bounded/swept like the LC-77 dedup table. ## Out of scope Push-notification delivery failures (`push/mod.rs`) are also log-only but self-heal (410-Gone subscriptions auto-prune), so they are intentionally excluded; revisit only if a real complaint surfaces. ## Acceptance An operator can answer, from the admin UI without logs: did the last IMAP poll succeed and when; did the retention sweep run and how much did it delete; what was recently dropped at ingress and why. Anchored to LC-207 (admin diagnostic surfaces). Severity: operability.