LC-77-MID-DEDUP: exactly-once dedup via Message-ID table #202

New issue

Closed

opened 2026-05-25 21:22:16 +02:00 by longjacksonle · 0 comments

longjacksonle commented

2026-05-25 21:22:16 +02:00

Owner

Background

LC-77 v1's poll loop posture is at-least-once with \Seen-after-attempt:

FETCH BODY.PEEK[] for an unseen UID.
Process the message (resolve, parse, post or drop).
STORE +FLAGS (\Seen) on that UID — ALWAYS, success or fail.

A crash between (2) and (3) means the message has been processed (the chat row exists) but the IMAP UID is still UNSEEN, so the next tick reprocesses it and posts the same message a second time.

Duplicates are rare in practice (crashes between two adjacent IMAP commands are infrequent), but possible. v1 accepts the tradeoff because the alternative (Seen-then-process) would lose a message on crash, which chat tolerates worse than a duplicate.

When to act

If operators report duplicate posts in practice, OR if a future change makes the processing step heavier (e.g. external API calls during attachment processing) such that the crash window widens.

Scope

Add an exactly-once dedup layer keyed on the message's RFC 5322 Message-ID header.

New table processed_message_ids (chat.db or settings.db; chat.db likely) with (inbox_id, message_id_hash, processed_at) and a UNIQUE constraint on (inbox_id, message_id_hash).
Hash the Message-ID via HMAC under LETS_CHAT_SECRET_KEY so the table doesn't leak which senders the operator has been receiving mail from.
Before processing: check the row. If present, skip directly to STORE +Seen.
After processing: insert the row.
TTL: opportunistic sweep at poll-tick time, drop rows older than 30 days (Message-IDs older than that won't recur).

Anti-scope

Doesn't change the always-\Seen-after-attempt posture (we still want poison messages to not retry-forever).
Doesn't apply to messages without a Message-ID (very rare; the standard's "SHOULD" makes this edge case operator-rare). Without a Message-ID, fall back to v1 at-least-once.

Test requirements

Test that simulates a crash between process and STORE \Seen (manually drive process_polled_message twice without mark_seen); the second call should be a no-op (no second row in messages).
Test that two distinct Message-IDs both process normally (the dedup is keyed correctly).

References

docs/email-ingress.md "Not supported" section names this as LC-77-MID-DEDUP.
LC-77 commit-3 brainstorm (the parent ticket) names this as the "defer until duplicates are observed in practice" decision.

## Background LC-77 v1's poll loop posture is **at-least-once with `\Seen`-after-attempt**: 1. FETCH `BODY.PEEK[]` for an unseen UID. 2. Process the message (resolve, parse, post or drop). 3. STORE +FLAGS (\Seen) on that UID — ALWAYS, success or fail. A crash between (2) and (3) means the message has been processed (the chat row exists) but the IMAP UID is still UNSEEN, so the next tick reprocesses it and posts the same message a second time. Duplicates are rare in practice (crashes between two adjacent IMAP commands are infrequent), but possible. v1 accepts the tradeoff because the alternative (Seen-then-process) would lose a message on crash, which chat tolerates worse than a duplicate. ## When to act If operators report duplicate posts in practice, OR if a future change makes the processing step heavier (e.g. external API calls during attachment processing) such that the crash window widens. ## Scope Add an exactly-once dedup layer keyed on the message's RFC 5322 Message-ID header. - New table `processed_message_ids` (chat.db or settings.db; chat.db likely) with `(inbox_id, message_id_hash, processed_at)` and a UNIQUE constraint on `(inbox_id, message_id_hash)`. - Hash the Message-ID via HMAC under `LETS_CHAT_SECRET_KEY` so the table doesn't leak which senders the operator has been receiving mail from. - Before processing: check the row. If present, skip directly to STORE +Seen. - After processing: insert the row. - TTL: opportunistic sweep at poll-tick time, drop rows older than 30 days (Message-IDs older than that won't recur). ## Anti-scope - Doesn't change the always-`\Seen`-after-attempt posture (we still want poison messages to not retry-forever). - Doesn't apply to messages without a Message-ID (very rare; the standard's "SHOULD" makes this edge case operator-rare). Without a Message-ID, fall back to v1 at-least-once. ## Test requirements - Test that simulates a crash between process and STORE \Seen (manually drive `process_polled_message` twice without `mark_seen`); the second call should be a no-op (no second row in `messages`). - Test that two distinct Message-IDs both process normally (the dedup is keyed correctly). ## References - `docs/email-ingress.md` "Not supported" section names this as `LC-77-MID-DEDUP`. - LC-77 commit-3 brainstorm (the parent ticket) names this as the "defer until duplicates are observed in practice" decision.