feat(email-ingress): MIME body + attachment upload pipeline (LC-77) #198
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/lc-77-mime-attachments"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Wires real MIME walking + attachment upload through the LC-77 email-ingress poll path. The poll loop posts a multipart/mixed email's text body to chat AND uploads its attachments through the same content-addressed pipeline web uploads use (magic-byte sniff, MIME allowlist, EXIF strip, sha256 storage).
What changed
parse.rs: addsextract_attachments(message) -> Vec<RawAttachment>, capped atMAX_ATTACHMENTS_PER_MESSAGE = 4. Walksmail-parser'smessage.attachments()iterator. EachRawAttachmentcarries the decoded bytes plus a sanitized basename (path separators stripped, control chars filtered, 255-char cap).attachments.rs(new):process_attachment(state, raw) -> Result<i64, AttachmentDrop>. Streams the in-memory bytes to a temp file underuploads_dir/.tmp,infer::get_from_path-sniffs (the sender's Content-Type is NOT trusted), allowlist-checks against jpg/png/gif/webp/pdf (matchesroutes/uploads.rs::allowed_ext_for_mime), runs images throughpipeline::process_imagefor EXIF strip, content-addressed by sha256 of the STRIPPED bytes, writes viawrite_atomic, inserts afile_uploadsrow withuploader_id=""(the same sentinel webhook posts use formessages.user_id, verified in commit 2).actor.rs:post_email_messagenow also takes&[RawAttachment]. Pipeline runs FIRST so drops are logged before we commit the message row. Successfully-processed uploads are linked to the new message viadb::uploads::link_upload_to_message. Attachment-only emails (empty subject + body, one+ successful upload) post with a single-space body so the markdown pipeline produces a valid row alongside the attachment partial.poll.rs::process_polled_message: extracts attachments alongside the body. The drop guard now requires BOTH body-empty AND attachments-empty to fireParseFail.uploads/mod.rsrefactor: liftedsha256_bytesandsha256_filefromroutes/uploads.rstopub fnincrate::uploads. Web upload path unchanged in behavior; it just imports the shared helpers now. Single source of truth.Threat model assertions (anchored in tests)
EXIF strip:
jpeg_re_encode_strips_exif_signature_from_stored_bytesinjects a synthetic APP1 EXIF segment into a JPEG (theExif\0\0signature), confirms the test setup invariant that the source carries it, then drives the polled message through the pipeline and reads the stored file off disk. The signature is GONE. Same decode+re-encode guarantee the web upload path provides.Magic-byte trust:
jpeg_claiming_zip_dropped_at_sniff_message_still_postssends a payload with zip magic bytes (PK\x03\x04) taggedContent-Type: image/jpeg.infer::get_from_pathidentifies the real type; the allowlist rejects; the attachment is dropped INFO; the message body still posts. The sender's Content-Type is never the trust boundary.MIME allowlist enforcement:
disallowed_mime_dropped_attachment_message_still_postssends an ELF binary; the sniff returnsapplication/x-executable(or similar), not in the allowlist, dropped INFO. Body still posts.Attachment count cap:
fifth_attachment_truncated_at_extract_layersends 5 jpegs; only 4 land. The 5th is silently dropped at the parse layer because the body still represents the sender's intent.Body cap with marker:
body_over_64kib_truncates_with_marker_message_still_postssends an 80 KiB body, asserts the stored body is ≤64 KiB AND contains the_[truncated]_marker.No raw HTML in body:
html_only_message_drops_or_posts_stripped_no_raw_html_in_bodysends an HTML-only message with a<script>tag in the source. The stored body either does not exist (ParseFail) or does not contain<script. The sender's HTML never reaches the chat markdown pipeline as raw HTML.Anti-scope
mail-parser'sbody_text()fallback provides. A dedicated HTML stripper (with link preservation and basic formatting carry-over) is a follow-up if operators ask for richer HTML-only rendering.LC-77-REPLYwork, not v1 ingress.MediaRecorderorigin that emails do not produce.Test plan
email_ingress_processtests still pass (actor signature changed but the existing call sites went throughprocess_polled_message).email_ingress_attachments.rscover the threat model and the truncation/cap surfaces.Followups (not in scope)
LC-77-SMTP-SEAL, LC-77-REPLY, LC-77-MID-DEDUP, LC-77-DEAD-LETTER tracked previously. Commit 6 lands
docs/email-ingress.md(operator deployment guide pinning the header precedence) plus the explicit threat-model integration tests for the parent brainstorm's named assertions (forged-From-still-posts, no-raw-HTML, unknown-secret-silent-drop, loop-header-drop).Builds on PRs #194-#197. Wires real attachment handling through the email-ingress poll path. parse.rs: - Adds MAX_ATTACHMENTS_PER_MESSAGE = 4 (matches the brainstorm cap). - Adds RawAttachment { filename, claimed_content_type, bytes } and extract_attachments(message). Walks mail-parser's message.attachments() iterator (which already covers Content-Disposition: attachment plus non-text multipart/mixed parts), takes up to 4, sanitizes the filename (basename-only, no control chars, 255-char cap). attachments.rs (new): - process_attachment(state, raw) runs the full upload pipeline against an in-memory RawAttachment: streams to a temp file under uploads_dir/.tmp, magic-byte sniffs via infer::get_from_path (NOT the sender's Content-Type), checks allowlist (jpg/png/gif/webp/pdf, matching routes/uploads.rs::allowed_ext_for_mime), runs images through pipeline::process_image (the EXIF-stripping decode+re-encode the web upload path already uses), content-addressed by sha256 of the STRIPPED bytes, writes via write_atomic, inserts file_uploads row with uploader_id="" (the synthetic-actor sentinel, verified in commit 2). - AttachmentDrop enum is the per-attachment failure taxonomy: OverSize, DisallowedMime, SniffFailed, ImagePipeline, Io, Db. Each is logged INFO with the inbox id, filename, and detail; the parent message still posts (non-fatal). actor.rs: - post_email_message now also takes &[RawAttachment]. Runs the attachment pipeline FIRST (so any drop logs before we commit the message row), then inserts the message, then links each successfully-processed upload via db::uploads::link_upload_to_message. - Attachment-only emails (empty subject + body, one+ successful upload) post with a single-space body so the markdown pipeline produces a valid row alongside the attachment partial. poll.rs: - process_polled_message extracts attachments alongside the body and forwards both to post_email_message. The drop guard now requires BOTH body-empty AND attachments-empty to fire ParseFail. uploads/mod.rs (refactor): - sha256_bytes and sha256_file lifted from routes/uploads.rs to pub fn in crate::uploads so the email-ingress pipeline can reuse them without duplication. routes/uploads.rs unchanged in behavior (uses the shared helpers via a `use` re-export). 8 new integration tests in tests/email_ingress_attachments.rs cover: - text + jpeg: body posts, upload row linked to the message, upload row has the sniffed MIME and >0 size. - EXIF strip: a JPEG carrying an APP1 EXIF segment in its source bytes has the "Exif\0\0" signature stripped after the pipeline round-trip; the test reads the file off disk and asserts the signature is gone. - jpeg-claiming-zip: PK\x03\x04 magic with Content-Type: image/jpeg drops at the sniff step; the message body still posts. - disallowed MIME (ELF binary): drops with sniffed "application/x-executable" not in allowlist; message body still posts. - 5 attachments: parse layer caps at 4; one is silently dropped at extract time. - attachment-only email: empty subject + empty body + one jpeg posts with a placeholder body. - 80 KiB body: truncated to <=64 KiB with the _[truncated]_ marker. - HTML-only message: either drops ParseFail OR posts with HTML stripped (mail-parser's text fallback varies); the stored body NEVER contains a raw <script tag. All 14 existing email_ingress_process tests still pass after the actor signature change. All 53+ server test binaries pass under both default and saas feature sets. cargo clippy --tests -- -D warnings clean. cargo fmt --check clean. The named link-filter-skip decision was anchored in commit 3 (finalize_email_inbox_message_send carries the explanatory comment in routes/room.rs). No change to that posture here; commit 5 just builds the attachment pipeline on top of the existing send path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>