Unified Mail — design doc & backend handoff

Laptop CLI

The standalone Unified Mail CLI connects to https://mail.benson.earth by default. It provides JSON output, search, message and attachment reads, compose/reply/forward, drafts, tags, contacts, settings, and revision watching. Run sh scripts/build-cli.sh for macOS and Linux binaries; sign in once on the laptop with unified-mail auth login. SSH is optional.

The historical backend design and handoff follows.

Written 2026-07-04 by Claude (session on blubar), after investigating the aerc/notmuch setup on blubar and auditing the copy of that setup here on alba. This doc is the context for building the new mail system on alba. Nothing has been changed on either machine — investigation only.

The plan in one paragraph

Ditch aerc. Keep the backend (notmuch + lieer + patched mbsync), which the investigation found to be sound. alba becomes the single mail server: it syncs both accounts on a launchd schedule, owns the only copy of the mail store, and exposes an HTTP API + web app. satch (VPS) later fronts it on a public domain via Tailscale + reverse proxy. Android app later, using the same API. blubar stops syncing mail entirely once this is live (single writer).

Verdict from the blubar investigation: backend good, aerc-layer bad

The flakiness ("have to check webmail to see new mail") was not the backend. notmuch DB was consistent, tags coherent, both accounts syncing correctly whenever sync actually ran. The problems were all in how aerc orchestrates sync and display:

1. Sync only runs while aerc is open. No launchd/cron job exists (on either machine). aerc triggers mail-sync via check-mail every 5m per account. aerc closed → mail silently ages.
2. Silent lock deadlock after timeouts. aerc SIGKILLs a check-mail command that exceeds check-mail-timeout (10m). SIGKILL means mail-sync's cleanup trap never runs, the lock dir stays, and — because mail-sync treats "lock held" as success (mkdir lock || return 0) — every subsequent sync silently no-ops for up to 30 minutes until the stale-lock self-heal kicks in. Also: aerc kills only the sh wrapper; the child gmi/mbsync keeps running orphaned.
3. Errors are invisible. check-mail output is discarded; failures only flash in aerc's status bar. There is no log anywhere saying when the last successful sync per account happened.
4. UI refresh is fragile. aerc's notmuch backend never refreshes the message list when check-mail completes; its only refresh trigger is a macOS FSEvents watcher on ~/.mail/.notmuch/xapian that ignores "modified" events (only create/rename/ remove). Verified empirically on blubar: xapian atomically recreates iamglass on every commit, so rename events do fire and the watcher should work — but the end-to-end aerc behavior was never confirmed (TUI can't be driven headlessly), and it's moot since aerc is being replaced.
5. Cosmetic: 64 messages (of ~24.6k) carry both gmail and migadu tags — same message delivered to both accounts; notmuch dedups by Message-ID and the copies share one DB entry. Harmless, but the new frontend should expect it.

Design consequences for the new system: sync must be daemon-driven (not client-driven), locking must be flock-based (auto-released on process death), every sync run must log its outcome, and the UI must display sync freshness.

Backend reference (identical architecture on both machines)

• Store: ~/.mail = notmuch database.path and mail_root.
  • ~/.mail/account.gmail/ — lieer dir (config .gmailieer.json, state .state.gmailieer.json, maildir under mail/). Sync: gmi sync (push+pull, tags↔labels). Send: ~/.local/bin/gmail-send.
  • ~/.mail/account.migadu/ — maildirs (INBOX, Archive, Sent, Drafts, Junk, Trash) synced by patched mbsync. Send: ~/.local/bin/migadu-send (msmtp). Password: security find-generic-password -a owner@example.org -s mail-migadu -w.
• notmuch config: new.tags=new; post-new hook (~/.mail/.notmuch/hooks/post-new) tags +gmail / +migadu by path and strips new. search.exclude_tags=deleted;spam. new.ignore covers mbsync/lieer state files.
• ~/.local/bin/mail-sync [gmail|migadu|all] — the sync entrypoint: per-account mkdir-based lock (30-min stale self-heal), runs gmi/mbsync then notmuch new.
• ~/.local/bin/unified-archive — archive semantics, reuse in the API: Gmail = remove inbox tag (gmi propagates to Gmail). Migadu = move maildir file INBOX→Archive, stripping the ,U=<n> filename marker (keeping it causes duplicate-UID collisions that abort mbsync).
• Patched mbsync — critical. Migadu's Dovecot sends unsolicited MODSEQ in FETCH responses; stock isync (1.5.1 and git HEAD) aborts with malformed FETCH response, deadlocking Archive sync. ~/.local/bin/mbsync (version string isync 0949519) is a local build that tolerates it. Never replace it with brew's mbsync. ⚠ The rebuild source of truth exists only on blubar: blubar:~/.local/share/isync-patch/modseq-tolerance.patch (apply to isync git master, ./autogen.sh && ./configure && make, copy src/mbsync). Copy that directory to alba before blubar is ever wiped.
• ~/.mbsyncrc: Expunge Both (deletes/archives finalize server-side), SyncState *, Timeout 600, PipelineDepth 1. md5-identical on both machines.
• Folder semantics (from the aerc query maps — use as the spec for the web app):
  • Unified Inbox: (tag:gmail and tag:inbox and not tag:trash and not tag:spam) or (tag:migadu and folder:account.migadu/INBOX and not tag:deleted)
  • Gmail is tag-driven (labels); Migadu is folder-driven (maildir location).
  • Unread = tag:unread, Flagged = tag:flagged, per-account variants in gmail-queries.conf / migadu-queries.conf (in the aerc config dir).

Alba audit (2026-07-04) — differences & required fixes before building

The setup on alba is a near-exact copy of blubar's, but stale and unverified:

• Scripts are functionally identical to blubar's fixed versions (mail-sync, unified-archive, post-new hook, .mbsyncrc — diffs are comments only). The patched mbsync is present (isync 0949519, independent build). notmuch config is equivalent (incl. exclude_tags).
• State is stale: last sync Jun 23; 24,475 messages vs 24,642 on blubar. First task: run ~/.local/bin/mail-sync all manually (with notmuch from /opt/homebrew/bin on PATH), check it completes cleanly, and confirm counts roughly converge with the mail servers. Do not build on the store until a clean sync is demonstrated.
• No sync daemon (same gap as blubar) — building it is part of this project.
• Two aerc config dirs exist: ~/Library/Preferences/aerc/ (live — what aerc actually reads on macOS) and ~/.config/aerc/ (stale; its accounts.conf is older and differs). Only trust the Library one as reference; ignore/delete .config.
• Stray ~/.mail/Archive/ maildir (top-level, outside both accounts): contains 2 files from Jun 17 — an artifact of an early unified-archive bug. Both messages are duplicates (also present in their proper stores, verified via notmuch thread listing). Safe to delete the directory and notmuch new afterwards; until then those files are indexed under folder:Archive where no account query sees them.
• Keychain entry unverified on alba — confirm security find-generic-password -a owner@example.org -s mail-migadu -w works in the launchd context (keychain must be unlocked; may need the daemon to run as a LaunchAgent in the user session, not a LaunchDaemon).
• Tag divergence caveat: notmuch tags live per-machine. Gmail tags round-trip through labels (gmi), and Migadu read/flagged state round-trips through maildir flags — but any notmuch-only custom tags applied on blubar will not transfer to alba's DB. If any custom tags matter, export them from blubar (notmuch dump --output=blubar-tags.txt) before cutover.

Target architecture

Gmail API ──gmi──┐
                 ├─→ alba: ~/.mail + notmuch  ←─ sync daemon (launchd, 1-2 min)
Migadu IMAP ─mbsync┘         │
                             ├─ API server (localhost) ── web app
                             │       │
                   Tailscale tailnet │
                             │       │
              satch (VPS): Caddy reverse proxy ── https://mail.<domain>
                             │
              phone/laptop clients (later: Android app on same API)

• alba is the single writer. No other machine syncs these accounts after cutover.
• Sync daemon: LaunchAgent, StartInterval 60–120s, invoking a hardened mail-sync:
  • flock-based locking (lock dies with the process — no stale-lock states).
  • Stagger or serialize gmail/migadu (they currently fire simultaneously and can contend for the notmuch write lock; notmuch is built with retry_lock, but serializing is cleaner and mail-sync all already does it).
  • Log each run (timestamp, account, duration, exit status) and write a last-success-<account> state file the API can read.
• API server (small; FastAPI + notmuch Python bindings, or shell out to notmuch ... --format=json which is entirely adequate):
  • GET /search?q= — raw notmuch query syntax passthrough (this is the Gmail-quality search; don't invent a query language, expose notmuch's).
  • GET /thread/<id>, GET /message/<id> — bodies + sanitized HTML.
  • POST /tag, POST /archive (reuse unified-archive semantics), POST /send (route by from-address to gmail-send / migadu-send).
  • GET /revision + GET /changes?since=<rev> — notmuch's lastmod: counter is a built-in delta-sync cursor (lastmod:N..). This is the shared sync state for the future Android app; design it in from day one, it's nearly free.
  • GET /health — last-success sync timestamps per account. Surface staleness in the web UI header. This kills the "is it in sync?" anxiety for good.
• Web app: the hard part is HTML email rendering. Sanitize server-side (strip scripts/forms, neutralize links target, block remote images by default with a click-to-load that proxies through alba). Everything else is a list view over /search with the folder queries above as presets.
• satch (VPS): Tailscale member; Caddy proxying mail.<domain> → alba's tailnet address. Do not expose publicly without real auth in front — the entire mail archive rides on it. Phase 1 can skip satch entirely (clients on the tailnet reach alba directly); add satch when the public domain is actually needed.
• Prior art worth skimming for API/UI shape (probably not adopting): netviel, kukulkan (both notmuch web frontends).

Mutation model (Go port)

The Go API never moves mail files. Archive/unarchive/trash on an mbsync account write a per-account pending-intent tag (um-pending-{archive,inbox,trash}-<tag>); the query presets read that tag as an override of the message's folder, so the UI flips instantly. Lieer accounts are unchanged (tags only). mail-sync-go is the only process that moves maildir files, and it clears each pending tag once the maildir matches — idempotent across crashes.

Two locks in the state dir: sync.lock, held by mail-sync-go for the whole run so only one runner exists (exit 75 when busy), and lock, shared with the API, held only for local phases and deliberately free during mbsync's network half — which is what keeps mutations fast. mail-sync-go drain applies and clears pending intent with no network, for rollback.

mail-sync-go runs either as a one-shot (all, or one account — what launchd's StartInterval and the API's post-mutation spawn invoke) or as mail-sync-go daemon, a long-lived scheduler that is the recommended production form. The daemon runs one all pass at startup, then on a ticker (sync_interval_seconds, default 120, minimum 30), and additionally holds an IMAP IDLE connection open per mbsync account so new mail triggers a pass for that account within seconds. Per-account "idle": false opts out; Gmail/lieer accounts stay on the ticker because lieer has no push channel. IDLE credentials come from the account's mbsync rc file, never a second copy, and the PassCmd runs per connect and is never cached or logged. Push is only ever a hint: an unparsable rc file, a non-IMAPS transport, a server without IDLE, or a dropped connection all log one line and fall back to the ticker. A daemon pass is the same unit of work as a one-shot — same sync.lock, same status and log output — so the two are interchangeable, and exactly one of them may be scheduled for an instance.

API parity contract

Run tests/contract/parity.py to build disposable lieer/mbsync Maildirs, index a fixed RFC822 corpus with real notmuch, start the Python and Go APIs on scratch ports, and compare their read/write behavior and resulting state. The harness copies recorder fakes into an isolated temporary HOME; it never reads or writes live mail or configuration. It prints SKIP and exits successfully when notmuch, Go, or server/.venv dependencies are absent; CI can use --require-deps to make that condition fail. --smoke tests only the harness internals. The Go API intentionally applies a safer 64 MiB cap to the complete encoded multipart send request (including multipart framing) and returns 413 when exceeded; Python remains unchanged, so this divergence is covered by Go tests rather than the parity harness. Parsed multipart temporary files are always removed.

Safe Go shadow launchd templates, build/signing commands, backend-matrix test commands, and Python-preserving cutover/rollback steps are documented in deploy/GO-ROLLOUT.md.

Cutover checklist

1. Copy isync-patch/ from blubar to alba (~/.local/share/isync-patch/).
2. Manual mail-sync all on alba until clean; delete stray ~/.mail/Archive/; notmuch new; spot-check counts/queries.
3. (If custom tags matter) notmuch dump on blubar → notmuch restore review on alba.
4. Install sync LaunchAgent on alba; watch logs for a day; confirm keychain access works from launchd.
5. Build API + web app against localhost.
6. Stop using aerc/sync on blubar (its ~/.mail becomes a dead archive; the aerc configs in ~/Library/Preferences/aerc/ remain as reference).
7. Later: satch proxy + auth; later still: Android app on /changes.