IMAP in Ferroma#

Who should read this: anyone implementing ferroma-imap, anyone writing a compatibility test against Thunderbird or Apple Mail, and any operator whose user cannot get their mail client to log in.

Ferroma is an IMAP4rev1 server (RFC 3501) for third-party clients — Thunderbird, Apple Mail, Outlook, iPhone Mail and Android clients are first-class citizens and are not required to speak FCP (specification §53). This document specifies which commands ship in the first release, the session state machine, folder naming including the INBOX special case and the Maildir++ mapping, UID and UIDVALIDITY semantics, the flag vocabulary and its Maildir encoding, the FETCH items and SEARCH keys supported, IDLE, APPEND limits, and the compatibility target list.

Status: design specification. The ferroma-imap crate is implemented against this document; sections marked (planned) describe behaviour that is specified but not yet shipped. Everything in this file is (planned) as of now: crates/ferroma-imap/src/lib.rs is a skeleton. The folder, flag, Maildir and search behaviour it relies on is implemented, in crates/ferroma-storage/src/maildir.rs, crates/ferroma-storage/src/repository/mailboxes.rs, crates/ferroma-storage/src/repository/messages.rs and crates/ferroma-mail/src/flags.rs.

1. Protocol baseline#

imap.require_tls_for_login (default false) refuses LOGIN and AUTHENTICATE on an unencrypted connection with NO [PRIVACYREQUIRED]. docker-compose.prod.yml sets FERROMA__IMAP__REQUIRE_TLS_FOR_LOGIN to 'true'; a production deployment should keep it. LOGIN sends the password in cleartext, and there is no CRAM-MD5 or SCRAM-* to fall back on (see §8).

rustls only, and no LOGINDISABLED-versus-AUTH=PLAIN dance. When require_tls_for_login is on, LOGIN is not advertised as available over cleartext; the client sees STARTTLS in CAPABILITY and is expected to use it.

2. Commands: first release and planned#

Specification §12 splits the command set in two.

2.1 First release#

CAPABILITY   LOGIN    LOGOUT   NOOP
LIST         LSUB
SELECT       EXAMINE  STATUS
FETCH        STORE
SEARCH       UID

Plus the ones any client needs to work at all:

2.2 Planned#

APPEND       COPY         MOVE         EXPUNGE      IDLE

The gates are real config keys and are validated at boot: ImapConfig::enable_idle, ImapConfig::enable_move, ImapConfig::max_idle_secs, ImapConfig::max_append_size in crates/ferroma-core/src/config.rs.

2.3 CAPABILITY#

The advertised list is built from the configuration, never hard-coded:

* CAPABILITY IMAP4rev1
             LOGINDISABLED        (only when imap.require_tls_for_login and not encrypted)
             STARTTLS             (only when tls.enabled and not already encrypted)
             AUTH=PLAIN AUTH=LOGIN
             IDLE                 (only when imap.enable_idle)
             MOVE UIDPLUS         (only when imap.enable_move)
             UNSELECT
             LITERAL+
             UIDPLUS              (once EXPUNGE ships)
             CHILDREN             (Maildir++ has real hierarchies)

LOGINDISABLED is the RFC 3501 §7.2.1 way to say "LOGIN is not available on this connection"; it is advertised instead of removing LOGIN from the list, because a client that sees neither treats it as a broken server.

3. Session states#

   ┌──────────────────────────┐
   │     Not Authenticated    │◄── connection, "* OK [CAPABILITY …]"
   └────────────┬─────────────┘
                │  LOGIN / AUTHENTICATE succeeds
                │  (STARTTLS here returns to Not Authenticated)
                ▼
   ┌──────────────────────────┐
   │       Authenticated      │◄── CLOSE, or a failing SELECT
   └────────────┬─────────────┘
                │  SELECT / EXAMINE
                ▼
   ┌──────────────────────────┐
   │         Selected         │  one mailbox at a time
   └────────────┬─────────────┘
                │  LOGOUT (from any state)
                ▼
   ┌──────────────────────────┐
   │         Logout           │  "* BYE", then TCP close
   └──────────────────────────┘

Rules the implementation must respect:

Session struct (planned):

/// crates/ferroma-imap/src/session.rs
pub struct ImapSession {
    pub connection_id: Uuid,
    pub remote_addr: SocketAddr,
    pub state: ImapState,               // NotAuthenticated | Authenticated | Selected | Logout
    pub encrypted: bool,
    pub authenticated_user: Option<UserId>,
    pub session_id: Option<SessionId>,
    pub mailbox_id: Option<MailboxId>,  // the address currently open
    pub folder_id: Option<MailboxId>,   // the folder currently selected
    pub read_only: bool,
    /// UIDs the session has marked \Deleted but not expunged. Needed even with
    /// UIDPLUS, because a client that did not request UIDPLUS still expects
    /// EXPUNGE to remove exactly the messages it flagged.
    pub deleted_uids: BTreeSet<i64>,
}

4. Folders#

4.1 The schema behind them#

Folders are rows in folders, not directories walked at request time (migrations/0001_initial.sql):

Indexes: folders_name_key (mailbox_id, name) unique — two folders of one address cannot share a name; folders_special_use_key (mailbox_id, special_use) WHERE special_use IS NOT NULL — at most one \Sent per address.

The schema's CHECK restricts special_use to that list. INBOX is stored with special_use = NULL.

4.2 The INBOX special case#

RFC 3501 §5.1 makes INBOX special in three ways, and all three are implemented:

1. INBOX is case-insensitive. select inbox, SELECT Inbox and
   SELECT INBOX are the same folder. FoldersRepository::find_by_name compares
   case-insensitively for INBOX and case-sensitively for everything else, and
   normalise_folder in crates/ferroma-storage/src/maildir.rs canonicalises the
   spelling before it reaches the filesystem:
   normalise_folder("") == "INBOX", normalise_folder("/") == "INBOX",
   normalise_folder("/Sent/") == "Sent".
2. INBOX cannot be created, renamed or deleted. CREATE INBOX is NO
   (it already exists, conceptually); RENAME INBOX is NO; DELETE INBOX is
   NO. Maildir::delete_folder and Maildir::rename_folder refuse it too, so
   the protection holds even if a protocol layer forgets:
   StorageError::Invalid("INBOX cannot be deleted").
3. INBOX always sorts first in LIST and in Maildir::list_folders.
   Clients display the first folder as the inbox; a LIST that returns Archive
   before INBOX looks broken to a user.

INBOX carries no special_use. FoldersRepository::ensure_standard creates it with None deliberately: RFC 6154 reserves \Inbox for a folder that contains all mail, and a client that maps folders by special_use alone must fall back to matching the name INBOX.

4.3 The standard folder set#

FoldersRepository::ensure_standard (and Maildir::ensure_mailbox) create:

Idempotent and safe to race with itself — the second caller's INSERT loses to the unique index and is ignored.

4.4 Maildir++ mapping#

Nested IMAP folders are hierarchical with / as the separator. Maildir++ encodes the whole path into one directory name, dot-separated:

Archive/2026     ↔     .Archive.2026
Archive/2026/Q1  ↔     .Archive.2026.Q1

maildir_folder_name(folder) implements one direction (format!(".{}", folder.replace('/', "."))) and Maildir::list_folders implements the other (name.trim_start_matches('.').replace('.', "/")).

The nesting is flat on disk: .Archive.2026 is a sibling of .Archive, not a child of it. That is what Maildir++ means and it is why any mail tool can read the tree without understanding IMAP hierarchies. It also means a folder name may not contain a . in a position that would be ambiguous with a separator — a folder literally named a.b and a nested a/b would collide. Ferroma resolves this the way every Maildir++ implementation does: the IMAP name is authoritative and lives in folders.name; the directory name is derived. A name containing . is stored and served faithfully, and the on-disk directory is just an encoding. Maildir::folder_dir calls maildir_folder_name on every access, so the mapping is never cached and cannot drift.

Folder names are validated before they reach the filesystem. sanitize_component is applied to each path segment, so Archive/../../etc is rejected segment by segment rather than by pattern-matching the whole string:

// crates/ferroma-storage/src/maildir.rs
fn maildir_folder_name(folder: &str) -> Result<String> {
    for part in folder.split('/') {
        sanitize_component(part)?;
    }
    Ok(format!(".{}", folder.replace('/', ".")))
}

sanitize_component rejects "", ".", "..", anything containing /, \, \0 or : (see §10), and returns StorageError::Invalid otherwise.

4.5 LIST, LSUB, STATUS, SUBSCRIBE#

5. UIDs and UIDVALIDITY#

Two numbers, two different jobs. Confusing them is the classic IMAP bug.

5.1 Allocation is atomic#

The INSERT that stores a message allocates its UID in the same statement, under the folder's row lock:

WITH next_uid AS (
    UPDATE folders SET uid_next = uid_next + 1, updated_at = NOW()
     WHERE id = $1
     RETURNING uid_next - 1 AS uid
)
INSERT INTO messages (folder_id, mailbox_id, uid, …)
SELECT $2, $3, next_uid.uid, … FROM next_uid

crates/ferroma-storage/src/repository/messages.rs. Two simultaneous deliveries into one folder therefore cannot receive the same UID, and a crash cannot leave a gap that a later message would fill. copy_to_folder and move_to_folder use the same CTE.

5.2 UIDs are never reused, and expunged UIDs stay gone#

MessagesRepository::find_by_uid and list_by_uids filter expunged_at IS NULL: once a client has expunged a UID, referring to it again is NO rather than a resurrected message. expunge() sets expunged_at and returns the affected rows sorted by uid, because PostgreSQL's UPDATE gives no ordering guarantee and IMAP's EXPUNGE responses must be in ascending UID order (descending index-based order is what RFC 3501 actually requires for untagged EXPUNGE; see §6.3).

5.3 What changes UIDVALIDITY#

FoldersRepository::set_uid_validity(id, uid_validity) exists for exactly the cases where the UID space is rebuilt:

The rule the implementation follows: if any operation could make an old UID → message mapping wrong, bump UIDVALIDITY. A conservative extra bump costs a client one folder resync; a missing bump costs it a wrong message.

5.4 What a client must do when UIDVALIDITY changes#

Specification §55's "Server = Source of Truth" makes this unambiguous, and the FCP contract states it in fcp.md §4:

1. Notice that the UIDVALIDITY in the SELECT response (or in
   GET /api/v1/client/mailboxes) differs from the value cached for that folder.
2. Discard every cached UID for that folder, and every cached message that
   was identified only by its UID. Message bodies keyed by rfc_message_id or by
   the server's message_id may be kept.
3. Resync the folder from cursor 0. UIDs will be handed out afresh.
4. Never assume a UID from before the change refers to the same message.

A server-side corollary: because the client throws its cache away, Ferroma must not bump UIDVALIDITY casually. Bumping it on every delivery would make every client re-download the folder on every new message.

6. Flags and keywords#

6.1 The vocabulary#

RFC 3501 §2.3.2 defines six system flags:

\Seen     \Answered     \Flagged     \Deleted     \Draft     \Recent

Plus keywords: anything else a client sends — $Junk, $label1, $Forwarded, NonJunk. ferroma_mail::Flags stores the six system flags as a u8 bitset (bit 0 \Seen, 1 \Answered, 2 \Flagged, 3 \Deleted, 4 \Draft, 5 \Recent) and the keywords as a Vec<String> in insertion order. Keywords are lower-cased on the way in and deduplicated, which is what makes Flags::to_db_string() deterministic.

Flags::parse accepts (\Seen \Flagged $Label1), the bare form without parentheses, doubled spaces and quoted keywords. An unknown \Foo name is stored as a keyword — a client's private flags survive a round trip through a server that has never heard of them.

6.2 Three renderings, one flag set#

messages.flags is TEXT NOT NULL DEFAULT '', and the schema comment describes it as a space-separated list. Flags::to_db_string() writes comma-separated values (parts.join(",")), lower-cased with the backslash stripped: \Seen → seen, $Label1 → $label1. The column is opaque text; the comma form is what the implemented writer produces and what Flags::from_db_string() reads back. Treat Flags::to_db_string / from_db_string as the only correct way to touch that column — hand-written SQL against it is a bug waiting to happen.

\Recent is the awkward one. It is a system flag, but it is session state, not stored state: RFC 3501 says a message is \Recent if this is the first session to see it. Maildir expresses it by directory — a message that has not been seen by a mail client lives in new/, and one that has lives in cur/. So Ferroma derives \Recent from the directory, and flags_to_maildir("recent") returns the empty string because \Recent must never end up in a file name.

6.3 Maildir ↔ IMAP flag mapping#

The two vocabularies meet in exactly two functions, Maildir's flags_to_maildir and maildir_to_flags, and nowhere else. crates/ferroma-storage/src/maildir.rs is the single place to change if a letter is ever wrong.

The write order is the Maildir convention D F P R S T, implemented as D, F, R, S, T:

pub fn flags_to_maildir(flags: &str) -> String {
    // … if has("draft") { out.push('D') } if has("flagged") { out.push('F') }
    //    if has("answered") { out.push('R') } if has("seen") { out.push('S') }
    //    if has("deleted") { out.push('T') }
}

maildir_to_flags("FS") == "flagged seen"; unknown letters are ignored (maildir_to_flags("XYZ") == ""), which is what makes a store written by another tool readable.

Keywords are the lossy part, by design. Moving a message between folders rewrites its Maildir file name from the flag set, and custom keywords have no letter. They stay in messages.flags; the file name simply does not mention them. Re-reading is therefore database-first for keywords and Maildir-first for the letters — which is why iter_messages returning maildir_flags is a repair input, not the source of truth.

PERMANENTFLAGS on SELECT advertises (\Answered \Flagged \Deleted \Seen \Draft \*); \* is the RFC 3501 way to say "arbitrary keywords are accepted", and Ferroma does accept them.

6.4 STORE semantics#

Every STORE that changes something does three things, in this order:

1. MessagesRepository::set_flags / add_flags / remove_flags — the database
   row, which is what every other surface reads.
2. Maildir::set_flags(relative_path, flags) — renames the file, moving it
   between new/ and cur/ when the flag set becomes empty or non-empty.
   It returns the possibly-new path, and the caller must persist it via
   MessagesRepository::set_storage_path if it changed.
3. EventBus::publish(EventScope::User(user_id), Event::mail_flag_changed(…)), then
   one change_log append, so a synced client learns about it without polling.

Step 2 is idempotent: set_flags returns the original path when the computed name is unchanged, so a repeated STORE does not churn the filesystem.

FETCH writes \Seen implicitly when BODY[…] is fetched without .PEEK, and a \Seen transition likewise publishes Event::mail_read. Clients that want to peek — every mail client rendering a list — send BODY.PEEK[].

6.5 What Maildir cannot express, and what happens#

7. FETCH items#

Three practical rules:

• Fetched bytes are the stored bytes. BODY[] returns exactly what is in the
  Maildir — the message as received, with Ferroma's Received: header on top.
  Nothing is re-rendered, so a FETCH and a curl of
  GET /api/v1/messages/:id/raw return the same octets.
• max_fetch_messages caps one command. limits.max_fetch_messages (5000)
  bounds FETCH 1:* on a huge folder; a larger range is rejected rather than
  allowed to allocate. Clients that ask for more are broken anyway — they should
  page by UID range.
• A missing body is NO, not a truncated result:
  StorageError::BodyMissing becomes
  NO [SERVERBUG] message body missing, and the message is listed in the
  integrity report from ferroma storage verify (see storage.md §9).

Sequence sets and UID sets accept the full RFC 3501 §9 grammar: 1, 1:5, 1:*, 1,3,5:7, * (the highest UID or sequence number in the folder). * on an empty folder is an error, not an empty set.

8. SEARCH#

SEARCH is answered with a space-separated list of matching sequence numbers; UID SEARCH with UIDs. SEARCH is not a great fit for a relational store, so the translation is explicit:

MessagesRepository::search(MessageSearch) is the query builder behind all of this; the API's /api/v1/messages filters and the Admin search use the same function, so a SEARCH and an API query over the same fields return the same messages. That is the layering rule doing real work: the IMAP layer contributes only the parser.

Implementing BODY/TEXT search would mean either indexing message text in PostgreSQL or walking the Maildir per query. Neither is acceptable for v1, so it is explicitly (planned) and clients that use it get NO [CANNOT] BODY search is not supported rather than wrong results. The Admin/API side offers a subject-and-header search that does work.

9. IDLE and the 29-minute rule#

RFC 2177 recommends that a client not hold IDLE for more than 29 minutes, and requires the server to terminate it. config/ferroma.toml:

[imap]
enable_idle = true
# Longest a client may hold IDLE, in seconds (RFC 2177 recommends < 30 min).
max_idle_secs = 1740

1740 seconds is 29 minutes exactly.

C: A001 IDLE
S: + idling
      … the session stays Selected and pushes untagged updates …
C: DONE
S: A001 OK IDLE terminated

The event bus is in-process only. A session IDLEing on a folder gets pushed updates for changes made in the same process; a change made by another process sharing the database is not pushed and arrives on the next NOOP, CHECK or SELECT — and is always visible to a client that reconnects. This is the same limitation noted in architecture.md §6 and security.md.

IDLE is the reason most clients do not need CONDSTORE: the server pushes what changed, so the client does not have to poll STATUS to find out.

10. APPEND and the Windows info separator#

10.1 APPEND limits#

APPEND stores the literal verbatim. Unlike submission, it does not prepend a Received: header — the message did not travel over SMTP to get here, and inventing a hop would corrupt the header chain a client is trying to preserve. It does set flags, move the file into cur/ or new/ accordingly, and allocate a fresh UID in the target folder.

APPEND to Drafts is how a third-party client's "save draft" ends up in the same place as the official client's POST /api/v1/client/drafts; the two must not diverge, so a server-side draft is mirrored into the Drafts folder with the \Draft flag (fcp.md §7).

10.2 The Windows ; versus : separator#

A Maildir file name carries its flags after an "info" section:

1758012751.M4821P3210.mail:2,S
└─────── base ───────┘ └┬┘└┬┘
                     version flags

On Unix the separator is : — that is the de-facto standard every Maildir implementation has used for twenty years. On Windows it cannot be, because NTFS treats name:stream as an alternate data stream: creating a file literally named 1758012751.M4821P3210.mail:2,S either fails or silently creates something other than a normal file, and the flags would be invisible or the write would fail outright. Maildir implementations on Windows have therefore used ; instead for just as long.

Ferroma carries both, in crates/ferroma-storage/src/maildir.rs:

/// The canonical Maildir "info" separator (RFC-less de-facto standard, Dovecot et al).
pub const INFO_SEPARATOR_UNIX: char = ':';

/// The separator used on filesystems that reserve `:` in file names.
pub const INFO_SEPARATOR_WINDOWS: char = ';';

/// The "info" separator this build writes.
pub fn info_separator() -> char {
    if cfg!(windows) { INFO_SEPARATOR_WINDOWS } else { INFO_SEPARATOR_UNIX }
}

The rule, and why both constants exist:

assert_eq!(split_info("1234.M1P.host:2,S"), ("1234.M1P.host", "S"));
assert_eq!(split_info("1234.M1P.host;2,FS"), ("1234.M1P.host", "FS"));
assert_eq!(split_info("1234.M1P.host"), ("1234.M1P.host", ""));
// Some tools omit the `2,` version marker.
assert_eq!(split_info("1234.M1P.host:S"), ("1234.M1P.host", "S"));

Three consequences:

1. A mail store is portable. Copying Maildir/ from a Linux server to a
   Windows workstation and pointing Ferroma at it does not require rewriting a
   single file name — reads accept both forms. New writes use the local
   separator, so a tree can legitimately contain both, which split_info handles.
2. split_info ignores a one-character prefix. if base.len() <= 1 { continue; }
   exists because C: at the start of a Windows path looks exactly like an info
   separator; without the guard, a path prefix would be mistaken for flags.
3. Mounting matters. A Maildir on a CIFS/SMB share mounted on Linux will be
   written with : while the server is Linux — and if the far side is Windows,
   the write can still fail at the NTFS layer. The layout setting is deliberately
   per-deployment: storage.layout = "maildir" uses Maildir/.Folder
   subdirectories, and "maildirperfolder" uses one Maildir per folder
   (<local>/INBOX, <local>/Sent) with no dotted directory names, which is the
   safer choice on a filesystem with unusual name rules.

sanitize_component rejects : in every path component it is given, which keeps a folder name from smuggling an info section into a directory name. Message file names are built by unique_filename, never from user input.

11. Compatibility targets#

Specification §12: "Thunderbird、Apple Mail、Outlook、iPhone Mail、Android 邮件客户端能够逐步兼容" — progressive compatibility with those five.

The compatibility bar for the first release, from specification §44:

CAPABILITY   LOGIN   SELECT   FETCH   STORE   SEARCH   UID   IDLE

Each of these must be exercised against a real client, not only against a conformance script. openssl s_client -crlf -connect host:143 and a hand-typed session catch most of what a test suite does not.

Interaction with the special-use markers (§4.3) is what makes folder placement work across all five: a client that maps \Sent rather than guessing the name Sent still lines up when an account's folder is called Sent Items.

12. Testing IMAP by hand#

# Greeting, capabilities and a full login/select/fetch, unencrypted.
openssl s_client -crlf -connect 127.0.0.1:143

# Implicit TLS.
openssl s_client -connect 127.0.0.1:993

# Just the capabilities of a running server.
printf 'a CAPABILITY\r\nb LOGOUT\r\n' | openssl s_client -quiet -crlf -connect 127.0.0.1:143

Illustrative session output:

* OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN IDLE MOVE UIDPLUS LITERAL+ CHILDREN] Ferroma IMAP4rev1 ready
a LOGIN [email protected] "…"
a OK LOGIN completed
b LIST "" "*"
* LIST (\HasNoChildren) "/" "INBOX"
* LIST (\HasNoChildren \Sent) "/" "Sent"
* LIST (\HasNoChildren \Drafts) "/" "Drafts"
* LIST (\HasNoChildren) "/" "Archive"
* LIST (\HasChildren) "/" "Archive/2026"
b OK LIST completed
c SELECT INBOX
* 412 EXISTS
* 3 RECENT
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted
* OK [UIDVALIDITY 1] UIDs valid
* OK [UIDNEXT 118] Predicted next UID
c OK [READ-WRITE] SELECT completed
d UID FETCH 117 (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (SUBJECT FROM)])
* 117 FETCH (UID 117 FLAGS (\Seen) RFC822.SIZE 24831 INTERNALDATE "16-Sep-2026 09:12:44 +0000" BODY[HEADER.FIELDS (SUBJECT FROM)] {68}
Subject: Invoice for September
From: Bob <[email protected]>
)
d OK UID FETCH completed
e LOGOUT
* BYE Ferroma IMAP4rev1 server signing off
e OK LOGOUT completed

For "IMAP login fails", see deployment.md §11.

13. Related documents#