This is a technical blog post about standards, complexity and even a little bit about fashion and how it changes over time.
What you need to know:
Inboxmay change case to
INBOX(just a cosmetic change)
It’s always a hard choice to enforce change on people, but this upgrade is important to make things less complex and more robust for everybody in the future. For the first time, IMAP users will be able to create folder hierarchies that exactly match the FastMail web interface.
Our testing with popular clients and other major providers shows that these changes match the most common way of doing folder listings, which provides best compatibility and a good IMAP client experience for our users.
If you want to understand the gory detail of why we need to do this, read on.
The IMAP protocol is a whole lot of different specifications, but the key specification that defines the base protocol is RFC 3501, written by Mark Crispin and released in 2003. It’s a revision of the earlier RFC 2060 from 1996.
There were 114 changes/clarifications from RFC 2060 to RFC 3501, making it one of the most edited of all RFCs.
Like many standards created by committees, IMAP is a compromise that tries to be everything to everybody. Some people wanted to address email the way that they addressed newsgroups –
comp.lang.perl.misc, etc. Some people wanted to address it they way they addressed files on disk –
So IMAP supports arbitrary hierarchy separators. In practice only
. appear to be common, though I’m sure there are still systems out there using
which brings its own special hell in C-like programming languages which use
as an escape character (as anybody who’s ever programmed for Windows will know).
Some operating systems use case significant file systems, others don’t. So on some systems,
Readme.txt are the same file, but on others they aren’t. Of course IMAP chose to leave case sensitivity up to the implementation… sort of.
There’s one thing that IMAP didn’t leave up to the implementation. The special folder INBOX must always exist, and must not be case sensitive… sort of:
5.1. Mailbox Naming
The case-insensitive mailbox name INBOX is a special name reserved to mean "the primary mailbox for this user on this server". The interpretation of all other names is implementation-dependent. In particular, this specification takes no position on case sensitivity in non-INBOX mailbox names. Some server implementations are fully case-sensitive; others preserve case of a newly-created name but otherwise are case-insensitive; and yet others coerce names to a particular case.
6.3.8. LIST Command
The special name INBOX is included in the output from LIST, if INBOX is supported by this server for this user and if the uppercase string "INBOX" matches the interpreted reference and mailbox name arguments with wildcards as described above. The criteria for omitting INBOX is whether SELECT INBOX will return failure; it is not relevant whether the user's real INBOX resides on this or some other server.
So INBOX is case insignificant for SELECT, but it only has to show up in LIST output if the upper case version matches.
INBOX is even special when renaming a folder:
6.3.5. RENAME Command
... If the name has inferior hierarchical names, then the inferior hierarchical names MUST also be renamed ... Renaming INBOX is permitted, and has special behavior. It moves all messages in INBOX to a new mailbox with the given name, leaving INBOX empty. If the server implementation supports inferior hierarchical names of INBOX, these are unaffected by a rename of INBOX.
All other folders MUST rename their hierarchical inferior names (aka child mailboxes), but INBOX explicitly does the opposite.
When FastMail was started in 1999, RFC 3501 didn’t even exist yet, IMAP was still RFC 2060, and the netnews naming convention (dots as the separator) was all the rage. Cyrus IMAPd didn’t support the "altnamespace" yet , so folder names were all hierarchical inferiors of INBOX. This layout made sense at the time, allowing the user’s email to be kept neatly separate from the shared email "bulletin boards" of Carnegie Mellon University’s system.
We also didn’t have support for, or certainly didn’t choose to use,
/ as a hierarchy separator. This means that
. hasn’t been available to use in usernames or folder names at FastMail – and we’ve used hacks like allowing
_ in usernames and ‘^’ in folders to mean
. which has caused many bugs over the years. Even when we switch to
/, those issues won’t entirely go away, because subdomain addressing and web hosting still need a single name, and
. is still the domain name system’s separator.
Anyway, the standard
mail.messagingengine.com:993 IMAP server presents mailboxes as:
INBOX INBOX.Archive INBOX.Drafts INBOX.Sent INBOX.Trash
But it looks kind of ugly on systems where users only have their own email, and they want everything as top level folders:
INBOX Archive Drafts Sent Trash
Smarter clients recognise that INBOX is the top level namespace (there’s a facility in the protocol to discover namespaces), and transplant the subfolders to the top level. The FastMail web interface is one of these. But ALLCAPS INBOX is kind of ugly, so the FastMail interface also chose to translate
Inbox and back again. Then users also wanted to still be able to have subfolders of Inbox the way they can have subfolders of anything else:
Inbox Inbox/ProjectX Archive Archive/ProjectX Drafts Sent Sent/ProjectX Trash
So we made the interface show
INBOX.Inbox.ProjectX as a subfolder of Inbox, and everyone was happy. For a while…
I’ve written about altnamespace a bit already in the cross domain sharing post.
The alternative namespace presents the user’s top level folders alongside INBOX rather than as inferiors. It’s a configuration option to Cyrus IMAPd which was released in the early 2000s, after had already made our choice of configuration.
Until recently, Cyrus IMAPd’s handling of INBOX subfolders via Alt Namespace has been very problematic. You could wind up with duplicate INBOX folders and broken listings, which was a reason for not making it the default for new users.
I spent a lot of time workshopping our options, and in the end the RFC requirement that ALLCAPS INBOX had to match LIST and the fact that every other server always returns it as INBOX in that case convinced me that the folders really had to be subfolders of ALLCAPS INBOX, rather than using Inbox as the name. Also, any other spelling of INBOX needed to not be allowed at the top level, because of the case insensivity rule.
So I created another prefix, "Alt Folders" (actually it’s configurable), and names which won’t be valid at the top level get translated under there. You can read more about exactly how the name mappings work in my mailing list post written at the time.
This was the right choice, the compatible choice, everything good – except…
Unfortunately, all the FastMail users’ subfolders of Inbox in the web interface are
INBOX.INBOX.foo, because we chose that to be prettier back when we first implemented the option.
We could have built more layers of hacks on top of this, but the right answer is to rename those folders to be
INBOX.INBOX.foo so they appear to the IMAP client as
INBOX/foo rather than
There is now a per-account hidden setting inside the FastMail client so that the web interface uses the
INBOX.INBOX path for Inbox subfolders, and we are migrating users over and renaming their folders as we go. All users without subfolders and new accounts will be switched first, and then the users who require folder renames will be done.
The most complex cases are a handful of users who have folders both called
INBOX.Inbox.foo in their account. A naive rename will fail in those cases, and we may wind up contacting these users individually.
Fastmail customer, Xavier, gets the most out of his email account by using Fastmail Identities, Aliases, and Masked Email!
Fastmail Folders are the perfect organization tool for customers that love an empty inbox. Moving messages into customized folders is a quick way to keep your inbox tidy.
Fastmail CTO, Rik Signes, and Backend Team Lead, Matthew Horsfall, joined world-class open-source developers at the annual Perl Toolchain Summit.