Data resilience at Fastmail

(or: How I Learned to Stop Worrying and Love the Backups)

This is a story of a close call, where users lost entire mailboxes due to a bug, but backups allowed us to bring everything back.

We’ve been doing this quite a while

This is a blog post about principles. Principles rarely change, which is why you can read a blog post I wrote over 10 years ago and the “three types of data” is still as fresh as the day I wrote it.

This is also a blog post about specifics of implementation. We don’t change that very fast either, which is why almost everything else in that post is also still correct! We’re still using git, we’re still using Debian. Some mechanics about the Makefiles and search database compaction have changed, but we still do them too.

And importantly, we still store data replicated across multiple machines, but today that guarantee is even stronger. At least one of those machines is in a different physical location (currently Philadelphia, St Louis, or Amsterdam). The default configuration is two machines in your primary location plus a third machine offsite. We have sufficient compute resources to continue operating even if our biggest site was temporarily offline.

Here’s another thing we’ve been running for a long time — an in-house backup system for Cyrus IMAP, the server where most of our customer data is stored. We blogged in 2015 about a replacement system, which was eventually set aside and we kept improving the in-house system instead. That system is now available as part of the open-source Cyrus project. It will feature in this story.

A nasty bug

In April 2026, Fastmail decided to segment accounts into separate “stores” (individual Cyrus instances) for EU and non-EU customers, in advance of setting up our Amsterdam location.

As part of this, most EU customers were migrated to newly created stores, using the existing Cyrus-replication based method for moving users. This mechanism is regularly used to balance storage with automatic moves, as well as to move users together when accounts are combined — however in this case, a much higher volume of moves were performed over the course of about a month.

During this time, 8 user accounts had a subset of their folders entirely wiped by a bug. In every case that included their INBOX, rendering the entire account inaccessible. As is usual with software bugs, it was caused by the interactions of multiple behaviours which seemed sensible in isolation, but interacted badly.

For those interested in the full technical details of what happened, read on. Otherwise feel free to skip ahead to the next section to see how backups ensured no data was lost.

The proximate cause was a well-intentioned change to the no-copyback replication mode, to forcefully clear out any remote state for the user being synced. This was in pull request 5513. We only use no-copyback mode at Fastmail when we’re moving a user to a new store. There isn’t going to be anything there for the user yet, so being more aggressive makes sense.

The bug worked in conjunction with another change from pull request 4852, which ensured that when syncing a user, we follow its rename history.

Cyrus keeps tombstones with old names for every mailbox’s uniqueid for some time, to allow renames to be detected during regular replication, and to ensure we never re-use a UIDVALIDITY if the invariants that IMAP requires cannot be satisfied.

Individually, these all make complete sense. Combine them and there is a big problem — reused usernames. We made a decision way back in the early days to use the primary email address directly as the username in Cyrus, so folder sharing looked right via IMAP. This has caused some pain over the years! We reserve “used” usernames on our own domains for a while, but if you’re creating accounts in your own domain, then immediate re-use becomes possible.

Consider this case: user A has been renamed to B, and then a new different user A is created — same username, same store. We sync user A to a different server. This happens fine, the integrity checks pass, all good so far.

Later, we try to sync user B. The name history of B’s folders includes folder names for user A. The sync for user B reaches into the “new” user A on the destination server and wipes any folders with those names. For these 8 accounts that’s exactly what happened — the move system wiped entire folders on all three of the target store’s copies!

The initial attempt at a fix was in pull request 5961, which didn’t actually stop the issue. It was based on an incorrect assumption that the user was coming from a separate store, and the tombstones were on the destination rather than the other way around. Due to the incompleteness of the fix, one user who had not been fully moved before the initial restore from backup was corrupted again, and this was only finally fixed with pull request 5988.

This only happened for Fastmail users who had created a user, renamed it to something else, then re-created the same original username. And of those, only users who were moved during the period when the bug existed.

Backups save the day

Fastmail runs cyrus-imapd with a slots and stores layout as documented on our help pages. The replication protocol has multiple safeguards built in, checksums on everything, and regular rolling replication can never wipe something immediately, only mark it for later cleanup in a week or so.

This protects against damage by regular users, and most possible bugs. This particular bug cut straight through those protections though, by using the replication protocol to immediately wipe the folders. The move system needs to be able to delete things, because as its final step after ensuring that a user has been successfully migrated, it cleans up the copy on the source store!

Fastmail also has a backup system, which stores all the data and metadata in an append-only incremental backup with occasional repacking, meaning that all data from the most recent backup was safe. We regularly back up each user every few hours, and also immediately before and after moving their account between servers, which is what saved us here!

All those missing folders were restored from backup, recovering all emails for all but one user.

For that one user, 5 sent emails were unrecoverable from backups. Luckily, identical copies of those sent emails were recoverable from the #jmapsubmission temporary files inside Cyrus, using some additional custom-built tooling. Ultimately, no email was lost.

Future protection

The solution in 5988 includes a test case which reproduces the exact pattern that was actually happening during those moves, which means we will detect if we write a similar bug again.

We’re also considering moving the replication protocol to be more based on the user’s inbox uniqueid for everything rather than mboxname, which would have avoided this problem entirely, but that’s a bigger change that will likely be a couple of years away.

Regardless of the ongoing improvements, the core principle which saved us here was exactly what I wrote back in 2014: “Primary Copy - back it up. Replicate it. Everything you can to ensure it is never lost. This stuff is gold”. By building defence in depth, we ensured that even this nasty bug didn’t lead to us losing the data we had taken responsibility for.