This is the sixth post in the FastMail 2015 Advent Calendar. Stay tuned for another post tomorrow.
This post is about technical stuff, because that’s what I do, but it’s relatively
short and simple compared to my usual posts!
Being able to provide the quality of service that we do for such a low price is partially about big things like major efficiency improvements to our server software, but it’s also about optimising small things.
I blogged about our servers in last year’s advent calendar. If you look at the section for one of the older servers (imap14) you’ll notice that the search partition was less than 50% full, yet the spool
partitions were quite full. This pattern was repeated across all our 2U servers.
The OS partitions were also 1TB and had tons of free space. This was carved out of the 16TB RAID6, leaving room for only 15 slots.
In the process of rebuilding machines for Amsterdam, we decided to make use of that
free space, and reconfigure them, so instead of the RAID being:
RAID r6:[ 0 1 2 3 4 5 6 7 8 9 ] r1:[ 10 11 ] VOLUME [ 1TB, 15TB ] [ 2TB ]
The RAID1 has been moved to the first two disks, and OS is shared with search instead of spool:
RAID r1:[ 0 1 ] r6:[ 2 3 4 5 6 7 8 9 10 11 ] VOLUME [ 500Gb 1.5TB ] [ 16TB ]
Giving 16 slots instead of 15. It also means that the OS is shared with search rather than spool. With our patterns of IO usage, this makes the OS more responsive. So a machine now looks like:
/dev/sda1 15995848 2393272 12783376 16% / /dev/sda5 462344068 9734232 429101052 3% /local /dev/mapper/sdb1 1441724500 736678888 705029228 52% /mnt/si10search /dev/mapper/sdc1 961104216 748308608 212779224 78% /mnt/si10t01 /dev/mapper/sdc2 961105240 793163232 167925624 83% /mnt/si10t02 /dev/mapper/sdc3 961105240 812993724 148095132 85% /mnt/si10t03 /dev/mapper/sdc4 961105240 835672212 125416644 87% /mnt/si10t04 /dev/mapper/sdc5 961105240 820427704 140661152 86% /mnt/si10t05 /dev/mapper/sdc6 961105240 789696468 171392388 83% /mnt/si10t06 /dev/mapper/sdc7 961105240 821484964 139603892 86% /mnt/si10t07 /dev/mapper/sdc8 961105240 737563548 223525308 77% /mnt/si10t08 /dev/mapper/sdc9 961105240 759814248 201274608 80% /mnt/si10t09 /dev/mapper/sdc10 961105240 821480792 139608064 86% /mnt/si10t10 /dev/mapper/sdc11 961104216 787875416 173212416 82% /mnt/si10t11 /dev/mapper/sdc12 961105240 754964180 206124676 79% /mnt/si10t12 /dev/mapper/sdc13 961105240 765869784 195219072 80% /mnt/si10t13 /dev/mapper/sdc14 961104216 73364 961014468 1% /mnt/si10t14 /dev/mapper/sdc15 961105240 73364 961015492 1% /mnt/si10t15 /dev/mapper/sdc16 961103192 73364 961013444 1% /mnt/si10t16 /dev/md0 384320508 106689472 277614652 28% /mnt/si10ssd
(search and spool partitions are encrypted with LUKS, the SSDs have on-drive encryption, so they’re just software RAID1 without OS level encryption as well)
We’re slowly moving all the old machines over to this layout. Right now imap24 and imap29 have been reconfigured, and are syncing up new replicas from imap25 and imap26. That should be finished in about a week, then I’ll empty those two servers and reconfigure the drives.
There are almost always long term migrations like this happening in the background. The latest IMAP hardware has 4TB big drives and 800GB SSDs along with 192GB RAM and some serious CPU, and we add new servers every few months and move users to them – but it’s still good to get the best we can out of the older machines.
Fastmail customer, Xavier, gets the most out of his email account by using Fastmail Identities, Aliases, and Masked Email!
Fastmail Folders are the perfect organization tool for customers that love an empty inbox. Moving messages into customized folders is a quick way to keep your inbox tidy.
Fastmail CTO, Rik Signes, and Backend Team Lead, Matthew Horsfall, joined world-class open-source developers at the annual Perl Toolchain Summit.