Copyright © 1999–2016 FastMail Pty Ltd
SMTP stage spam checks
One of the most powerful ways of stopping spam from entering your mailbox is to stop it from being received in the first place!
The way email is transmitted on the internet is via the SMTP protocol. We apply a number of checks at the SMTP stage to stop spammers.
RBLs (real-time blackhole lists) and RHSBLs (right hand side blackhole lists) are a very common way to detect and stop known spam sending machines. The main problem with RBLs is that they often have "false positives" and block legitimate systems, incorrectly causing email to be bounced back to senders. To stop that happening, FastMail minimises the use of RBLs and RHSBLs for outright blocking.
We use only one RBL for blocking, the SpamHaus XBL. The XBL is a highly accurate block list that lists the IPs of machines with known trojans and proxies. Independent testing shows that the ZEN RBL (of which XBL is a part) has a high block rate, with basically no false positives.
We use a few RHSBLs for blocking the hostname or helo string of connecting servers. RHSBLs usually have lower false positives because the domains tend to have to be explicitly added or require multiple collected reports, unlike RBLs which can easily auto-collect IPs based even on single user reports in some cases.
A number of other RBLs and RHSBLs/URIBLs are used during the spam checking phase that occurs after email is received, but these RBLs never outright block email; they're used as part of an overall scoring heuristic along with many other factors to filter spam into your Spam folder.
Address enumeration detection
Address enumeration detection is designed to stop other people trying to find what email addresses are valid at FastMail. It does this by detecting attempts to send to many different, but similar, email addresses in a short period of time from a non-email server host.
Address enumeration detection provides a defence against spammers trying to discover valid email addresses simply by trying many different addresses over and over.
When a server is transferring email from one system to another, there's a series of conventions they have to follow defined in the SMTP RFC 2821 specification. Spam bots are often poorly implemented, or try and use tricks to work around these conventions. Many spam bots can be found and blocked by detecting some of the common violations.
Custom spambot detection
Through our extensive, long-term experience running an email service, we've been able to closely analyse how spam bots try to deliver email to our servers. By using this data, we're able to pre-emptively detect spam bots soon after they connect, and block them from delivering their spam email at all.
Sending host rate limiting
Using a number of heuristics, we limit the rate that servers can send email to our servers. Again through our extensive, long-term experience running an email service, we've been able to tune these heuristics to minimise any effect on legitimate sending services, while significantly limiting the ability of spammers to deliver spam emails.
Greylisting is a process designed to detect and stop spam being accepted from poorly written sending email servers, which almost all spam bots are. One of the main concerns users have with greylisting is that poor implementations will often delay all email. The FastMail implementation uses a number of tweaks to ensure that legitimate email is delivered with no delay, while spam email is delayed or stopped from delivery entirely.
Greylisting works by returning a temporary failure response (
4xx code) to the first attempt to deliver an email, but accepting it on the second attempt. Every proper email server will attempt to redeliver a message after a temporary failure response, while currently almost all spam software does not, thus effectively blocking the emails from the spam software.
To avoid delaying proper email servers, our greylisting implementation performs a number of checks on a connection before applying the greylisting policy:
- We only greylist hosts that appear to be dialup/dsl hosts of some sort, or hosts that don't have any valid reverse DNS. This ensures that the vast majority of email servers are immediately not subject to greylisting, and their email is not delayed.
- If a host has been greylisted, and it successfully passes greylisting twice in a 24 hours period (e.g. it correctly attempts to re-deliver a piece of email twice in 24 hours), then that host is whitelisted and not subject to greylisting for the next 24 hours. If it continues to deliver emails, each new delivery will extend the whitelist period. This means that any real email servers (a real email server will always retry) connected via dialup/dsl will quickly be whitelisted and not subject to email delays.
If a host opens an SMTP session with a HELO that is not an IP address and is not the same reverse DNS as its connecting IP, but the forward DNS of the name does resolve to the connecting IP, then that host is not subject to greylisting.
For example: The machine at IP
18.104.22.168connects to us. The reverse DNS for
206-223-169-73.beanfield.net, which looks like a common dialup/dsl IP name, and would be a candidate for greylisting. However, the machine advertises itself to us with a
HELO mx3.hub.orgline. Doing a forward lookup of
mx3.hub.orggives the IP
22.214.171.124, which is the same as the connecting IP, so we exclude it from greylisting
- If the SMTP MAIL FROM address passed in the SMTP session is in the user's address book, then the email is not greylisted.
When combined, these features provide an excellent balance of greylisting hosts which should not be sending email, and allowing those hosts which should be sending email to get their email straight through.
Additionally, to help with the tracking of any problems, once a message passes greylisting and is accepted, a new header is added called
X-Spam-greylist. This header tells you how many seconds the email was delayed and whether that host has been whitelisted for 24 hours. (Technically, the delay figure actually indicates the length of the last delay for the ip/sender/recipient combination, so in the case of multiple emails from the same person, to the same person, from the same machine in a short time period, the figure will be a bit messy and hard to calculate.)
If you feel that email is being delayed by greylisting, please check the headers of your email for the
X-Spam-greylist header first. If it's not present, then the email was not delayed by greylisting, and it was probably just held up on the sender's side, something beyond our control.
Forwarding and custom domains
The result of all these tests is that most spam is blocked before it even enters our systems. However, we can only run these tests when the email is delivered directly to us, not forwarded on from another mail server. If you use your own domain, we strongly encourage you to host your email directly with us rather than having it forwarded on from your registrar, as this will enable us to be much more effective at filtering out spam.