One of the subtle changes in Microsoft’s new outlook.com replacement for Hotmail is that the messages about suspicious mail have changed.  In Hotmail messages are simply described as suspicious, in outlook.com it is now clearly stated that the “sender failed our fraud detection checks”.  Not only that, whereas Hotmail displays this text in “warning yellow” outlook.com displays it in “danger red”.  So when I switched to outlook.com and noticed that many messages in my Inbox were now labeled as failing fraud detection (and only showing up in my Inbox because I’d placed the sender’s domain on my safe list) I decided to investigate further.

Before I dig inlet’s discuss the meta issue.  The world is awash in SPAM.  While SPAM started out as being primarily (semi-)legitimate Unsolicited Commercial Email (UCE) it has now become primarily a distribution mechanism for Malware (usually by getting you to follow a link to a malware distribution website) and Phishing scams.  Attempts to fight SPAM have really focused on two things, one is attempting to determine if the mail is from a legitimate sender and the other is content analysis of messages to see if they are SPAM-like.    The former is an architectural nightmare given the origins of the Internet, and the latter is prone to all kinds of failure.  For example, a message from a wife to her husband of “Big plans.  Don’t forget to take your Viagra before leaving work :-)” would likely be flagged as SPAM by content filters.   To combat this problem mail systems contain overrides such as ignoring content filters if the sender is one of your contacts or on your “safe sender” list.  But this places a burden on the user of scanning their junk folder periodically to see if something has ended up there inappropriately and adding the sender to their safe sender list so future mails from the sender go to the Inbox.  And worse, it means that mail systems have to allow potential Phishing and Malicious emails into the Junk folder just in case they are actually legitimate mail.

To really solve the SPAM problem you first have to solve the problem of determining the legitimacy of senders.  Unfortunately since the Internet was designed as a research project authentication of email senders and messages was not designed in, and we’ve been paying the price ever since it was opened up for general use.  While dreams of every email message being authenticated may be just that, dreams, various techniques for allowing senders who wish to authenticate their mail have been proposed and somewhat implemented.  The problem is, not enough senders are properly and completely using these techniques.

If the bulk of legitimate senders were fully using already existing authentication techniques (SPF, Sender-ID, DKIM, DMARC), then SPAM-filtering systems could get much more aggressive about just deleting SPAM rather than putting it in your Junk folder for you to look at.  For example, I get some email from my bank that is fully authenticated and some that isn’t.  Because some isn’t, SPAM-filters can’t really be sure of the difference between real mail from my bank and phishing mail that looks like it is from my bank.  So occasionally real mail from my bank goes into my Junk folder and phishing mail that looks like it’s from my bank ends up there too.  Occasionally phishing mail actually makes it into my Inbox.  If every mail from my bank was known to be authenticated then the SPAM filters could more easily determine what a phishing attempt was and make sure it never reached me.

So why does my bank, or any other legitimate sender, ever send an unauthenticated mail?  Because businesses, even small businesses, are surprisingly complicated.  In addition to their own email systems, almost all use third-party bulk mailing services or allow partners to send mail on their behalf.  So if you look at their SPF records in DNS, which is the most widely used authentication technique, you find they usually specify their own mail servers as legitimate sources of email from them and then “~all”.  “~all” is also known as “soft-fail” meaning that what they really are saying is “there are other legitimate parties sending mail on our behalf but we don’t know who they are, so we can’t help you decide which are legitimate and which aren’t”.  And this is why I find so many messages being marked as failing fraud detection checks, and why so many legitimate mails go into the junk folder; They hit the “soft-fail” condition when the actual sending server is evaluated against the purported sender’s SPF record.

Why not add the Third-Party’s servers to the sender’s SPF record, essentially authorizing them to send mail on the sender’s behalf?  Let’s say you create a customer advisory board for your product and want to be able to send out notices to the group.  Maybe you even want it to be a discussion list.  you go out and find an inexpensive third-party bulk email service you can use for this.  How do you, some junior product manager buried 12 levels deep in the organization, get IT to change the corporate SPF record to make the Third-Party a legitimate sender of email on the company’s behalf?  You can’t.  They won’t.  They’ll laugh at you.  Really.  Not just because they don’t want to change the corporate DNS entry every time an employee goes outside the box, but also because they can’t authenticate the Third-Party just for you.  Adding them to the SPF record means receivers will think any mail coming from the Third-Party claiming to be from your organization are legitimate.  And without a corporate-level agreement with the Third-Party that violates corporate security.  For a small business things are similar except that the real problem is that “what’s an SPF record?” is the problem.  In other words, while their ISP or IT consultant probably created an SPF record for their primary mail server no one inside their organization even knows what an SPF record is; Or who to contact to change it.  Instead they tell people to add “foo@ourlittleorganization.com” to their safe sender list so mails don’t go to the junk folder.

For those who want a real example of what I’m talking about, here is one from an organization that should be sophisticated enough to address this issue.  The IEEE Communications Society.  Here is how the email looks:

So, notice the “This sender failed our fraud detection checks” message.  When we view the message source we find that this message was sent with a service called Magenta Mail, which you can see here (though you may need a magnifying glass):

And when you look at the SPF record for comsoc.org you find no mention of the Magenta Mail servers and a soft-fail (~all) indicator as the catch-all case for this:

v=spf1 ip4:140.98.0.0/16 a:conan.comsoc.org a:cmsc-ems.ieee.org
a:cmsc-ems2.ieee.org a:cmsc-ems3.ieee.org a:comsoc-listserv.ieee.org
mx:hormel.ieee.org mx:lemroh.ieee.org include:ieee.org ~all

Most mail systems will throw this into the Junk folder unless you add comsoc.org to your safe sender list.  And even when it is on the safe sender list Hotmail, I mean outlook.com, will warn you it is suspicious or potentially fraudulent.

When I switched to outlook.com and started noticing that a lot of mail from a few organizations were marked as “failing fraud detection” I investigated  and found many were using third-party mailer  Constant Contact.  I contacted the organizations, as well as Constant Contact, about this.  According to Constant Contact they provide a feature called Constant Contact Authentication to address the problem that their user’s have with changing their own SPF records.  They also mentioned that many of their users don’t use this feature which is exactly what I was seeing.  They are going to look at ways to further encourage users to turn it on.  BTW, Constant Contact Authentication works by turning the problem on its head and giving you a new domain that authenticated emails come from.    They also document how you can make them an authenticated sender of email for your organization (via SPF, Sender-ID, and DKIM) as an alternative.  And Constant Contact’s website says that eventually all their users will either have to use Constant Contact Authentication or make them an authenticated sender.

If all bulk mailing services went down the path of  requiring their clients to use an authentication mechanism it would represent a huge step forward in cleaning up the SPAM mess.  A legitimate service like Constant Contact might still be used for the traditional UCE type of SPAM, but you could trust that it was safe to use the unsubscribe link.  And abuses reported to the authorities could be traced back to the actual sender.  Much of this mail, like the IEEE ComSoc example above, falls into what Microsoft now calls the “GreyMail” category.  Microsoft’s tools for managing GreyMail would work more effectively with proper authentication.

If email authentication was really widespread then mail system’s SPAM filters could adopt a more aggressive approach.  They would only put authenticated email into your Inbox.  And they would be more comfortable just throwing away potential phishing and other harmful mails rather than putting them in the Junk folder.  With no unauthenticated mail in your Inbox, and a very small number of mails in your Junk folder, the Internet would be a much safer place.

So here is a little call to action.  First, if you are a user of a bulk mailing service pleasemake sure your mails are being properly authenticated.  Second, if you get unauthenticated mails from a legitimate sender please contact them and ask them to fix this problem.  One of the organizations I contacted turned on Constant Contact Authentication the day I brought the problem to their attention.  Another is taking a look at it.  This suggests that with a little bit more user pressure we could make email much better, much sooner.