You can’t defeat SPAM when legitimate mail looks like SPAM!

One of the subtle changes in Microsoft’s new replacement for Hotmail is that the messages about suspicious mail have changed.  In Hotmail messages are simply described as suspicious, in it is now clearly stated that the “sender failed our fraud detection checks”.  Not only that, whereas Hotmail displays this text in “warning yellow” displays it in “danger red”.  So when I switched to and noticed that many messages in my Inbox were now labeled as failing fraud detection (and only showing up in my Inbox because I’d placed the sender’s domain on my safe list) I decided to investigate further.

Before I dig inlet’s discuss the meta issue.  The world is awash in SPAM.  While SPAM started out as being primarily (semi-)legitimate Unsolicited Commercial Email (UCE) it has now become primarily a distribution mechanism for Malware (usually by getting you to follow a link to a malware distribution website) and Phishing scams.  Attempts to fight SPAM have really focused on two things, one is attempting to determine if the mail is from a legitimate sender and the other is content analysis of messages to see if they are SPAM-like.    The former is an architectural nightmare given the origins of the Internet, and the latter is prone to all kinds of failure.  For example, a message from a wife to her husband of “Big plans.  Don’t forget to take your Viagra before leaving work :-)” would likely be flagged as SPAM by content filters.   To combat this problem mail systems contain overrides such as ignoring content filters if the sender is one of your contacts or on your “safe sender” list.  But this places a burden on the user of scanning their junk folder periodically to see if something has ended up there inappropriately and adding the sender to their safe sender list so future mails from the sender go to the Inbox.  And worse, it means that mail systems have to allow potential Phishing and Malicious emails into the Junk folder just in case they are actually legitimate mail.

To really solve the SPAM problem you first have to solve the problem of determining the legitimacy of senders.  Unfortunately since the Internet was designed as a research project authentication of email senders and messages was not designed in, and we’ve been paying the price ever since it was opened up for general use.  While dreams of every email message being authenticated may be just that, dreams, various techniques for allowing senders who wish to authenticate their mail have been proposed and somewhat implemented.  The problem is, not enough senders are properly and completely using these techniques.

If the bulk of legitimate senders were fully using already existing authentication techniques (SPF, Sender-ID, DKIM, DMARC), then SPAM-filtering systems could get much more aggressive about just deleting SPAM rather than putting it in your Junk folder for you to look at.  For example, I get some email from my bank that is fully authenticated and some that isn’t.  Because some isn’t, SPAM-filters can’t really be sure of the difference between real mail from my bank and phishing mail that looks like it is from my bank.  So occasionally real mail from my bank goes into my Junk folder and phishing mail that looks like it’s from my bank ends up there too.  Occasionally phishing mail actually makes it into my Inbox.  If every mail from my bank was known to be authenticated then the SPAM filters could more easily determine what a phishing attempt was and make sure it never reached me.

So why does my bank, or any other legitimate sender, ever send an unauthenticated mail?  Because businesses, even small businesses, are surprisingly complicated.  In addition to their own email systems, almost all use third-party bulk mailing services or allow partners to send mail on their behalf.  So if you look at their SPF records in DNS, which is the most widely used authentication technique, you find they usually specify their own mail servers as legitimate sources of email from them and then “~all”.  “~all” is also known as “soft-fail” meaning that what they really are saying is “there are other legitimate parties sending mail on our behalf but we don’t know who they are, so we can’t help you decide which are legitimate and which aren’t”.  And this is why I find so many messages being marked as failing fraud detection checks, and why so many legitimate mails go into the junk folder; They hit the “soft-fail” condition when the actual sending server is evaluated against the purported sender’s SPF record.

Why not add the Third-Party’s servers to the sender’s SPF record, essentially authorizing them to send mail on the sender’s behalf?  Let’s say you create a customer advisory board for your product and want to be able to send out notices to the group.  Maybe you even want it to be a discussion list.  you go out and find an inexpensive third-party bulk email service you can use for this.  How do you, some junior product manager buried 12 levels deep in the organization, get IT to change the corporate SPF record to make the Third-Party a legitimate sender of email on the company’s behalf?  You can’t.  They won’t.  They’ll laugh at you.  Really.  Not just because they don’t want to change the corporate DNS entry every time an employee goes outside the box, but also because they can’t authenticate the Third-Party just for you.  Adding them to the SPF record means receivers will think any mail coming from the Third-Party claiming to be from your organization are legitimate.  And without a corporate-level agreement with the Third-Party that violates corporate security.  For a small business things are similar except that the real problem is that “what’s an SPF record?” is the problem.  In other words, while their ISP or IT consultant probably created an SPF record for their primary mail server no one inside their organization even knows what an SPF record is; Or who to contact to change it.  Instead they tell people to add “” to their safe sender list so mails don’t go to the junk folder.

For those who want a real example of what I’m talking about, here is one from an organization that should be sophisticated enough to address this issue.  The IEEE Communications Society.  Here is how the email looks:

So, notice the “This sender failed our fraud detection checks” message.  When we view the message source we find that this message was sent with a service called Magenta Mail, which you can see here (though you may need a magnifying glass):

And when you look at the SPF record for you find no mention of the Magenta Mail servers and a soft-fail (~all) indicator as the catch-all case for this:

v=spf1 ip4: ~all

Most mail systems will throw this into the Junk folder unless you add to your safe sender list.  And even when it is on the safe sender list Hotmail, I mean, will warn you it is suspicious or potentially fraudulent.

When I switched to and started noticing that a lot of mail from a few organizations were marked as “failing fraud detection” I investigated  and found many were using third-party mailer  Constant Contact.  I contacted the organizations, as well as Constant Contact, about this.  According to Constant Contact they provide a feature called Constant Contact Authentication to address the problem that their user’s have with changing their own SPF records.  They also mentioned that many of their users don’t use this feature which is exactly what I was seeing.  They are going to look at ways to further encourage users to turn it on.  BTW, Constant Contact Authentication works by turning the problem on its head and giving you a new domain that authenticated emails come from.    They also document how you can make them an authenticated sender of email for your organization (via SPF, Sender-ID, and DKIM) as an alternative.  And Constant Contact’s website says that eventually all their users will either have to use Constant Contact Authentication or make them an authenticated sender.

If all bulk mailing services went down the path of  requiring their clients to use an authentication mechanism it would represent a huge step forward in cleaning up the SPAM mess.  A legitimate service like Constant Contact might still be used for the traditional UCE type of SPAM, but you could trust that it was safe to use the unsubscribe link.  And abuses reported to the authorities could be traced back to the actual sender.  Much of this mail, like the IEEE ComSoc example above, falls into what Microsoft now calls the “GreyMail” category.  Microsoft’s tools for managing GreyMail would work more effectively with proper authentication.

If email authentication was really widespread then mail system’s SPAM filters could adopt a more aggressive approach.  They would only put authenticated email into your Inbox.  And they would be more comfortable just throwing away potential phishing and other harmful mails rather than putting them in the Junk folder.  With no unauthenticated mail in your Inbox, and a very small number of mails in your Junk folder, the Internet would be a much safer place.

So here is a little call to action.  First, if you are a user of a bulk mailing service pleasemake sure your mails are being properly authenticated.  Second, if you get unauthenticated mails from a legitimate sender please contact them and ask them to fix this problem.  One of the organizations I contacted turned on Constant Contact Authentication the day I brought the problem to their attention.  Another is taking a look at it.  This suggests that with a little bit more user pressure we could make email much better, much sooner.


This entry was posted in Computer and Internet, Phishing, Privacy, Security and tagged , , , , . Bookmark the permalink.

8 Responses to You can’t defeat SPAM when legitimate mail looks like SPAM!

  1. Rob says:

    Why doesn’t Microsoft support DKI though and only supports their own Caller ID?

    • halberenson says:

      Does Yahoo support Caller ID? DomainKeys is Yahoo’s own technology just as Caller ID was Microsoft’s, and neither is a standard or proposed standard. They were replaced by DKIM and Sender-ID IETF proposals respectively. Then you have to look at Microsoft from two perspectives, the email server software it sells (Exchange and FPE) and rents (Office 365 and FOPE) versus the email service it runs (Hotmail/ Hotmail DOES support DKIM and you can see more on this at Exchange requires a third-party add-in for (generating) DKIM, and of course you can pick a SPAM filtering solution that supports DKIM if you wish.

      I don’t really know why Microsoft hasn’t put native support for DKIM into Exchange et al, but I’ll venture a guess that it is a patent problem. DKIM is covered by a Yahoo patent and it may be that Yahoo’s licensing terms don’t work for Exchange, FPE, and FOPE (just as Sender-ID is covered by a Microsoft patent whose terms don’t work for software licensed under GPLv3, which is why Sender-ID is moribund on the standardization track). By referring you off to a third-party solution Microsoft avoids the patent problem.

  2. Bryan Dunphy says:

    I assume you will find it interesting (and more than a little ironic) that the email subscription message I received regarding this blog posting was itself tagged as Spam by Gmail and I only found it in my junk folder.

    • halberenson says:

      I love it!

      Actually that seems really odd of Google to do given that it appears uses both DKIM and Sender-ID in its notification emails.

  3. Nancy says:

    More irony for you. From MS themselves. I wanted to know how such a flagrant spam email could get through both my email provider and into my client Outlook past my Norton and only today realized because it’s legit. Now if you got an email from INFOW OTLK NA 00 EN CVG GUR TS 2RC PBS 00 PL and the subject line only said SRX1183610931ID-, would YOU open that? I got a few of these about a month back. Today I got one and when I went to my webmail to check it in a safer mode than on my PC, I could see a tad of the first sentence and it turns out it’s the follow up mail to a support call to MS for Office 2010. It turns out that if I opened it, it would show this after the from string I listed above, but I didn’t see that part when it’s in my inbox without widening the From column to an unnatural width. First glance and gut reaction says, “Whoa! NOT opening THAT thing.” So do you think MS, who should know even MORE than IEEE, would make their emails less frightening looking? Granted, MS did all the techie part ok, which is how they passed all the checkpoints, but what about the human checkpoint as well?

  4. John Morris says:

    I don’t think that SPF, DKIM, etc. ‘authenticated’ mail is really the whole solution. About half the spam that makes it through my spam filters comes from hotmail servers.

Comments are closed.