I wanted to do a bit of a followup to my earlier SenderID post both to better capture the comments and make another observation.  First I wanted to bring John Scarrow’s comment to the front of the discussion.  John is the Microsoft General Manager responsible for the various Windows/Windows Live safety features so this is an authoritative position from Hotmail on SenderID, SPF, and DKIM:

Microsoft continues to be committed to SenderID as well as DKIM.  We find them to be complementary in many ways and current support both of them on inbound mail.  SendierID and SPF currently have the largest adoption with our senders and thus we check them first.  SendierID record have the added benefit of improved Phishing protection as we verify the address that is shown to the customer to match the sending domain.   If this fails we then check against DKIM.  Checking against DKIM helps us reduce false positives that may arise from email forwarding.  Although adoption of DKIM has been slower due to the complexity of deployment for mailers we continue to urge senders to adopt both Sender ID and DKIM to insure the highest quality delivery as well as Phishing protection.

Thanks for the clarification John!

As I noted in my previous post one of the issues I have with current SPF/SenderID implementation is that few senders specify a hard FAIL (“-all”) when a message is found to have come from an IP address other than those specified in the DNS record for the alleged sender.  Many sites seem to be specifying either SOFTFAIL or NEUTRAL to be returned if the mail didn’t come from their servers.  A SPAM filter would tend to treat SOFTFAIL as a big strike against the message, and if it found anything else suspicious it would then flag the message as SPAM.  NEUTRAL is supposed to mean the same thing as if there was no SPF record at all, but I imagine different SPAM filters treat it quite differently.  Some treat it the same as SOFTFAIL , some treat it as if there was no SPF record, and others give it special significance.   They each have their own experience built-in as to how often each kind of response is really SPAM or not.  On the other hand, an SPF PASS is a big vote of confidence.  So big in fact that I find most SPAM that makes it through to my Inbox has passed SPF/SenderID checking.  This makes a lot of sense.  You don’t want “your prescription has shipped” email from MedCo to end up in your Junk folder, yet the same mail wording sent from a sender who can’t be verified likely triggers filters to flag it as SPAM.

Interestingly both CVS and Walgreens specify SOFTFAIL in their SPF policies.  In a world where the majority of SPAM is either about sex or drugs, why wouldn’t a pharmacy protect its customers by being extremely clear about making it easy to verify if mail sent under their name really came from them?  Well, Walgreens uses DKIM so if a receiver checks DKIM they are in good shape.  But CVS doesn’t.  CVS in fact confuses me, and I’ll get to why in a moment.

As the recent data loss by Epsilon demonstrates, most large companies use Email Service Providers (ESP) to send bulk emails on their behalf.  So that loyalty program newsletter you get from Marriott Rewards or customer newsletter from Tivo don’t come from their servers they come from Epsilon.  Of course they don’t say Epsilon on them, they appear to come from Marriott or Tivo.  And that is where the SOFTFAIL comes in.  Marriott (or Tivo or Walgreens or CVS) has to either include the IP addresses for the servers of anyone allowed to send email on their behalf as part of their SPF records or they have to SOFTFAIL so that mail from one of their ESPs isn’t rejected out of hand as a forgery.  I imagine they all find that keeping their DNS record in sync with all the potential senders of email on their behalf unmanageable and so simply choose not to try to control it.  What I find interesting about CVS is that they DO include the IP addresses of their ESPs’ servers in the SPF record, and yet they still choose SOFTFAIL when there is no match.  Perhaps they have discovered from experience that they can’t keep the SPF record up to date, but they can at least give SPAM filters a hint that mail from certain ESPs is legitimate.

It seems to me that big Enterprises should get their act together and either go to including all potential legitimate sending servers of email on their behalf in their SPF record with a FAIL for anyone who isn’t on the list, or use DKIM when they can’t validate the server itself.  But what about small businesses?  That is more difficult and a personal experience demonstrates why.  I recently examined a piece of email from the car service I use for airport transportation that had ended up in my Junk folder.  This car service has a Hotmail email address, but they use a cloud service for handling reservations.  The cloud service sends out a reservation confirmation, which of course is made to appear to come from the car service’s Hotmail address.  Hotmail’s SOFTFAIL policy gave this email a legitimate chance of ending up in my Inbox, while a FAIL policy would have certainly sent it to the Junk folder.  (As it turns out the mail went to my Junk folder anyway because the mail is nothing more than an image of a reservation form, and a piece of mail containing nothing more than an image that is from an unverifiable email address is a red flag for SPAM).  Obviously because the car service doesn’t use its own domain for email none of the existing mail authentication tools (SPF, SenderID, DKIM) can be set up to work properly.  This demonstrates the complexity of the problem, particularly as companies outsource more of their operations to cloud application services.

I know that is a bit of rambling on this topic but it really leads to the point that the industry still seems to be at the stage of attacking the SPAM problem with lots of fly swatters rather than the thermonuclear device that is called for.  There is no comprehensive solution, and even the solutions that do exist are poorly utilized.  I have to wonder if we’ll have this problem solved by the time we get to the next decade.