Microsoft Security Essentials has recently come under some criticism for poor test results. The obvious, or maybe not so obvious, question to ask is how much of this is about the performance of Microsoft Security Essentials (MSE) and how much is it about testing methodology? Is MSE really so mediocre or are testers doing Apples vs. Oranges comparisons?
The core problem here is that many test organizations design their test regimes to test Security Suites (“demonstrate their capabilities using all components and protection layers“) and then apply those tests just to MSE rather than to the entire “Microsoft security suite” of which it is part. Why? One key reason is that Microsoft doesn’t explicitly offer a security suite, instead it spreads security capabilities across its products and components. Rather than MSE being the cornerstone of its security efforts, as an anti-malware engine is for a traditional security vendor, for Microsoft MSE is a component that fills in a missing piece in the Windows security effort.
Why is this so important a distinction? Simple, to get the full protection that a third-party suite offers you must be using the other components that Microsoft considers part of its suite. If you don’t use, or don’t test, those components then you will indeed see less protection than you could get from third-party suite. To focus on a prime example, Microsoft has focused many recent security advances on Internet Explorer. MSE does not try to duplicate those efforts, so if you use Firefox or Chrome you don’t get those benefits. Meanwhile a third-party security suite will implement similar advances in a browser neutral way or provide add-ins to bring those capabilities to all popular browsers.
The most important advance in security technology the last few years is the use of Reputation to decide if it is safe to run a program. With a reputation-based system you flip the security problem on its head, running programs you know are safe and either blocking or applying more scrutiny to programs whose safety is unknown or suspect. This helps solve the problem that malware authors can write undetectable malware faster than anti-malware signatures can be updated to detect them. That makes it particularly effective at blocking 0-day attacks, the area where MSE has been doing so poorly in testing.
The reason testing methodology is in question here is that Microsoft splits its use of Reputation over one (Windows XP), two (Windows 7), or three (Windows 8) components. MSE uses Reputation in all cases to decide if an image is safe and should be run without further evaluation or if it should be given closer scrutiny for its malware potential, but it doesn’t block execution just based on an unknown reputation. Microsoft brought reputation-based blocking into the picture with Internet Explorer 9 SmartScreen on Windows 7. With Windows 8 the picture expands even more fully with Windows 8 itself using SmartScreen reputation. Basically Microsoft assumes that you are using IE9 and MSE together if you want the full benefits of reputation-based protection. Use Firefox, Chrome, or another browser and you aren’t using Microsoft’s full security suite. What browser due testing organizations use in their tests? I don’t even think they reveal such details.
These testing methodology issues go beyond the reputation of executables. Microsoft relies on IE’s SmartScreen for URL filtering as well. Security Suite vendors offer their own browser add-ins for URL filtering, so they cover the major browsers and not just IE. And Microsoft assumes more server-based filtering of email or catching bad executables when they are transferred to disk. Security Suite vendors offer “end-point” (i.e., client) email filtering. Test methodologies appear to intentionally try to force a requirement for end-point email SPAM/Malware filtering, putting MSE at a disadvantage.
So where does this leave users? First, you can’t rely on the headlines as they don’t provide enough details for decision-making. If you are a Windows 7/IE9/MSE/Hotmail (or Exchange with FPE/FOPE or another well protected email server) you are likely as well protected as with any of the security suites. But if you start swapping out the components for third-party components, particularly browsers, then you may have cause for concern. When paired with Chrome or Firefox a third-party security suite probably does provide better protection than Microsoft Security Essentials!
What about Windows 8? That is a more interesting story since SmartScreen-based reputation as well as the MSE-equivalent Windows Defender are built-in to the operating system itself. Unless SmartScreen is intentionally bypassed by the testing methodology I would expect Windows 8 to fare better in testing than we’ve seen with MSE. And if not, then Microsoft needs to really explain why users should feel safe despite the tests.