Microsoft Security Essentials has recently come under some criticism for poor test results. The obvious, or maybe not so obvious, question to ask is how much of this is about the performance of Microsoft Security Essentials (MSE) and how much is it about testing methodology? Is MSE really so mediocre or are testers doing Apples vs. Oranges comparisons?
The core problem here is that many test organizations design their test regimes to test Security Suites (“demonstrate their capabilities using all components and protection layers“) and then apply those tests just to MSE rather than to the entire “Microsoft security suite” of which it is part. Why? One key reason is that Microsoft doesn’t explicitly offer a security suite, instead it spreads security capabilities across its products and components. Rather than MSE being the cornerstone of its security efforts, as an anti-malware engine is for a traditional security vendor, for Microsoft MSE is a component that fills in a missing piece in the Windows security effort.
Why is this so important a distinction? Simple, to get the full protection that a third-party suite offers you must be using the other components that Microsoft considers part of its suite. If you don’t use, or don’t test, those components then you will indeed see less protection than you could get from third-party suite. To focus on a prime example, Microsoft has focused many recent security advances on Internet Explorer. MSE does not try to duplicate those efforts, so if you use Firefox or Chrome you don’t get those benefits. Meanwhile a third-party security suite will implement similar advances in a browser neutral way or provide add-ins to bring those capabilities to all popular browsers.
The most important advance in security technology the last few years is the use of Reputation to decide if it is safe to run a program. With a reputation-based system you flip the security problem on its head, running programs you know are safe and either blocking or applying more scrutiny to programs whose safety is unknown or suspect. This helps solve the problem that malware authors can write undetectable malware faster than anti-malware signatures can be updated to detect them. That makes it particularly effective at blocking 0-day attacks, the area where MSE has been doing so poorly in testing.
The reason testing methodology is in question here is that Microsoft splits its use of Reputation over one (Windows XP), two (Windows 7), or three (Windows 8) components. MSE uses Reputation in all cases to decide if an image is safe and should be run without further evaluation or if it should be given closer scrutiny for its malware potential, but it doesn’t block execution just based on an unknown reputation. Microsoft brought reputation-based blocking into the picture with Internet Explorer 9 SmartScreen on Windows 7. With Windows 8 the picture expands even more fully with Windows 8 itself using SmartScreen reputation. Basically Microsoft assumes that you are using IE9 and MSE together if you want the full benefits of reputation-based protection. Use Firefox, Chrome, or another browser and you aren’t using Microsoft’s full security suite. What browser due testing organizations use in their tests? I don’t even think they reveal such details.
These testing methodology issues go beyond the reputation of executables. Microsoft relies on IE’s SmartScreen for URL filtering as well. Security Suite vendors offer their own browser add-ins for URL filtering, so they cover the major browsers and not just IE. And Microsoft assumes more server-based filtering of email or catching bad executables when they are transferred to disk. Security Suite vendors offer “end-point” (i.e., client) email filtering. Test methodologies appear to intentionally try to force a requirement for end-point email SPAM/Malware filtering, putting MSE at a disadvantage.
So where does this leave users? First, you can’t rely on the headlines as they don’t provide enough details for decision-making. If you are a Windows 7/IE9/MSE/Hotmail (or Exchange with FPE/FOPE or another well protected email server) you are likely as well protected as with any of the security suites. But if you start swapping out the components for third-party components, particularly browsers, then you may have cause for concern. When paired with Chrome or Firefox a third-party security suite probably does provide better protection than Microsoft Security Essentials!
What about Windows 8? That is a more interesting story since SmartScreen-based reputation as well as the MSE-equivalent Windows Defender are built-in to the operating system itself. Unless SmartScreen is intentionally bypassed by the testing methodology I would expect Windows 8 to fare better in testing than we’ve seen with MSE. And if not, then Microsoft needs to really explain why users should feel safe despite the tests.
Hal's (Im)Perfect Vision 
The problem is that neither Microsoft nor third party’s making recommendations stress that MSE is not meant to be used in isolation like the other competitor’s suites are. If they would stress that MSE is only one-third of the solution and that IE9/10 and Hotmail are also parts of their “security suite” (from the point of view of the mostly computer illiterate anyway) and that not using all three parts together means you need something other than MSE for protection then reviewers would be forced to test MSE with its two allies properly in place and not in isolation.
Even the MSE download page at http://windows.microsoft.com/en-US/windows/security-essentials-download states “Key Features Comprehensive malware protection” and has no mention at all of the two allies you state it is designed to depend upon.
I believe this is the result of ongoing tension between what is and isn’t part of Windows. And MSE has further been in a bit of a marketing no-man’s land as the security product group owned it but their Forefront marketing team did not have marketing responsibility for it. Meanwhile Windows marketing attempted to be neutral about which anti-malware software users run. So pointing out how all the pieces work together is in the domain of bloggers and not the core marketing materials.
If you look at Windows 8 marketing materials you find a clearer story. For example http://windows.microsoft.com/en-US/windows-8/windows-defender#1TC=t1 describes both Windows Defender and Windows SmartScreen as co-horts in defending against malware.
I was also going to mention that Microsoft has put a lot of effort into trying to get testers to adapt their test methodology to accommodate Microsoft’s suite definition.
The ‘reputation’ tool is a nice thought but in practise absolutely worthless. Only the companies who are willing to pay for MS to certify their products will be ‘ok’. However consumers use mostly software that comes from small vendors who are not willing to spend the time and money to get certifications. So the end result is that the consumers get accustomed to accepting a couple of extra warnings per install and nothing has changed – well except even more annoyed users as end result and a steady cash flow to MS for the companies that do deem it necessary to get rid of the warnings.
Three points:
1) You don’t need to pay Microsoft to establish reputation. Ideally you pay for a certificate to sign the application, so the app developer establishes a reputation that can be applied to future executables, but that money doesn’t go to Microsoft.
2) They’ve made it far more difficult than a few extra clicks to override the reputation check. It’s actually painful to find the override and typical consumers aren’t going to do it. Not only that, but one major point of the reputation check was to minimize the number of times per year that a consumer even gets asked. I think I’ve had an install lead to a reputation failure twice since IE9 was released. That’s a huge improvement from the pre-IE9 days where most downloads triggered a warning and users learned to just click through it.
3) I don’t agree with your point that most software comes from small vendors, or at least that most consumer software comes from them. Most consumer systems I’ve seen the last few years has little software on it that hasn’t come from Microsoft, Google, Apple, Adobe, Oracle (i.e., Java), a printer vendor, or crapware pre-installed by the PC vendor. Most “long tail” applications moved to the web a long time ago. The Windows Store is intended to change that (and by definition provides its own reputation certification), but right now I don’t believe most consumers are downloading apps written by “Joe Developer”.
1) You need to pay for the certificate – whether its MS or someone else taking the money it doesn’t really matter
2) Any application that’s not certified will ‘fail’ the check and require annoying extra clicks during the installation. Even a perfectly safe application will get lame warnings for nothing.
3) A huge amount of business software and different hobbyist software comes from small ISV:s or hobby coders who will never invest money for any certificates. If you haven’t run into any, your computer use is limited strictly to mainstream software.
Obviously you don’t know how SmartScreen works.
He isn’t far off. I’ve seen my own legit unsigned apps get blocked time and again because I don’t seen a point to paying several hundred dollars a year (or every two years) just to run my own apps without issue. It’s for that reason, that I turn off the Windows SmartScreen “feature” on every new install of Windows 8 that I do.
If you develop for yourself than that is indeed a consequence. But you are the .001%
Sorry but there are thousands and thousands of ISVs who don’t go through the certification, especially on business side. It’s an unnecessary extra cost, nothing more.
What I see on workstations: various archivers, windows commander, music players, video players other than WMP, incerasingly often there’s OOo or LibreOffice, all sorts of small games, media library management, drawing apps, educational software, alternative browsers, chat clients, and yes, LOB apps developed specifically for the niche in which the company owning the workstation acts – thing bookkeeping, registration software for medical practices. I woudn’t call these web apps, nor would I expect them to move to the web anytime soon.
I guess you need to look around more closely. Certainly, Grandpa and Grandma may not, but Grandson and Granddaughter have no qualms about loading stuff not from MS or other major players who charge – even Son and Daughter are more apt to download OSS they here is good – e.g., VLC, Firefox, etc.
But those are fairly major and very likely to have established reputations.
And it is those grandkids and their arbitrary app downloads that most need the protection that reputation provides!
The ‘reputation’ is a pain in the behind for any starting software developer who faces typical MS ‘guilty untill proven innocent’ approach to security. They’ll have a hard time distributing their apps when IE screams out danger for no reason and effectively tries to hide the options for installation.
People like halberenson seem to live in a small bubble where ‘software’ consists of a couple of major vendors products. This couldn’t be any further from the truth.
After having nothing but problems for 20 years, first with Norton, then ca Security Suite, AVG, and finally Kaspersky, I tried using MSSE on my private notebook. After a year without infection and everything working behind the surface without my caring for anything, I decided to use MSSE on my 3 business computers. That was 3 years ago, and I’m still the happiest guy on earth for security.
I’m in the same boat. As a former colleague who has held senior positions at a few of the big name security companies said to me, “unless someone goes out an intentionally tries to infect their machine with malware they are safe” with MSE and the rest of MS’ protections.
This is the most misinformed phrase I’ve ever heard! 😀
MSSE is one of the worst products you can use due to its bad detection rates and its popularity (think Norton which actually became a channel for attacks when hackers exploited it to gain control of the host machines).
If someone is browsing the net using IE and having javascript, activex etc. enabled they’re doomed to get caught by a drive-by attack. The only safe way to use the net is not to use windows, second safest is using FF/Chrome + noscript + adblock and a good AV. Just a few months ago my sons computer got infected by a nasty malware despite him having MSSE installed. Actually most computers I’m called to clean are running major AVs at the time of infection (AVG, Avast, F-secure, Norton, MSSE, Sophos etc). Most AVs won’t even try to detect malware and malware scanners do a pretty bad job at it too. Not to mention that running AV and malware scanners will destroy the computers performance much the same way a virus infection would so you lose in any case. My windows installations lack any sort of ‘protection’ but they keep virus/malware free because I don’t do anything risky with them.
My approach was different, but also driven by different reasons. The days when you could get infected by simply browsing the web in a safe and informed way are already history, and anyway, even if I used Windows since 3.0, I never had one infection on my personal systems. But Windows in itself became increasingly inconvenient for me. As a developer and power user, I want to on one hand be able to extend and customize my system the way I see fit, and on the other hand I simply don’t have the money (like several thousand dollars a year) to buy all software that I’d like to use to achieve what I need. So I switched to Linux, some three years ago. At that time, I found that even on my brand-new, custom-built, powerhouse desktop and my similarly new, shiny and glittery laptop there were simply no issues with drivers, installation or app availability. And of course, no security issues. I never looked back. Then again, a regular Windows user might not understand why he needs to enter his password whenever he wants to install new software or modify system configuration …
Why I’m mentioning this here: I think that both the time and desktop Linux are ripe for a more radical approach to the security issues with Windows. Unless you’re bound to legacy LOB apps (i.e. non-web apps), or you also use your machine intensively for gaming, I think it may be worth (like in several hundred dollars worth, depending on the apps you use, and whether you do license them or pirate them) to try out Linux. For me, this was a moral issue too: even if MS or Adobe never came after me for using unlicensed apps on my desktop at home, I started feeling increasingly uneasy about not licensing all the apps I was using. Things have changed since then, and there are several free apps available for Windows which I would’ve had to pay for three years back, but the OSS ecosystem is IMO richer and nicer to use, if you need it.
For me, it comes down to “does the product conflict with other AV or not”?
If not, then it gets judged on the same basis as full av suites, since it prohibits other AV from running. Anything that recommends disabling other AV better cover your but when you use Firefox or Chrome, since by its nature it is sidelining other protection you would have had.
If MSE does not claim to be your system’s AV and other AV has no problem running alongside, then the huge gaps of coverage when using other Windiws programs is fine. It’s just another layer of protection.
If it does conflict, and doesn’t cover other browsers or third party programs well, then it’s a poor AV substitute.
I would never buy NOD32 under the assumption it only covers me if I use a certain browser or mail suite.
How do you define conflict? I can install and use several products side-by-side with MSE that do provide additional protection for Firefox or Chrome, just not all products. Indeed I have WOT installed on my systems, use a URL-filtering DNS service for my home network, and install parallel security products from time to time. A number of smaller players, like Immunet and Spybot S&D, specifically target working concurrently with the major anti-malware products. And of course Firefox and Chrome use Google’s Safe Browsing service for their own URL filtering. Plus MSE does protect Firefox and Chrome users, just not at the browser level.
And this isn’t unique to MSE. Norton AV doesn’t provide URL filtering nor email filtering for Opera, for example.
I mean products claiming to be anti-virus tend to conflict, lock up your computer, or report errors when you try to instal them concurrently. Almost every antivirus, or product claiming to be an actual realtime antivirus will tell you the first thing when you install it that you should only have ONE actual AV running. Some will go so far as to refuse to install unless you uninstall the other AV. One example, Nod32 and AVG. I’m sure you get the same thing trying to install McAfee AV and Nod32 to run concurrently, and so on. Of course you can add a router and other browser plugins or other security components, or run a malware scan now and then, but we’re comparing security suites that include an AntiVirus component, I think. So if MSE has a component that counts as an “antivirus” that conflicts with other antivirus products that cover other browsers, that is what I mean. If MSE can exist just fine alongside other products sold as “AntiVirus” software that protect your other browsers, without locking you up or causing other issues you get with multiple AV products running concurrently, then it is fine to evaluate it as a supplemental product but not a replacement. If it interferes with having actual antivirus that protects your other browsers, then it moves to a negative for me. Sorry for rambling trying to explain what I mean by conflict. I don’t mean you can’t do multiple things for different browsers. What I mean is pretty clear when you try to install two security suites that include realtime “AV” concurrently.
You have a factual point, but I find it irrelevant. If a piece of security software does not operate fully in a common real-world scenario (for non-IE users), then it is perfectly acceptable to report its reduced effectiveness as compared to all-in-one competitor suites without restricting the comparison to a subset of users using a browser favorable to MSE’s results.
I use MSE because it has a much smaller footprint on my CPU’s time, and because I manually seek out reputation info before downloading, negating much of the need for SmartScreen… But the average user does not browse with security in mind, and expect software like MSE to protect them from things they don’t know about.
Exactly why I pointed out that non-IE users might want to use something other than MSE.
I probably would use one browser if I could. I still come across sites where they layout or scripting doesn’t seem to work right in IE or Firefox, but does in the other. When I think security suite or AV I think of something that covers both browsers fine. I don’t use Norton, but I’ve used McAfee and Eset and they seem to cover both browsers fine. I don’t need to be protected from going to “bad sites”, I just need my AV to block any viruses and let me know if one attacks, regardless of what browser I’m using.
Good post.
Great!due to the fact the virus creators shift too speedily. That may be prompting start-ups and other businesses to acquire artistic about new approaches to laptop stability.