The security analysts at NSS Labs tested 13 different network IPS products, including stand-alone IPS and multi-function gateways, and one unified threat management product. If your organization is evaluating IPS solutions, or is looking to benchmark your current vendor, then this is the definitive report to read. Data and analysis are based on multiple man-years of complex, real-world testing that mimics how cyber-criminals are working to penetrate corporate defenses (see methodology). No surveys, interviews or soft trends. This is the hard test data upon which organizations base critical, big-dollar decisions.

The report includes valuable information not available anywhere else:

• Total cost of ownership analysis: are you getting the most security for your budget?
• Security effectiveness: how much effort is required to protect all your assets?
• Real-world performance benchmarks: can the device handle your traffic?
• Management and usability insights: how much time is really required to achieve results?

While the full breadth and depth of the research is available only to our subscribers, we are making a summary available to non-clients. Next week/year I will blog more about the key findings and what they mean for IT buyers in 2011.

Tested Products include (alphabetically):
1. CHECKPOINT POWER-1 11065
2. CISCO IPS 4260
3. ENDACE CORE-100 (IDS)
4. FORTINET FORTIGATE 3810A
5. IBM PROVENTIA NETWORK IPS GX6116
6. JUNIPER IDP-8200
7. JUNIPER SRX 3600
8. MCAFEE M-8000
9. NSFOCUS NIPS-1200
10. PALO ALTO NETWORKS PA-4020
11. SOURCEFIRE 3D 4500
12. STONESOFT IPS-1205
13. STONESOFT IPS 3205

This week we released another report on socially-engineered malware protections delivered by browsers. While most articles and blogs seemed to interpret the results properly, there were some inaccuracies that we wish to address. Security is a complex field and the terminology can sometimes be misinterpreted. This can be compounded when vendors who did poorly add their spin, or when the data challenges one's own beliefs.

Stopping Malware vs. Security:
Some of the key security terms are clarified in a previous post, especially relating to "most secure" browser. Some articles incorrectly stated that we found IE to be the "most secure" browser. What we tested was browser effectiveness at stopping malware from reaching a user or their PCs, not the security of the browser itself or its plug-ins. Modern browsers are a wonderful, free additional layer of protection. They work well with your favorite antivirus software. Browsers however will not stop malware coming via email or USB drives.

The Malware threat:
Malware is arguably the largest security threat facing users today  with more than 60,000 unique, new samples entering circulation each day (source: McAfee). There's a $14B industry addressing the problem of malware. These test results challenge the comfortable status quo of many of the vendors. The notion that a free product adds so much protection can easily upset the industry apple cart. The assertion that the focus of the test was narrow (made by Google and some others) flies in the face of all generally accepted data. To say one's browser was "built with security in mind" is nice, but marketing speak. What we've offered is hard data about malware protection. The exploitability of the browser is also a very important topic. But, even in this case, data doesn't support Chrome being "more secure" (i.e. less vulnerable) than other browsers (see CVEs, Secunia disclosures, etc.). We do sincerely applaud the innovations and bug bounties though, and encourage all vendors to build more security in.

Versions tested:
The claim that we tested an "old" version of Chrome is patently false. As stated in the report, the test was run in late September, when version 6 was the current browser. Since then Google has released two other so-called "major" releases, none of which have claimed improvements to the tested SafeBrowsing functionality. Here is the Timeline for Chrome Version Release:
Stable Version Release Date
5.0.375      2010-05-25
6.0.472      2010-09-02
7.0.517      2010-10-21
8.0.552      2010-12-02
Furthermore, Chrome's sandboxing is designed for exploits protection; it does not protect against socially-engineered malware. You click it, it runs.

At the time of the test, Opera's website marketed protection from malware as a feature, yet our results showed no protection was available. AVG officials have separately acknowledged that the integration of its technology was not yet complete, confirming our results. Features should only be marketed after they are actually in the product.

Application Reputation:
In a world of dizzying information, there's much rush to a sound bite of who "won". The most significant technological message of this test may have been overlooked. This test benchmarked the world's first implementation of an application reputation system within free web browsers that goes beyond simple black lists. Nascent stand-alone security products such as SolidCore (acquired by McAfee), Bit9 and CoreTrace utilize what is commonly referred to as white listing, and commercial endpoint security products are starting to include some form of this as well. Apple uses a "walled garden" approach to limit exposure to malware on its tightly controlled platforms by pre-approving apps.

In web browsers, so far we've seen just black listing. A URL or application is either known to be bad, or unknown. What's unique about Microsoft's approach in the IE9 browser is that applications have 3 states: known good (white), known bad (black) and unknown (grey). The combination of good and bad indicators is clearly powerful, stopping 99% of malware via the web download vector. The use of application reputation to identify good applications and bad ones is unique to IE, for now. Will other vendors follow Microsoft's lead?

Methodology and Open Invitations:
No vendor has influence over what/how we test or where we get malware from. We constantly run our Live Test network with a variety of security products - antivirus, browsers, and other network devices.
Over the past 2 years we've been running these tests, NSS has discussed results and methodology (which is included in the report) with all of the browser vendors; even providing sample URLs for validation to them in past tests. Some had even privately acknowledged issues.

Our past invitations to become more involved in the ongoing testing and review of results still stands.

Terminology used to describe attacks is often misunderstood by the broader public. Thus, we are providing this brief explanation of threat types and the terms we use in our reports.

End users and their computers face a number of different attack types. At a high level there are two: 1) Socially-engineered attacks target the user, and work only when the user is tricked into performing an action; running a malicious file or giving up personal data to a fraudulent site. 2) Other attacks target vulnerabilities in systems and applications. The following chart gives a rough breakdown of common threats against end user systems.


Layers of Security
These types of security threats can be mitigated by a range of security products; including IPS, UTM, SWG appliances, and on the endpoint: Internet security suites, most anti-malware products, and even web browsers. Modern browsers have implemented an additional layer of security to help users differentiate between good and bad web sites and downloads.

When selecting security products, either for home or business environments, it's often hard to tell from the marketing literature which products actually stop threats. And protection levels offered by products in the different categories can vary greatly. The above taxonomy should help you ask more specific questions of vendors. It also acts as a guide to terminology used in NSS Labs test reports.

Security products protecting users and their computers
When someone says "Product X stops more malware, exploits etc." or "Product X offers better malware or exploit protection", what they mean is that Product X inspects traffic passing through it and stops these attacks from reaching and/or affecting the end user or the operating system.

Security products themselves susceptible to threats
In addition, security suites and browsers (and their plug-ins) can be susceptible to exploits if the software has vulnerabilities in them. When someone says "browser X is more secure" what they are trying to say is that browser X has fewer vulnerabilities. Unfortunately, most software, and all browsers have vulnerabilities. For example in the first 9 months of 2010, Microsoft Internet Explorer had 43 new published vulnerabilities, while Google Chrome had 106, according to Secunia research.

For more exhaustive treatments on threat types including product test results, consult our research services at nsslabs.com.

An update on our current 2010 IPS group test.

Those of you following network IPS know that the last NSS Labs IPS group test in Q4 2009 made quite an impact in the marketplace. The testing soundly destroyed the notion that an organization could buy one of the "leading products" and rest easy
- Vendors with leading market share and analyst accolades were shown to have mixed to poor results on our robust exploit testing
- Over half the vendors lacking coverage for well-known evasion techniques
- Many put performance ahead of security effectiveness

We were pleased to work collaboratively with many of the vendors in ensuing months to further identify and rectify issues in advance of our next group test. In the process, several vendors released new hardware and software. Excited as we were to test drive the new improved models, we were disappointed in more than a few situations to discover the latest and greatest versions had show-stopper level flaws that could cost their customers a great deal of money and time. A number of vendors requested additional time to remediate flaws, and it was clear that much would change very quickly.

Meanwhile, the size of the NSS Labs 2010 IPS group test has also grown in number of participants and complexity, making this by far the largest, most in-depth group test of its kind.
- We've added several new vendor products;
- We've refreshed the exploits used, changing and upgrading a third of the content;
- Added additional HTML evasions;

Determined to not let further delays keep our much awaited group test off the streets any longer, we decided to take a phased approach, releasing the test in two parts; Part I will contain five vendors, and Part II will contain the remaining vendors. While this phased release provides enterprises with much needed information on some of the top vendors, it arguably leaves them waiting to see the rest. This may be more or less relevant depending on one's situation, and which products are under consideration. Certainly some vendor sales teams maybe playing catch up. In the meantime, NSS clients can utilize analyst advisory sessions to receive additional guidance and help fill in the gaps.

So, look for part one of the NSS Labs IPS group test to arrive middle of next week here, and part two in the middle of December. Clients will automatically be notified when it is posted via email alert. If you're not a client, register for free here, or contact our advisory services group to learn more.

We have secretary's day, presidents day, mothers day, fathers day, etc. But I never suspected there would be a 'software developers day' or other professional recognition day in our field. Twitter educated me otherwise, so I'm going with it. Our profession could certainly use a little recognition and attention. The auspicious occasion as this blog explains is the discovery of the first (literal) bug in a computer system on Sept 9, 1947.
Since then, we've had a meager few decades to hone our software development skills, and our bug-finding skills are not far behind. To that end, a short homage to the hard-working testers and QA professionals who often hear these responses from developers.

24. "It works fine on MY computer"
23. "Who did you login as ?"
22. "It's a feature"
21. "It's WAD (Working As Designed)"
20. "That's weird..."
19. "It's never done that before."
18. "It worked yesterday."
17. "How is that possible?"
16. "It must be a hardware problem."
15. "What did you type in wrong to get it to crash?"
14. "There is something funky in your data."
13. "I haven't touched that module in weeks!"
12. "You must have the wrong version."
11. "It's just some unlucky coincidence."
10. "I can't test everything!"

For the top 9, see this site for a familiar chuckle...

NSS has been testing exploits and exploit-detection/protection products for the last 10 years or so, dating back to the early days of IDS and IPS . It's arguably one of our specialties. And we have written before about the differences between malware, exploits and vulnerabilities. Yet, as these pernicious threats are elevated into mainstream consumer and enterprise awareness, we are seeing quite a bit of terminology confusion. So, for our latest Host Intrusion Prevention System (HIPS) test of enterprise anti-malware products, we created some demonstration videos of client-side exploits used in the test.

Client-side exploits are aggressive weapons used by cybercriminals that allow them to silently take control of computers that visit a web site. The infected widgets on nearly 5 million Network Solutions sites are a prime example (see Krebs' report on Armorize discovery). And NSS Labs' Q2 2010 test of 10 endpoint protection products for HIPS evaluates protection against client-side exploits.

In recent discussions with clients and journalists we've found the need to clarify some definitions of how these attacks work, and how they're different from typical socially-engineered malware campaigns. So, the following definitions and analogies are provided in an effort to provide clarification, as well as to bridge an ongoing communication gap between security vendors and their customers.

Vulnerability

Like a locked door that can be opened with the right key or combination, a vulnerability is a bug in software code that allows a product to be exploited. An example of a software vulnerability is improperly-defined memory usage within a function that enables content sent to a specific memory location to be run with privileged rights.

Exploit

An exploit is a specially crafted code sequence which can 'trigger' or 'unlock' a vulnerability within an application, such as a heap spray, buffer overflow attack, etc. An exploit can be hiding in an infected website (client-side attack) where it ambushes visiting computers or be launched from an another computer (remote attack).

Payload

The payload is the content that gets delivered once the vulnerable application has been exploited. Payloads are the actions that are performed on the compromised target computer, such as command execution, writing a file to disk, returning a reverse shell, etc. This may be malware, but does not have to be.

The Test

The test utilized 123 common and public vulnerabilities dating from 2006 to 2010. These vulnerabilities were exploited when a user visited an infected web page hosting the attack code. The attacks occurred in two stages:
1. The attacker caused a specially-crafted stream of data and code to be delivered to a precise location. This exploited the victim's computer, gaining the attacker the ability to perform arbitrary code execution.
2. Malicious code was silently executed on the victim's computer.

If the attack can be thwarted in Stage One (successful exploit), then it cannot progress to Stage Two, where a malicious payload can be delivered. As long as the exploit is not defeated, then installing malware is just one of many possible actions the attacker can take. And the choice of malicious code is nearly infinite. Since there are far fewer exploits than malware, it is imperative that attacks be defeated in the earliest possible stage. In other words, it is advantageous for AV suites to detect the exploit vs. chasing malware samples.

Results and Next Steps

Unfortunately, 75% of corporate users are under-protected, based on vendor market share and their respective scores, which ranged dramatically from 29% to 100%; and even worse for variants. Depending which product you have, you may have significant cause for concern. So, what's next?

For starters, if you're an organization with critical data behind any of these products, I suggest you buy and read the full report here. There has been a lot of press coverage of the report highlights, but not the details, generally reporting failure of AV suites in this area. While you may not be ripping out thousands of endpoint deployments, you may be asking your vendor some tough questions and setting expectations for improvement. Exploit detection & blocking is clearly an area that the security industry needs to focus on going forward. Got questions about other threat mitigation options, consult our other reports, or contact us.

Michelle is a passionate infosec pro and assessor. She gets some kudos today for expressing on a personal level the frustrations of many infosec practitioners whose job it is to audit, assess and help improve their clients' defenses. PCI DSS forces those who would do little or nothing for security to do something more. It also encourages those who would do more to do less because it is just enough to deal with a clear and present threat: the audit.

As Josh Corman at the 451 Group likes to say: "Why focus on compliance instead of security? I might be hacked, but I will be fined." (if you handle cardholder data). Given the amount of client-side attacks and botnet infection data we see, the case could be made otherwise. Corporations are getting attacked daily. They might not be aware of it though, due to the holes in their security defenses, logs, and even alerting practices.

After all, security products can only alert and report on what they have detections for. Based on our testing, that leaves a significant gap with every vendor, between 12 and 83%. Do you know which holes matter on your network and where they are? Want to hear ideas on how to improve and not just pass?

I'm happy to echo Michelle's call for more heart and less check box.

Rick