FAQ: About Group Tests

How do we select test topics?

Our goal is to help enterprise IT security organizations make educated decisions about purchasing and optimizing security infrastructure products and services. Our expertise in threat and vulnerability research lends itself to addressing the harder problems of information security: exploits and malware protection, and all the ways attackers can circumvent security products. All this can be confusing to even seasoned IT security pros. So we focus on testing that helps show differentiation between products, especially when it seems there may be a better mousetrap.

What do we measure?

What's important to enterprise IT security organizations: effectiveness, performance, usability, cost of ownership.

How are test methodologies devised?

NSS Labs stays abreast of current threats and current solutions from a wide range of security vendors. We regularly hold complimentary briefings with enterprise users and vendors alike. Both constituents provide valuable feedback into what is important to them: the foundation of our test methodologies. These are further vetted with our advisory group. Methodologies are available as part of the test reports.

How are vendors selected and involved? 

We select vendors and products for involvement in our testing based on market share and end-user requests. If a vendor declines, we may include them in the test anyways based on reader interest. Products for testing in this case are typically purchased, but may also be donated by interested parties. In either case, the vendor is given a chance to review and update software and settings. Vendors also frequently request to participate in testing as well, in which case we make the best call possible given our resources and priorities.

How does NSS Labs rate products and why?

Our goal is to help IT organizations make informed decisions based on empirical data. Buying criteria will vary from company to company, and there are many competing product requirements. Thus, what may be the optimal product for one company may be the 2nd or 3rd choice for another.

Tested products are rated as either Recommend, Neutral, or Caution, based strictly on their results in the test. See our Product Guidance Ratings.

What settings are used in testing?

Depending on the product group, default settings are usually the base component of the test. In product areas like network IPS, for example, where tuning of signatures is important, we invite vendors to participate in that process. Clients can obtain specific settings used in most cases by contacting our sales department.

We always give vendors the ability to review the methodology, ask questions and provide feedback. We also provide ample opportunity for them to install their products and review settings and adjust configurations as they deem necessary. Some vendors have been caught trying to game the test in order to achieve higher scores. This practice wastes our valuable time and can potentially mislead customers. Historically, we have given warnings, required settings to be corrected and allowed the vendor to proceed without incident. In the future, such activity will be disclosed publicly with no warning.

If a vendor declines to participate and configure their product, default/recommended settings will be used. In cases where it is reasonable to expect that alternate settings may produce better results, NSS Labs will endeavor to configure and test the product for optimal results.

For consumer products, default/recommended setting are used since "mom and dad" rarely have the expertise to tune an information security product.

Who pays for the testing? How do you make money?

Group tests are performed at no expense to the vendor. NSS Labs fronts the cost of the test (which is often considerable) in expectation that enterprises and government organizations will purchase the report for a reasonable fee. This allows us to test products as they need to be tested - with a gloves-off approach that best reflects the way attackers operate.

How can we review test results in detail?

Purchase the test. Given the highly valuable nature of the competitive test results, these are not available for free. Vendors may purchase the test results like any other customer. As a courtesy, and in an effort to incorporate relevant feedback, an early access copy of the group test report is made available to participating vendors-- willing and otherwise-- prior to public release. If a vendor passes up this opportunity, they can book consulting time in our lab to review specific tests and threat content used on live systems. Like any business, we do not provide this valuable QA service for free, and our standard rates apply. Please allow 60 day lead time for scheduling out-of-band projects.

Why do NSS Labs results differ so much from those of magazines and other tests?

We test harder. It's hard to test security products properly. And it gets harder the higher you raise the bar. NSS Labs specializes in threat research and expert testing of complex security products. We believe if you're not testing like the bad guys, what's the point? And our gloves-off approach is designed to inform users who need to know where the holes in their defenses are - so they can plug them.

Just about every product we test is certified by various other labs, but due to our rigorous testing standards, less than 30% of tested products are NSS Labs recommended.