How We Approach Testing

 

How do we select test topics?

We select test topics based upon enterprise demand. Our expertise in threat and vulnerability research lends itself to addressing the harder problems of information security: exploits and malware protection, and all the ways attackers can circumvent security products. Things that can be challenging for even seasoned security pros. 

What do we measure?

We measure those things that we are important to enterprise security organizations: effectiveness, evasions, performance, stability, usability, and cost of ownership.

How are test methodologies devised?

NSS Labs stays abreast of current threats and current solutions from a wide range of security vendors. We regularly hold complimentary briefings with enterprise users and vendors alike. Both constituents provide valuable feedback into what is important to them: the foundation of our test methodologies. These are further vetted with our advisory group.  We make our methodologies available to the public for free and encourage feedback.

Why does NSS Labs rate products?

We rate products to help readers understand the relative strengths and weaknesses of a product. Our goal is to help IT organizations make informed decisions based on empirical data.  Tested products are rated as either Recommended, Neutral, or Caution, based strictly on their results in the test. See our Product Ratings.

What settings are used in testing?

Depending on the product group, default settings are usually the base component of the test. In product areas like network IPS, for example, where tuning of signatures is important, we invite vendors to participate in that process. Clients can obtain specific settings used in most cases by contacting our testing department.

We always give vendors the ability to review the methodology, ask questions and provide feedback. We also provide ample opportunity for them to install their products and review settings and adjust configurations as they deem necessary. 

If a vendor declines to participate and configure their product, default/recommended settings will be used. In cases where it is reasonable to expect that alternate settings may produce better results, NSS Labs will endeavor to configure and test the product for optimal results.

For consumer products, default/recommended setting are used since "mom and dad" rarely have the expertise to tune an information security product.

Are NSS Labs tests “pay to play”?

No.  No NSS Labs publication will ever be the result of a paid engagement, so if you see it in print, the test or research conducted was not paid for by a vendor. 

NSS Labs makes the upfront investment in the cost of conducting a test (which is often considerable). We are able to recover these costs in three primary ways:

1)    NSS Labs sells subscriptions to enterprises and other organizations that include access to our research, our test results, and our experts.  

2)    NSS Labs sells single reports to individuals who want to see the results from a given test.  

3)    NSS Labs offers marketing rights post-publication. While we don’t know how any given vendor will fare in a test, the tests reports do provide valuable differentiation for those that do well.  

This allows us to remain objective and independent (not subject to pressure from individual vendors).

How can we review test results in detail?

Test reports are available for purchase through a subscription.  Our sales department can provide you with options that best fit your needs.  

How are vendors selected and involved in the testing process?

For products that align to the enterprise use cases for a given methodology, NSS Labs determines inclusion of the vendor in a group test based on the following criteria:

    1. Market presence

    2. Enterprise and/or organization requests

    3. Innovative technology/solution (requires internal vetting for emerging vendors)

Vendors are notified upon consideration and upon formal selection. It is important to note that vendors do not get to choose whether or not they will be tested. Vendors frequently request to participate in testing, and we accommodate where possible given our resources and priorities.  

Unfortunately, some vendors are reluctant to participate in an independent test that they do not control. We prefer that selected vendors come willingly and support the testing process, but if they decline, we may include their devices anyway, particularly if they have a significant market share and/or have gained broad visibility through bold claims to the marketplace. Under this scenario, products for testing are typically purchased, but may also be donated by interested parties. 

 

If a product is good enough to sell to the public, it is good enough to be tested.