if a product is good enough to sell, it’s good enough to be testED

How We Approach Testing

How do we select test topics?

We select test topics based upon enterprise demand. Our expertise in threat and vulnerability research lends itself to addressing the harder problems of information security: exploits and malware protection, and the many different ways attackers can circumvent security products — things that can be challenging for even seasoned security pros.

What do we measure?

We measure those things that are important to enterprise security organizations: effectiveness, evasions, performance, stability, usability, and cost of ownership.

How are test methodologies devised?

NSS Labs stays abreast of current threats and solutions from a wide range of security industry participants. We regularly hold complimentary briefings with enterprise users, analysts, and vendors. All constituents provide valuable feedback and insights that ultimately become the foundation of our test methodologies. These methodologies are further vetted by our advisory group. When we perform a test, we make our methodologies available to the public for free. We welcome and encourage feedback; Get in touch. Read our latest methodologies.

How does NSS rate products?

There is no perfect product; they are created by people and susceptible to the same errors that all products can suffer. Our testing routinely identifies issues in many of the leading products on the market. As such, experience has taught us that it is prudent for all products to start with a caution rating and earn their way to a rating based on their results during testing. Tested products are rated as either Recommended, Neutral, or Caution, based strictly on their results during testing. See our Product Ratings.

An NSS Unverified Statement indicates that there were products that were not tested, and we were unable to validate; therefore, NSS cautions against purchasing a product that has not been validated without a comprehensive and thorough assessment of its capabilities. If you need assistance or advice on how to do this, please reach out to one of our analysts.

Why does NSS Labs rate products?

We rate products to help readers understand their strengths and weaknesses relative to other products in that category. Our goal is to help IT organizations make informed decisions based on empirical data.  Tested products are rated as either Recommended, Neutral, or Caution, based strictly on their results in a test. See our Product Ratings.

What settings are used in testing?

Depending on the product group, default settings are often the base component of a test. In product areas such as network IPS, for example, where tuning of signatures is important, we invite vendors to participate in the process. Check our methodologies where we explain the settings used for a given product group and clients can contact an analyst for more information.

We always give vendors the ability to review the test methodology, ask questions, and provide feedback. We also provide ample opportunity for them to install their products and review settings and adjust configurations as they deem necessary. 

If a vendor declines to participate and configure their product, default/recommended settings will be used. In cases where it is reasonable to expect that alternate settings may produce better results, NSS Labs will endeavor to configure and test the product for optimal results based on documentation and experience. In cases where it is appropriate or deemed necessary, we will engage qualified consultants to configure the products.

For individual products used at home or in small and medium sized offices, default/recommended settings are used since "mom and dad" rarely have the expertise to tune an information security product.

Are NSS Labs tests “pay to play”?

No. No NSS Labs group test will ever be the result of a paid engagement. If you see it in print, you know the test or research was conducted independently. 

NSS Labs makes the upfront investment in the cost of conducting a test (which is often considerable). We are able to recover these costs in three primary ways:

1)    NSS Labs sells subscriptions to enterprise consumers and other organizations that include access to our research, our test results, and our experts.  

2)    NSS Labs sells single reports to consumers who want to see the results from a given test.  

3)    NSS Labs offers marketing rights post-publication. While we don’t know how any given vendor will fare in a test, the test reports do provide valuable differentiation for those that do well.  

This allows us to remain objective and independent (not subject to pressure from individual vendors).

What is the process for testing?

It is important to note that participation in an NSS group test is always free. The simplified process is:

Engage with our customers and relevant constituents

  • Enumerate consumer needs

  • Solicit consumer feedback on the challenges they would like assistance solving

  • Vendors provide perspective on their customers’ use of the products

  • Assess vendor strategy and capabilities

Create the methodology and request market feedback

  • Create repeatable test cases and validate their suitability and repeatability

  • Invite participation and feedback

  • Update tests and methodology as appropriate

  • Define a test schedule and execution timeline

  • Lock the methodology for this test

Note: In rare cases, events necessitate updates to the methodology during a test cycle. e.g., A new attack technique becomes known or is widely publicized.

Upon test completion, NSS Labs begins review

  • Audit findings and if necessary, develop test cases to prove or disprove findings.

  • Perform results analysis

  • Where appropriate, invite vendors to question results (dispute phase)

  • Review results and challenge findings and assumptions

  • Establish final factual review and validation

Prepare test reports for publication to the NSS Labs website

How can we review test results in detail?

Test reports are available for purchase through a subscription.  Our sales department can provide you with options that best fit your needs. Get in touch.

How are vendors selected and involved in the testing process?

NSS Labs determines inclusion of a vendor in a group test based on an analysis of the market and an understanding of the criteria important to our customers. Some of the elements considered are:

1.     Market presence

2.     Identified by industry analysts covering the specific technology area

3.     Enterprise and/or organization requests

4.     Innovative technology/solution (requires internal vetting for emerging vendors)

Vendors are invited to participate in our group tests. We find that the vast majority are proud of their products, interested in validating their strengths, interested in identifying areas of opportunity to improve their products, and eager to participate in testing. Note that while we invite vendors to participate, it is our perspective that a product good enough to sell is good enough to test and vendors do not get to choose whether or not they will be tested.

Why isn’t product XYZ in your test?

Unfortunately, some vendors are reluctant to participate in an independent test that they cannot control. We always prefer that vendors participate willingly and support the testing process, but if they decline, we may include their products anyway, particularly if they have a significant market share and/or have gained broad visibility through bold claims to the marketplace. Under this scenario, where vendors attempt to avoid testing, products are often donated by interested parties and if necessary, will be purchased for testing. When a product is not included in one of our tests, it is typically because of one of a few reasons:

  • There was insufficient interest in the product from the NSS customer community; let us know you are interested!

  • The product was not suitable for the test

  • The configuration of the product was found to be inadequate

  • It was not possible to acquire the product through ordinary means

  • Acquisition was blocked

  • The product was not activated or was deactivated during testing

How can I suggest a product for testing?

Please let us know if you think a product should be in one of our tests. You can easily provide that feedback and we will consider inclusion and invite the vendor’s participation. As a customer or potential customer of a product, you have the right to know how well that product is performing and protecting you. You should always ask your vendors where their test results are if you believe they should be in one of our tests.

When things go wrong.

Things can and do go wrong during a test. Sometimes we are unable to complete a test due to technical problems such as a license expiration or a product failure. Sometimes we are unable to complete a test because a product is disabled during testing. Sometimes we are unable to complete a test because the product or configuration was materially changed during the test and invalidates the results. When these things happen, we work to rectify the situation, but if it cannot be resolved, we will report results to the best of our ability and call this information out in the report.