Answers to Questions from the AEP 3.0 Webinar

Which products offered on-premises management vs. cloud-delivered management?

Most of the tested products offer a cloud-delivered management console. However, many vendors provide both options. We allowed vendors to decide which console would be tested, since both options met our minimum product requirements. For the ESET, Fortinet, Kaspersky Lab, G Data, Symantec, and Trend Micro products, on-premises management consoles were evaluated in the AEP v3.0 Group Test.

What is the difference between AEP and Endpoint Detection and Response (EDR) products in today’s market?

NSS completed its first group test of EDR products in the fall of 2018; four products were tested. We wrapped our third group test of AEP products this past spring. Currently, we distinguish between these two product categories in two primary ways: 

1.     Blocking vs. Detection Capabilities – AEP products first block/prevent and then are expected to provide enough threat detail to aid remediation or incident response. In the EDR v1.0 Group Test, products were not configured for blocking (even though some have this capability) since few EDR products can provide the same level of blocking/prevention as an AEP product.

2.     Threat Visibility Capabilities – The NSS Labs AEP Test Methodology v3.0 defines threat visibility expectations for an AEP product: “After a product has detected or blocked malicious or anomalous behavior, it should support follow-on investigation, incident response, and/or remediation efforts. The product should convey detailed and timely threat event metadata to its central management system. Metadata should be paired with the basic threat event data to support the detection or block conviction. This also enables responders to act more quickly or establish incident severity more readily. 
Decisions on which attributes to assess will depend on the threat types or on the techniques used but may include information such as source and destination IPs, file hashes, file system paths, registry information, shell commands, command-and-control or outbound connections, and convicted blacklist IP matches. The product must also be able to correctly separate irrelevant data and system noise from a threat’s activities.”

AEP products typically provide the threat visibility described in the NSS AEP Test Methodology v3.0 while EDR products typically provide both this visibility and systems visibility. Most EDR products also attempt to group similar threat activity across an enterprise into aggregated attack timelines. This helps simplify enterprise incident response efforts because analysts can use this aggregated view to dig into comprehensive threat details and associated system characteristics. Some EDR products will log as many events as they can on an endpoint so they have as much information as possible to understand breach activity.

How do I choose between AEP & EDR products?

Endpoint security products have multiple use cases, and we see organizations that require both AEP and EDR. While some organizations find value in deploying a dedicated endpoint visibility tool (i.e., an EDR product), for others, it is just another agent to manage and another console to review. Most of our clients have indicated they prefer an AEP product with strong prevention capabilities and enough visibility to be useful. 

The line between AEP products, traditional endpoint security products, and EDR products is blurred; most enterprise endpoint products have some visibility or advanced detection capabilities that allow them to be categorized as AEP products. We expect this trend to continue with AEP products gaining more EDR functionality over time, gathering broader system and telemetry data, performing deeper analysis, and even expanding into system management. Note that although the markets are consolidating, we do not see a single product that meets all enterprise requirements for both categories.

Did NSS Labs test application compatibility?

The AEP v3.0 test did not specifically include tests for application compatibility; however, our test engineers had extensive hands-on experience with the products during testing. Our test environment used many standard office productivity applications. Additionally, during testing, we execute as many false positive test cases as we do malicious samples, which helps to identify any application compatibility issues. 

During the AEP v3.0 test, we did not observe compatibility issues. If incompatibilities are ever found during testing, they will be described in the relevant test reports.

Were vendors permitted to configure their products prior to AEP v3.0 testing?

We invited vendor teams to help properly install products in our test environment with the appropriate recommended enterprise configurations. We then performed assessments of these deployments by running representative test cases for the threat categories described in the test methodology. This allowed us to eliminate false positives and ensure our automation and other test processes functioned properly with the security products installed. 

Once a test begins, vendors are not permitted to configure, tune, upgrade, or update their products in response to what they may observe during the test. Minor updates such as signatures are allowed, as are transparent cloud-side updates. In addition, it is a requirement that tested products must be generally available for purchase and deployment at the time of publication. While NSS always advises against submitting products that have not reached a GA status for public testing, in very rare cases, a product may be accepted that is not generally available at the commencement of testing. In all cases, products must be available for purchase by publication. 

Which vendors/products were tested in AEP v3.0?

Did the AEP v3.0 test include an anti-phishing category?

This test did not include anti-phishing. Additional information about the test categories can be found in the AEP Test Methodology v3.0.

Our test methodologies are constantly evolving; new test cases are added (and old test cases may be removed) as we interpret the enterprise needs. While anti-phishing testing did not make the cut for v4.0 of the AEP Test Methodology, it could be integrated into a future test. Extensive anti-phishing testing was performed in our 2018 Web Browser Security Group Test.

How are the AEP v3.0 test threats sourced to ensure they are as “life-like” as possible?

At the heart of many of our tests is a patented technology called BaitNET™. This cloud-based, distributed environment is used to identify and harvest malware from the wild to ensure testing of prevalent, real-world threats. In addition to BaitNET, we develop custom threats to cover use cases that could not be obtained from the wild and to mimic threats that emulate certain adversary behavior and techniques. This also allows us to test less prevalent and/or unknown malware and ensures that security products are not only tested against well-known and understood threats.

Where is the AEP v3.0 Security Value Map (SVM)™?

Download a free copy of the 2019 NSS Labs AEP v3.0 SVM

Why are some vendors/products not depicted in the AEP v3.0 SVM?

During the webinar, many questions were asked about vendor placement in the Security Value Map; for example, “Where is vendor X in the NSS Labs SVM?” A vendor’s product may not be depicted in the SVM for a variety of reasons, such as:

1)     The product does not meet all of the minimum product requirements/inclusion criteria outlined in the relevant test methodology. NSS may make this determination during a test or after a test has been executed, since a comprehensive evaluation can reveal additional detail that may not be evident in vendor marketing materials.

2)     Our testing focuses on informing the market about the capabilities of products in consistent ways. NSS considers a product for inclusion based on a number of factors that can include: consumer interest, NSS inquiry requests, product alignment with an NSS product definition, and product market share.

3)     A vendor that meets test inclusion criteria may actively resist being tested by blocking purchase, deactivating products, or restricting license activation. 

4)     A product’s test results placed it in the Caution quadrant of an SVM. In this case, a vendor’s product may be represented anonymously on the SVM. NSS recently adopted a policy that focuses more on positive capabilities demonstrated in independent testing and does not dis-incentivize participation for products that have opportunities to improve. Test reports for all tested products are available to our Research Library subscribers through our research portal. All of the vendors that participated in this test have demonstrated a commitment to providing the best possible protections to consumers and should be commended for their commitment and transparency. 

NSS cautions against the deployment of any product where we are unable to perform a comprehensive evaluation of its effectiveness and suitability. Products that meet inclusion criteria but are not validated through testing may be depicted in a callout box on the SVM.

How is total cost of ownership (TCO) calculated?

Total cost of ownership can be a complex question involving many data points. Subscribers can learn more on the NSS approach to calculating the TCO of AEP products by reading the NSS Labs AEP v3.0 Comparative Report – TCO.

Enterprises that do not deploy an AEP product, or that deploy an AEP product with low protection, will incur less security savings in the event of a breach since additional operational overhead will be required to remediate the effects of a compromised system and protect the business. Furthermore, security products that do not provide visibility and forensic detail will realize less security savings because compromises will have to be investigated manually, which would increase operational cost. Therefore, for the purposes of this TCO model, management (of security products) and investigation (of breaches) is conservatively estimated as requiring two full-time employees at $100,000 per year (Product Operational Cost). 

The NSS TCO model calculates the projected cost of operating a business without deploying an endpoint security product as well as the potential value of deploying an endpoint security product. This potential value is based on product cost, Product Operational Cost, and the Overall Capability score of a product. 

Since each AEP product has a unique set of capabilities, the NSS TCO model assigns a value that normalizes the operational overhead, potential consequences, and associated costs a security breach can have on an organization. This allows decision makers to better analyze risk and make calculated choices that are ideal for their organizations.

Why do the TCO figures for several products appear to be approximately half of what was depicted in the AEP 2.0 SVM?

A couple of factors contributed to the difference in TCO:

  • Based upon feedback from our customers, we increased the number of agents modeled from 500 in AEP v2.0 to 2,500 in AEP v3.0. 

  • Threat visibility improved across several products, which reduced operational costs.

It is also important to consider that a 1 – 2% difference in block rate can impact a product’s TCO per Protected Agent since this influences the operational costs depicted in our model. For example, if a 2% lower block rate is combined with the absence of a detection or threat detail, there would be significant change in operational cost for organizations using the product over time. 

How can my company get more detail about a specific vendor’s security performance or total cost of ownership?

NSS’ published reports are available to subscribers to our Research Library. Briefings and inquiries with NSS analysts are also available to our customers. Finally, vendors may acquire licensed reprint rights that would allow them to distribute copies of their test reports.

Can you elaborate on the Microsoft Advanced Threat Protection (ATP) callout box on the NSS Labs SVM?

A callout box in the lower left of the 2019 NSS Labs AEP SVM states “NSS Labs was unable to test the Microsoft Advanced Threat Protection (ATP) product because it does not have a centralized management console that integrates Microsoft Windows 7 endpoints.” 

More specifically, endpoint clients must forward all pertinent security event information to a central management console or non-local log aggregator. As such, the product did not meet the minimum product requirements outlined in the AEP v3.0 Test Methodology. NSS will continue to monitor development of the Microsoft ATP product and plans to include it in a group test once we determine it meets the minimum product requirements/inclusion criteria outlined in the test methodology.

How should I choose between products?

The endpoint security market is fiercely competitive; there are a host of vendors, and the features and capabilities of these products are constantly evolving. 

NSS clients considering AEP products for proofs of concept (PoCs) often look at security efficacy first, but there are many reasons to deploy endpoint security. Security effectiveness and threat visibility are top considerations as well as areas of differentiation. Other components that our clients discuss include management UI, central management server form factor (hardware appliance on-premises or cloud-delivered), a need for lower resource utilization, threat resilience, low satisfaction with vendor support, and interoperability.

Any one of these—or perhaps all of them—may be important to an enterprise depending on its needs. The point is, there is no single product that can be considered “the best” in the market. The advice we give to our clients focuses on which products are the best fit for their specific organizations.