Breach Detection Systems (BDS)


Why BDS is Important

Breach Detection Systems (BDS) utilize both static and dynamic analysis techniques to detect advanced malware, zero-day attacks, and targeted attacks that have bypassed network security controls.

Through constant analysis of suspicious code and identification of communication with malicious hosts, BDS are capable of providing enhanced detection of threats. Such threats range from commodity malware to targeted attacks from state-sponsored threat actors that could bypass traditional network defenses such as next generation firewalls (NGFWs) and next generation intrusion prevention systems (NGIPS).

Threat actors are demonstrating the capability to bypass protection offered by conventional endpoint and perimeter security solutions. Enterprises must evolve their network defenses to incorporate a different kind of protection using advanced techniques, such as the BDS.

According to an NSS Labs research survey, BDS are most commonly deployed by the banking industry (49%) and the durables industry (manufacturing) (41%). Survey participants cited false positives and a lack of corporate image support as challenges experienced with various BDS products.

Sandbox False Positives - Cyber Security Tests

NSS research study reports the following are the top challenges experienced by the enterprises using BDS technology:

  • Nearly all alerts are false positives
  • Sandbox failed to detonate known malware
  • No support for corporate machine images

Detection products have more flexible deployment options than do blocking products, and thus they are capable of "seeing" more than blocking products. There is still concern over the issue of false positives, whether we are talking about detection products or blocking products. Detection products still need to produce minimal false positives for detection products still need to be minimal for enterprises to be operationally efficient.

The failure of BDS products to detect known malware that can exploit existing vulnerabilities is a growing concern, which is why organizations continue to rely on complementary technologies such as NGFW, NGIPS, and endpoint security.

What We Tested & Why

Time to Detect - Cyber Security Tests

The time it takes for an organization to become aware of a breach is popularly known as dwell time (time to detect). The longer this period is, the more damaging it's likely to be. For a BDS product, rapid detection and analysis of both successful and attempted breaches is critical in halting the damage caused by potential malware infections or breaches. Learn which BDS products took the least time to detect both known and unknown malware.

Drive-by Downloads (Web-based Exploits) - Cyber Security Tests

Drive-by downloads happen when users visit websites or click on malicious links. There has been a rise in exploit packs that attackers leverage to execute drive-by download attacks. These packs exploiting vulnerabilities are ready to use, which makes it relatively easy for attackers to carry out the attacks. It is important to know which products are capable of detecting and reporting on successful attacks in a timely manner-earlier detection is better. NSS Labs' 24/7 exploit testing is unique in the world, and our test results provide metrics regarding time to detect on initial compromise and on callback to the command and control server.

Social Engineered Malware (SEM) - Cyber Security Tests

Today's malware is delivered over multiple vectors including social exploits, web-based HTTP traffic, for example, malware delivered over HTTP (e.g., executables, docs and scripts) and non-HTTP traffic such as email, a cloaked executable (.jpeg, .exe, .zip), FTP, or an infected USB device. Find out which BDS products successfully detected SEM attacks.

Blended Exploits - Cyber Security Tests

Organizations are highly susceptible to blended exploits, which are typically delivered via commonly used applications, such as Microsoft Word or Excel. Blended exploits are commonly used in phishing attacks where users are tricked into clicking on malicious links. Learn which of the products are capable of keeping up with attacks using blended exploits.

Offline Infections - Cyber Security Tests

In today's world, remote work is becoming more and more popular. Remote users can become infected while outside the protection of corporate network security. Once infected devices are reattached to corporate networks, infection can move laterally within these networks. If your organization has remote employees, you need to know which BDS products will fully protect you against offline infections.

Evasions - Cyber Security Tests

Attackers use evasions to bypass security controls. A single evasion can grant an attacker access to your network. Evasion types impacting BDS products include binary obfuscation, packers, compressors, HTML obfuscation, HTTP evasions, attacks on nonstandard ports, and more. What's worse, when attackers successfully use an evasion technique to bypass defenses, they leave no trace. There are no logs, and there are no alerts. Find out which products were the most- and the least-resistant to evasions in our latest BDS Group Test.

Stability - Cyber Security Tests

An unstable device that disrupts traffic unexpectedly can have disastrous consequences. Failure can result in serious breaches going undetected and thus not being remediated. Testing revealed some products had stability issues with certain versions. Find out which ones.

Performance - Cyber Security Tests

The rapid adoption of social media, streaming video, and other bandwidth-intensive technologies means real-world traffic patterns are changing constantly. Find out which BDS products successfully balanced detection with performance.

Security Effectiveness - Cyber Security Tests

For a BDS, security effectiveness is determined by its ability to detect and log breaches accurately while remaining resistant to false positives. Additionally, it should be able to detect both known and zero-day malware. A system's overall security effectiveness score is determined by factoring the results of evasion testing and stability and reliability testing into its breach detection rate. In the BDS 5.0 Group Test, the Security Effectiveness of the tested products ranged from 77.2% to 99.6%. Find out how the different products fared.

Total Cost of Ownership - Cyber Security Tests

Your security savings don't kick in when you purchase a product. One reason for this is because the additional operational overhead cost required to remediate infections and incidents will negate any security savings and result in high TCO. Find out which products have the lowest TCO over a multi-year period so you can better analyze risk and make informed purchasing decisions for your organization.

Value To You - Cyber Security Tests

Your BDS should be able to detect and report on threats in a timely manner, so you don't find yourself in data breach situation. Just because a product is the least expensive doesn't mean it provides the best value or meets your needs. NSS Labs can help you determine which products are right for you.

Is Your Enterprise Looking for Insights in addition to the BDS Test Reports?

Research Analysts Help - Cyber Security Tests

Still have questions? Ask us! Any time we conduct a test, there is more information than we can possibly include in our test reports. NSS Research Analysts help clients from the world's most demanding organizations get the answers they need.

Who We Tested


What are the Key Takeaways from BDS 5.0 Group Test?

TCO per Protected
Ranged from
US$58 to US$450

All 3 Products Achieved
a Recommended Rating

Overall Security
Between 77.2% and 99.6%

Time to Detect and Alerts - Cyber Security Tests

The time it takes for malware to be detected can influence the overall impact of an attack. The more time an adversary has to operate after breaching a network, the greater the possible damage. It is important to know which products provide timely alerts on attacks, potential malware infections, or C&C callback within the time-to-detect window.

Attack Vector - Cyber Security Tests

Attackers are using different vectors to deliver malware. That is why the BDS 5.0 Group Test introduced various real-world scenarios to determine security effectiveness. Tests were run using web-based malware attacks that rely on social engineering, drive-by downloads, socially engineered malware delivered through email and USB device, blended exploits, and offline infections. Results from the BDS 5.0 Group Test reveal which products will protect you from these attacks.

Evasion Techniques - Cyber Security Tests

Providing results for a product's detection capabilities without fully factoring in evasions can be highly misleading in terms of understanding a product's security efficacy. That is why NSS Labs' security effectiveness score includes evasion techniques. The more classes of evasion that are missed (e.g., binary obfuscation, packers, compressors, HTML obfuscation, HTTP evasions, VM and sandbox evasions, metamorphic and polymorphic evasions, layered evasions, and attacks on non-standard ports), the lower a product's security efficacy. In the BDS 5.0 Group Test, products were tested against 374 evasions to evaluate how well they were able to detect evasions.

Paths Trigger Vulnerability - Cyber Security Tests

When an attacker is presented with a vulnerability, the attacker can select one or more paths to trigger the vulnerability. NSS measured the resiliency of BDS products in the test by introducing a vulnerability along with its triggers and then asking the products to protect against the vulnerability. By testing resilience, NSS Labs shows you which BDS products detect different variations of an exploit.

Total Cost of Ownership (TCO) - Cyber Security Tests

No two network security systems deliver the same security effectiveness or performance, making precise comparisons extremely difficult. From a security perspective, to quantify the projected cost of operating a business as part of the TCO model, three primary components were used to calculate the TCO per Protected Mbps: Projected Total Cost (No Security), Expenses, and Cost Savings (With Security). From a financial perspective, the TCO model is calculated using three-year TCO, security effectiveness, and NSS-tested throughput. Since each BDS has a unique set of capabilities, the TCO metric assigns a value that normalizes the operational overhead, potential consequences, and associated costs of a security breach for an organization. This allows decision makers to better analyze risk and make calculated choices that are in the best interests of their organizations.

Product Tuning - Cyber Security Tests

NSS research has found that BDS often require little or no tuning, and in fact, several vendors provide products with little or no tuning options. However, where possible, all BDS products must be tuned prior to testing to eliminate false positives and provide the most appropriate coverage for the systems they are protecting.

NIST Special Publication 800-53 (Rev 4) SC-44 calls for organizations to employ a detonation chambers, also known as dynamic execution environments.

NIST Cybersecurity Framework - Cyber Security Tests

Detonation chambers allow organizations to open email attachments, execute untrusted or suspicious applications, and execute URL requests in the safety of an isolated environment. The BDS is being used as an effective technology by organizations that are adopting NIST 800-53 (Rev 4) SC-44.

Do you know how effective your BDS technology is at identifying malicious code in order to limit the lateral movement of threats within your organization?

BDS Product Test Example Reports

Security Comparative Report

Security Comparative Report

The Security Comparative Report provides high-level analysis of the security effectiveness of different BDS products in the market. The report provides comparisons of blocking capabilities, stability and reliability, and resistance to common evasion techniques.

Using this report, enterprise security teams can compare security effectiveness and resistance to evasion techniques across different BDS products.

Security Comparative ReportSecurity Comparative Report

Example Report

Performance Comparative Report

Performance Comparative Report

The Performance Comparative Report provides analysis of various performance metrics for tested BDS products. The report contains comparisons of maximum capacity, HTTP connections per second and capacity, and UDP throughput and latency, all while using real-word traffic mixes.

Using the Performance Comparative Report, enterprise networking teams can compare performance across BDS products and select those that will support their volume and type of network traffic.

Performance Comparative ReportPerformance Comparative Report

Example Report

Total Cost of Ownership TCO Comparative Report

Total Cost of Ownership (TCO) Comparative Report

The TCO Comparative Report provides a comparison of the costs associated with product purchase, installation, maintenance, and support, as well as threat-associated costs.

Using the TCO Comparative Report, the enterprise C-Suite and management can understand the true TCO of a product over a three-year period, incorporating product purchase cost, product operational cost, and the overall capability score of a product.

Total Cost of Ownership TCO Comparative ReportTotal Cost of Ownership TCO Comparative Report

Example Report

Security Value Map SVM Comparative Report

Security Value MapTM (SVM) Comparative Report

Empirical data from individual Test Reports and Comparative Reports is used to create NSS Labs' unique Security Value Map (SVM). The SVM illustrates the relative value of security investments by mapping the Security Effectiveness and the Total Cost of Ownership (TCO) per Mbps of tested product configurations.

The SVM Comparative Report provides an aggregated view of the detailed findings from the NSS Labs group tests. Using this report, enterprise security decision makers can see the relative value of security investments.

Security Value Map SVM Comparative ReportSecurity Value Map SVM Comparative Report

Example Report

Individual Vendor Test Reports

Individual Vendor Test Reports

Test Reports provide detailed analysis for each product tested. Data from these reports is used in the NSS Labs Comparative Reports.

Test Reports enable enterprise security teams to understand the impact of features and limitations across different products. These reports are used to shortlist products for further evaluation and proof-of concept testing.