Next Generation Firewall Reports

NSS Labs Latest Tests Show Next Generation Firewalls Rapidly Maturing To Meet Needs of the Enterprise

The Next Generation Firewall (NGFW) Security Value Map and Comparative Analysis Reports evaluates 9 of the leading NGFW products on the market for security effectiveness, performance, enterprise management capabilities and total cost of ownership. This was the second group test for NGFW that NSS has conducted and overall there was marked improvement from most vendors’ 2012 test scores.

Key conclusions:

  • Check NGFWs’ firmware before deployment:  Out of a total of 9 products tested, 6 vendors submitted products that required firmware updates or configuration changes to complete the NSS tests. Only Check Point, Fortinet and Stonesoft submitted products that worked the first time.
  • New Metric Highlights Enterprise Management Failings: If a device cannot be managed effectively, the security effectiveness of that device is compromised. As part of this test, NSS performed in-depth technical evaluations of all the main features and capabilities of the enterprise management systems offered by each vendor and factored it into the final score as a new and unique metric called “managed security effectiveness”. Managed security effectiveness scores ranged from 29.1% to 98.5%.
  • NGFWs’ Security Effectiveness Scores Improve Significantly:  In the latest 2013 tests, 8 of the 9 products scored over 90% for security effectiveness (excluding management). This is a marked increase compared to 2012, when only half of tested vendors scored above 90% in this category. The overall scores for security effectiveness in 2013 ranged from 34.2% to 98.5% compared to 18% to 98.9% in 2012.
  • Total Cost of Ownership Remains Fairly Stable:  While the overall range of TCO decreased in 2013 testing, prices per protected megabit per second remained fairly stable with most tested devices costing below $44 per Protected-Mbps. The overall 2013 range was $18 - $124 per Protected Mbps, down from a range of $30 - $375 in 2012 testing.
  • More Vendors Back their Performance Claims:  Only 2 of 9 products tested had throughput rates that were significantly less than their vendors’ stated claims. In 2012 testing, 5 of the 8 products tested performed well below their advertised speeds.  In 2013, three vendors – Dell SonicWALL, Sourcefire and Palo Alto – performed better in tested performance than their stated throughput and two vendors – Check Point and Stonesoft – had throughputs that were virtually equal to their stated performance.


Receive Our Free Guide: What You Need to Know About NGFW

Available Reports


Inside the Reports

Each of our reports is based on thousands of repeatable test cases. Below are just a few examples of they types of rich information you will find inside NSS Labs research. Looking for something else? Ask an analyst.

Over 1,500 live exploits and evasions
Target role: Security team

How exposed are your assets despite your security investment? Without comprehensive, accurate testing, there’s no way to know for sure. With NSS Labs reports you can look inside and see how well specific target assets are protected.

But not all testing is equal. Proper security effectiveness testing is hard to do right. Running pre-canned tests and replay tools can be inaccurate and is not a realistic representation of the current threatscape. A test that does not utilize a sufficiently large live attack library targeting mainstream applications can impart a false sense of security. NSS selects and validates over 1,500 exploits relevant to large enterprise networks before using them in testing.

Furthermore, using mindset and techniques of motivated attackers, we apply common and layered evasions to the testing. Missing just a single evasion can completely negate the threat protection in your NGFW or IPS.

Connection dynamics and their real-world impact
Target role: Network team

How do you buy the right sized devices for your network, for today and the next few years? Making the wrong choice can have costly repercussions, in terms of reliability, productivity, security and cost. Security products impact performance in many ways, some of which are difficult to identify without the right analysis. NSS testing has found that vendor performance claims are often exaggerated, between 30 and 125% on average. And some “10G devices” have failed to pass more than 500Mbps.

But, finding the right sized device for your network is not just about throughput. The number of new connections per second a device can establish is a critical factor; some current devices have severe CPS limitations despite relatively high throughput. And a device with high latency can cause unacceptable user response times and application failures, leading to help desk calls and lost productivity.

Trying to guard against these issues causes many enterprises to overbuy. NSS Labs tests devices against hundreds of real-world metrics and industry-specific traffic mixes. Knowing  how a security device will behave in your network, will help you make the right decision.

All new management criteria and analysis
Target role: Operations team

Speeds and feeds mean nothing if you cannot manage the device effectively. It is like having the fastest car on the block and finding the steering doesn't work. You will drive into the wall.... Really fast!

In large-scale enterprise deployments the central management systems becomes significantly more important – a poor management system means that critical tasks are often overlooked or performed in a sub-optimal manner. All too often this vital element of the security system is overlooked in testing. This is why NSS Labs has introduced a brand new and extensive management methodology to determine strengths and weaknesses of NGFW management.

What's the right security investment?
Target role: C-Level/financial personnel

Organizations must balance competing factors to achieve their goals within budget. Smart buyers know the least expensive product does not necessarily offer the greatest value if it does not protect your assets and meet your performance requirements. But how do you optimize for security, performance and cost?

There is no magical answer, but there is a scientific one. It is embodied in NSS Labs’ proprietary Security Value Map (SVM), the industry’s only truly empirical quadrant analysis tool. It maps the absolute value of different product configurations in terms of TCO, performance and security. This detail-rich graphic provides 100% empirical information, which NSS Labs clients can modify and tailor to their own environment and needs to take the magic out of apples-to-apples value comparisons.

Completely independent – not vendor funded

NSS Labs is unique. NSS Labs is the only provider of information security research, analysis and advice that is derived from rigorous and unbiased testing. We do not accept advertising or vendor sponsorship for any published research, and we do not sell products. Our mission is to provide our enterprise clients with the most expert, accurate, and unbiased information possible in the market today, backed 100% by empirical evidence, not just analyst opinion.

There are analyst firms, but they don't test. There are testing houses, but they are not independent, nor do they perform analysis. And there are many VARs, but these lack vendor independence and scientific testing and analysis capabilities. NSS Labs is the only completely independent research and analysis organization with in-house testing capabilities. NSS has no ties with publishers, carriers or product vendors, and has no parent company to introduce conflicts of interest.

Be cautious of certifications and product reviews that don’t have the rigor and depth of testing, as well as sponsored evaluations, reports and white papers. Each comes with its own inherent bias. Get the hard facts & don't let your organization be the crash test dummy for unproven technology.


Go to top