By Vikram Phatak
On Tuesday - February 14, 2017 - NSS Labs released the results of our Advanced Endpoint Protection (AEP) group test. You are probably aware that CrowdStrike sought to prevent NSS from testing their product and then sued us in Federal Court, where they sought a Temporary Restraining Order and Preliminary Injunction to prevent us from publishing our findings. You are also probably aware that the Federal Court denied their request. (http://www.ded.uscourts.gov/sites/default/files/opinions/gms/2017/february/17-146.pdf )
Then, despite the court’s ruling and memorandum, CrowdStrike published a blog accusing NSS Labs of various nefarious doings.
Why did we wait to publicly respond to CrowdStrike? We felt the CrowdStrike management team might need some time to reflect on the choices they made, and that some distance would provide them the opportunity to make a course correction. Unfortunately, nothing has changed.
So here are our thoughts on the matter:
NSS Labs is committed to providing our enterprises customers with accurate test results so that they can make informed decisions. We anticipated that testing products in the endpoint market would meet with resistance given the historical relationship between vendors and testing organizations. NSS does not charge any vendor for participation in any of our public group tests. The entire test is done on our dime, and all we ask from vendors is that they provide us with their product, along with engineering support before and during the test, should we need it.
From NSS Labs’ perspective, it comes down to the public’s right to know how a product stacks up against attacks, and in comparison to other products in the market. Our test reports are a source of empirical and unbiased information required to distinguish fact from marketing hype. We have witnessed that in other cyber security markets the effectiveness of products improves when vendors compete based upon measurable results.
So we are disappointed that in the weeks since the AEP group test was published, CrowdStrike did not reach out to NSS to understand the attacks and evasions they missed. Instead, they have made a concerted effort to obfuscate and divert attention away from their test results – vilifying NSS in an effort to justify their actions. The question the people should be asking is, “Do CrowdStrike’s actions serve the public interest? Do they help make their customers safer?”
In their blog, CrowdStrike mentions two other vendors, Palo Alto Networks and FireEye, with whom we have had public disagreements in the past. But unlike CrowdStrike, these companies did (despite unfortunate first responses) put their customers’ interests first, and worked with NSS to understand and fix deficiencies in their products.
Given the serious inaccuracies CrowdStrike has been promoting in their blog and elsewhere, we decided that we needed to tell our side of the story:
NSS’ report plainly states that testing of CrowdStrike Falcon was incomplete, and therefore, the results are invalid.
We were unable to complete testing of the CrowdStrike Falcon Host product via a few attack vectors because CrowdStrike remotely disabled the product partway through our tests. However, during testing of the attack vectors that we were able to successfully complete, their product missed numerous attacks – those attacks would still have been missed had CrowdStrike not remotely disabled the product.
Including Falcon in the report based on an incomplete analysis is contrary to basic industry standards for testing.
NSS only included results for which we completed testing. There was no penalty in the score for tests that were not completed – CrowdStrike did not receive a zero (0) for the parts of the test we were unable to complete - because we believed that penalizing CrowdStrike for disabling the product could mislead the public.
All Falcon prevention capabilities were disabled during the testing, and therefore the report results are wrong.
Had this been true, then Falcon could not have prevented any attacks during our tests, which is not true.
CrowdStrike declined to participate in a public test after completing a private test with NSS, based on NSS’ flawed and improper testing execution.
Participation in an NSS group test is not at the vendor’s discretion – if you are an identified market leader, or if our enterprise clients want to see your products tested, then we will test them. It is always worrying when a vendor is resistant to having its product tested – we have found it to be a reliable indicator the vendor knows something is not working as well as it should.
1. In April and then in August 2016 CrowdStrike engaged NSS Labs to perform private testing to determine how an attacker could bypass their product.
Engineering deep dives are intended to find protection flaws in a vendor’s product.
We make it clear to any vendor who hires NSS to “break” their product that they will not receive the “questions” or “answers” to a public group test.
We also make it clear to any vendor who hires NSS to break their product that they cannot use the fact that we found flaws to avoid participation in public testing.
CrowdStrike’s private engineering engagement results are under NDA, therefore we cannot disclose those particular findings.
2. In December 2016, CrowdStrike told NSS Labs that they did not want to be publicly tested. NSS Labs informed CrowdStrike that our position, as always, is that if a product is good enough to sell to the public, it is good enough to be tested and that we would purchase their product if necessary.
3. In January 2017, CrowdStrike continued to state that they did not want their product to be tested, so we attempted to purchase the product. CrowdStrike blocked that purchase. So we found an enterprise who would be willing to work with us to purchase the product.
4. In January 2017, NSS Labs conducted a public test of Advanced Endpoint Products. Partway through the test, CrowdStrike disabled their product, preventing us from completing the test. They then tried to get their enterprise customer and the reseller to help them block publication of the public test results.
5. As stated above, on February 10, 2017, CrowdStrike filed a lawsuit (under seal) against NSS Labs claiming Trade Secret violation and seeking a Temporary Restraining Order (TRO) and Preliminary injunction to prevent NSS from publishing CrowdStrike Falcon Host test results.
6. On February 13, 2017, a Federal Court judge denied their request for a Temporary Restraining Order (TRO) and Preliminary injunction.
7. On February 14, 2017, NSS Labs published the results our AEP group test including partial results of their Falcon Host product.
8. On February 14-15, 2017, CrowdStrike responded with a blog post that attempted to obfuscate the facts and justify CrowdStrike’s actions by portraying NSS Labs as a villain.
We sincerely hope CrowdStrike executives will begin to take the results of this test seriously and move quickly to protect their customers. As ever, NSS is more than willing to work with any vendor to identify and fix problems and help make our networks and computers safer. Further, if/when CrowdStrike fixes its product, we will try to test it and publish the results. Until that happens, we are also happy to work with CrowdStrike customers to help minimize their risk as far as possible given the current limitations of the product.
Thank you for your continued support.
Vikram Phatak, CEO
Follow us on Twitter (@NSSLabs) to keep informed as new research is released.