Re: NSS Certification - Credible?

I would contend that this is "best of a bad thing."

I have done an enormous amount of testing myself on network security products 
for over 20 years, and Bob Walder's NSS tests are the best out there.

The first thing you have to understand is that this kind of testing is VERY 
expensive; it costs a lot of money for the equipment, but it costs even more 
money for the time.  Only when a lab like NSS is actually getting paid do they 
have the luxury of doing a very good job.

When we test for publications like Network World, we are on a dramatically lower 
budget--we'll test 5 to 10 products for about a 10th of what NSS charges to test 
a single product.

I think that the "certification" thing is a pile of crap (not just with NSS, but 
with every vendor that offers a 'check mark' or 'gold' or 'certified' level). 
However, what comes out of NSS, in addition to the useless badges, is an 
ENORMOUS report based on what they actually saw and didn't see.  That's the 
value of their work, and that's why I continue to believe that they are the best 
private test lab in our space.

Yes, all of the criticisms you mount (such as the ability of the vendor to have 
a 'do over') are valid, but if you want someone who at least has the veneer of 
independence (despite their being paid by the vendor), then the NSS reports are 
very worthwhile reading.

This may change over time---it's no longer Bob and the South of France; it's now 
a real company in the US with bigger pressures to perform.  And this is what has 
caused other previously-reputable testers to have lost their reputation.

So, take it with a grain of salt, but anyone who does NOT read the NSS reports 
on products that they have tested is cutting themselves off from a huge supply 
of very high quality data. I won't make that statement for most of the other 
"labs" out there who are doing commercial testing.

jms

Disclosure: I've never taken money from NSS, ever.  I'm just a fan.


Ravi Chunduru wrote:
> One interesting and provactive slide "Effectiveness" here:
> http://nsslabs.com/webinars/NSS%20Labs%2010g%20webinar.pdf
> I agree some what  with what was said there, but testing with private
> exploits alone does not make NSS testing credible.  I feel that there
> are some points which IDP buyers would like to know while selecting
> the IDP vendor.
> 
> How many times vendor failed in testing before the product was
> certified?  My understanding is that NSS allows vendors to provide
> signature pack during testing if it does not meet the pass criteria.
> Shouldn't this failed number be known to IDP buyers?  I also  feel
> that buyers would like to know the Initial coverage number.  Without
> that I don't see the difference between public testing houses and NSS.
> 
> To make buyers comfortable, I believe testing should be done
> periodically (Once in a month?) on certified products and take them
> off the certified list if they don't meet the criteria.  I noted that
> there are some products in the certified list dating back 2004/2005.
> 
> From the test report, it appears that NSS certifies if 30-40% of
> client side attacks are detected.  Are buyers comfortable with this
> number?
> 
> Number of tests made are dismal around 500+.  Does that number good enough?
> 
> Buyers know their internal assets (protocols, applications, operating
> systems etc..) and would like to see certifications providing detailed
> information on security effectiveness of common protocols and
> applications.  I don't see these details on NSS reports.  I am not
> sure whether this was the intention of testing by public houses, but
> one knows clearly on products and their coverage with respect to
> vulnerabilities and exploits.
> 
> By the way, are there any testing & certification houses targeting
> measurement of security coverage with respect to individual protocols
> servers HTTP, FTP, SSH, SIP, LDAP, SQL Server etc.?
> 
> Thanks
> Ravi
> 
> 

-- 
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One       Phone: +1 520 324 0494
[email protected]                http://www.opus1.com/jms