RE: ICSA Labs Network IPS Testing

Rahul:

It's true that the baseline set of criteria does focus on remotely
exploitable, server-side vulnerabilities found in enterprise software.
What is less well known is that there is an optional criteria module
focused entirely on coverage protection for client-side vulnerabilities
(also in enterprise software).  As with the evolving server-side set of
vulnerabilities, this client-side set is published online as well.  You
can find it here:

http://www.icsalabs.com/icsa/docs/html/communities/nips/criteria/Vulnera
bilitySet_ClientSide_070703.xls

So, when we do the research twice a year (approximately) to determine
the vulnerability set - it's done for both the server-side set required
by the baseline set of criteria as well as for the optional client-side
set.  

You probably noticed that no one has been tested successfully against
this optional module.  Enterprise end users may have to demand that
products be tested for client-side vulnerability coverage protection (if
they are interested in ensuring proper protection for attacks targeting
both sets) before developers will pursue such testing.  I can and have
recommended it, but it really does take a push from end users.

As I said before, if folks have questions about ICSA Labs Network IPS
testing, please feel free to get in touch.

Take care,
Jack Walsh
Technology Programs Manager,
Intrusion Detection & Prevention
ICSA Labs
717.790.8126
[email protected]

> -----Original Message-----
> From: [email protected] 
> [mailto:[email protected]] On Behalf Of Rahul K
> Sent: Tuesday, December 11, 2007 2:03 PM
> To: Stefano Zanero
> Cc: Focus-Ids Mailing List
> Subject: Re: ICSA Labs Network IPS Testing
> 
> Hi,
> 
> Having some experience in developing and testing IPS, I have my two
> bits to add. Most IPS tests, like Stefano said, are tricky at best and
> pointless at worst. I don't want to take any potshots at ICSA or
> anyone else, but it is not simple for anyone to do an exhaustive test
> of an IPS and that too with the same test plan for every IPS.
> 
> ICSA, to their credit, say that of all the vulnerabilities they will
> only focus on remote server-side vulnerabilities and that too only
> those that they (and other vendors) think will affect enterprises.
> Fair enough. They don't care about client-side vulns, local vulns and
> vulnerabilities in Shoutcast.
> 
> They test a particular subset (however small it may be) and certify
> the IPS. So even if one buys an IPS that blocks all server side
> attacks launched by ICSA, it does not mean that the server behind the
> IPS is secure from remote attacks. Vendors and buyers need such
> certifications so that it is easier to make a sale and deploy an IPS
> respectively - after all, not everyone subscribes to focus-ids.
> 
> It would be reasonable to criticize ICSA if one finds out they are not
> doing what they promise correctly. But if the criticism is for not
> testing exhaustively, that seems excessive.
> 
> Cheers,
> Rahul
> 
> On 12/5/07, Stefano Zanero <[email protected]> wrote:
> > Hi, didn't mean to interfere in your ongoing flame, but:
> >
> > > IPS certification testing, I thought I ought to correct 
> some misleading
> > > information
> >
> > Oh, good, let's see! You don't mind if instead of going through your
> > whitepapers I just use your own email as a source, right?
> >
> > > IPS certification testing program.  The truth is that we 
> do not "pick
> > > specific attacks and say that you must block these."
> >
> > That's wonderful to hear. So, what do you do instead?
> >
> > > provides coverage protection for all attacks targeting an 
> evolving set
> > > of medium-to-high severity vulnerabilities that we and a 
> consortium of
> > > 15 network IPS vendors
> > > 
> (http://www.icsalabs.com/icsa/topic.php?tid=6a87$5813f3e2-37b7
7ee3$3b4a-
> > > f1d4a32d) believe are relevant to enterprise end users.
> >
> > So, you pick specific attacks (which are a snapshot of a set of
> > vulnerabilities that you + the tested vendors believe are 
> relevant) and
> > say "you must block these", right ?
> >
> > This seems exactly the same sentence that Joel posted, only 
> a bit more
> > elaborate :)
> >
> > And just to shoot another shot in the dead horse of IDPS testing,
> > testing MISUSE based detectors (as most IPS are) on 
> "detection rate" is
> > pointless. Testing them on coverage is tricky at best, and does not
> > really provide any useful insight at all on IPS where (as 
> Joel pointed
> > out) having 60k signatures instead of 30k does not really 
> mean anything.
> >
> > Oh, and on a side note:
> >
> > >  a) is in no position to speak authoritatively about ICSA 
> Labs network
> > > IPS testing,
> >
> > The sheer fact that someone is "in no position to speak" about your
> > tests means that your tests are lacking. If a test is properly
> > documented and scientific, everybody is in a position to 
> speak about it.
> >
> > In the particular case of Joel Snyder, who has been doing excellent
> > tests for a long time, I'd say he is in a particularly good 
> position to
> > comment.
> >
> > If this email sounds harsh, well, it is. I just don't like people
> > commenting AGAINST other people, instead than pointing out 
> the specific
> > flaws in their posts.
> >
> > Best,
> > Stefano
> >
> > 
> --------------------------------------------------------------
> ----------
> > Test Your IDS
> >
> > Is your IDS deployed correctly?
> > Find out quickly and easily by testing it
> > with real-world attacks from CORE IMPACT.
> > Go to 
> http://www.coresecurity.com/index.php5?module=Form&action=impa
ct&campaign=intro_sfw
> > to learn more.
> > 
> --------------------------------------------------------------
> ----------
> >
> >
> 
> --------------------------------------------------------------
> ----------
> Test Your IDS
> 
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it 
> with real-world attacks from CORE IMPACT.
> Go to 
> http://www.coresecurity.com/index.php5?module=Form&action=impa
ct&campaign=intro_sfw 
> to learn more.
> --------------------------------------------------------------
> ----------
> 
> 

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw 
to learn more.
------------------------------------------------------------------------