Re: PCI DSS & Firewalls

Chris Blask wrote:
> having more Pen Testing done in the world is itself a move in a positive direction, so that's a good thing by any metric.


I disagree.

What does pen testing show?? Pen testing can show one of two things:
- your security sucks
- your security is better than your pen tester

Neither of those two determinations are equal to "your security is
good."

Ultimately, any kind of "security proofs" attempt to prove a negative:
i.e: "there are no security holes" and simple logic tells us that you
can't prove a negative.

The reason pen testing is popular - in spite of the fact that it
is a flawed idea - is because "your security sucks" is still a
useful answer for a lot of organizations. I'd go a step further
and suggest that if the answer is "your security sucks" there's
a root cause and it's that "your managers are stupid" or "your
executive management is clueless" or both. Those are not especially
popular results but we both know of infinite numbers of stories of
executives who didn't take security seriously until some pen
test rubbed their nose in it. Pen testing may be a short-term
cure for stupid, but it's a fairly expensive way of doing
it and I doubt that it works particularly well in the long-term.

If we were to ever move security past the "your security sucks"
stage, it would have to result from systems being designed with
security built in from the ground up, rather than bolted on
(or, more likely, as the case is, stuck on with bubble gum
and duct tape) after it's safely too late. Don't worry about
that happening any time soon, though - Web2.0 and cloud
computing are in the process of blowing a gigantic smoking
hole through any notion of trust in computing. How do you
make a statement about assurance and critical data in an
environment where, by design, you aren't to know anything beyond
"it's in our cloud; trust us" ??  I am guessing that the
pen testers are already drooling at the feast to come.

As they used to say, "you can't make a silk purse out of
a sow's ear" - implying that there's no amount of improvement
that you can make to something that just isn't capable of
meeting your expectations. The same applies to pen testing:
it is impossible to badness-test your security into being good.
If you try, all you'll find is that it's expensive. It's
only a coincidence, I'm sure, that the badness-testers are
standing by. There are also duct tape and bubblegum sellers
standing by. Its all coincidence.

So, generally I disagree with you, Chris. I think pen testing
serves as an indicator of stupid more than anything else.
Don't be confused by the fact that the indicator is in the
red zone; it doesn't mean what you think it does.

mjr.
-- 
Marcus J. Ranum		CSO, Tenable Network Security, Inc.
			http://www.tenablesecurity.com