RE: Another opinion on using extreme programming for security

All,

I read this list diligently, but rarely interject. This thread has explored some important misconceptions that need addressed. This post begins with a (potentially offensive) hypothesis which is used to address the few mini-threads present within this conversation.

[Hypothesis]
No process meets non-functional quality (such as Reliability or Security) goals intrinsically. 

--put another way--

Attaining non-functional quality goals often requires explicitly augmenting any process (whether it be XP, the Unified Process, or other processes) with steps, tools, and/or potentially expertise that tailor the process to meeting that quality goal. 

[Reliability, Security, and Set Theory]
The 'Reliability vs. Security: Which is the subset?' argument has been played out several times on a few lists. This argument repeats itself every so often on this list. If readers are interested in a good summary post, they can refer to Crispin's http://www.securityfocus.com/archive/98/304766 (be advised this post is larger in scope than this argument alone).

While there is a reasonably satisfying answer to this question, I submit that the real answer is that "it doesn't matter"; whether or not one quality goal is a subset of another doesn't matter. The reason: those organizations most effectively meeting a particular quality goal have adopted a custom way of assuring their software expresses that quality.
 
Organizations that effectively build fast software do different things than those that produce more secure software. Likewise, different means are employed to make software more reliable. Think about what you would do to produce more reliable software. Common answers would probably involve better unit testing, more integration testing, and system testing. Maturation would involve better tests, and might apply a coverage metric. As the process of test coverage improved, facilitative tools would be adopted and better test/data generation expertise would be developed. Some organizations would move closer to the 'front' of the lifecycle and focus on better requirements handling and traceability of those requirements into their implementation and tests.
 
Now, open your copy of 'Writing Secure Code 2' up to page 25. Here the authors present a set of process augmentations for developing more secure code. The additions involve threat modeling, (design-phase) security reviews, and yes testing (among others). The reader will note that the type of penetration testing suggested here is different (fundamentally) than the type of testing I described above for meeting Reliability goals, but we'll cover this later. Readers of this list should be familiar with the toolset being developed to combat security vulnerabilities. Code scanners, threat modeling tools, and other tools are far different than those afforded the tester looking at software's Reliability. Expertise too takes a different form. Rather than being able to ferret out bugs like an expert Tester, the Security Professional catalogs a rich database of implementation- and design-based vulnerabilities and attacks they seek to protect software against.

One can imagine the different activities associated with meeting Performance goals. Capacity planning, load balancing, and other techniques come to mind. My point should be clear. Each quality goal has supported an emerging and maturing set of process steps, tools, and expertise that (increasingly) are very different from each other: We're specializing.

[Why all this dirty water from my clean pipes?]
Each software process targets a different difficulty in building software. And as Mads' original post indicates there are advantages to using XP for security, as well as disadvantages. I'm not sure I 100% agree with Bishop's quoted text, but I'll handle that in the subsequent section. The important thrust of this section is that no matter what process one is using, security must still be an omnipresent focus in order to be expressed by deployed software. Microsoft has been using its security-aware process (page 25 of WSC2) and still turns out vulnerable software. Glib and unfair though this statement might be, it is fair to say there is no silver-bullet process answer to any quality goal. 

Those readers looking to adopt XP should strongly consider its weaknesses, their organizational quirks, and think about how security intersects those factors. They should augment XP in their environment to address those concerns. I assert that no one should blindly adopt a process and expect that the processes proclivities, as quoted by any security professional, will cause them to build secure software. Aspects of Security to consider with any process include (but are not limited to):

* How are security Stakeholders identified? 'Their interests understood?
* How are threats to those Stakeholders identified, assessed, prioritized?
* How is design shown to meet security goals/reqs.?
* How is the design proven 'good enough': free of security flaws?
* How is the code proven faithful to the design?
* How is the code proven 'good enough': free of security vulnerabilities (bugs)?
* How are threats, flaws, and bugs weighed against risks to the business? Stakeholders?
* How are attacks identified?
* How are attacks dealt with? 

Obviously, other factors should be considered including deployments, incident response and others.

[XP]
With respect to security and XP look at the tenants in a top-down fashion and work from there (I caveat the following by saying I'm not that familiar with XP, I've only used it a few times). XP embraces change. 

As requirements change, threats the software faces will change. Threat modeling should be done incrementally to evaluate prior threats that are no longer pertinent, new threats, or changing threats. As design or implementation changes, it will need to be re-evaluated for security flaws and vulnerabilities. Tools will help with vulnerability scanning, but are insufficient to stand on their own. Using such tools' output as input for security-savvy code review will help, but won't necessarily detect architectural flaws at a more macroscopic level. As for the testing impact security has on XP, please strongly consider that some components of security testing are absent in the unit testing practiced by agile methodologies. Risk-based testing (for example) is a type of testing that focuses on misuse and abuse cases within software. More than simple negative testing these efforts demand that the tester 'think like an attacker' and attempt to invalidate assumptions made by the software and underlying libraries, frameworks, platforms, and OSes. This type of testing is likely to reveal luring attacks on privileged components, overflow-type attacks, and other attacks where normal negative testing wouldn't.

Gary McGraw gave an interesting talk on security and XP, which is available at http://www.cigital.com/presentations/ but you'll need to email him for a password if you're interested.

[Conclusion?]
As a Software Consultant I've focused on a variety of non-functional quality goals such as Security, Performance, Availability, and Reliability. The client's I've helped have used a widely disparate processes including waterfall-like models, XP, and iterative (but heavier) processes as well. Those that meet goals like Security well have done so through constantly evolving means focused on that quality goal alone, not simply the 'reliability' of their product. I suggest that anyone seeking a high level of one quality goal focus on introducing process steps, tool use, and expertise into (and therefore customizing) their process, regardless of which one they use.

----
John Steven
Senior Technologist, Senior Consultant
Office of the CTO
Cigital Inc.


----------------------------------------------------------------------------
This electronic message transmission contains information that may be
confidential or privileged.  The information contained herein is intended
solely for the recipient and use by any other party is not authorized.  If
you are not the intended recipient (or otherwise authorized to receive this
message by the intended recipient), any disclosure, copying, distribution or
use of the contents of the information is prohibited.  If you have received
this electronic message transmission in error, please contact the sender by
reply email and delete all copies of this message.  Cigital, Inc. accepts no
responsibility for any loss or damage resulting directly or indirectly from
the use of this email or its contents.
Thank You.
----------------------------------------------------------------------------