Assurance measures: ATE (well, ehhrmm)

Hi!

The open source development method have a very different
approach to tests from the commercial development method:
Testing is a largely uncoordinated effort, driven by the
individual needs of users. That means that zero to none
evidence is gathered on tests actually ran, and no structured
approach can be claimed on it. Not if testing would not be
made, just the depth and coverage of them cannot be
establised.
Of course there are examples on the contrary, like some
packages which run elaborate tests on 'make test' or=20
www.linuxtestproject.org, but they are still enough only
to EAL2, which is close to nothing.
My favourite example is happydoc: the author first writes
test cases, and the code _after_ that. It might even enough
for ATE_DPT.3, which requires tests against the implementation
representation (code).

ATE_COV.2 Analysis of coverage (EAL3-EAL5)
ATE_COV.2.1D   The developer shall provide an analysis of the test
	coverage.
	(www.linuxtestproject.org)
ATE_COV.2.1C   The analysis of the test coverage shall demonstrate the
	correspondence between the tests identified in the test
	documentation and the TSF as described in the functional
	specification.
	(See ADV_FSP to realize that this is the second step to take)
ATE_COV.2.2C   The analysis of the test coverage shall demonstrate that
	the correspondence between the TSF as described in the
	functional specification and the tests identified in the test
	documentation is complete.
	(Writing tests is boring, eh?)

ATE_DPT.1 Testing: high-level design (EAL3, EAL4)

ATE_DPT.1.1D The developer shall provide the analysis of the depth of
	testing.
	(the lack of this may be the root cause that SLES got only
	EAL2)
ATE_DPT.1.1C The depth analysis shall demonstrate that the tests
	identified in the test documentation are sufficient to
	demonstrate that the TSF operates in accordance with its
	high-level design.
	(Writing tests is still boring.)

ATE_FUN.1 Functional testing (EAL2-EAL5)=20

ATE_FUN.1.1D The developer shall test the TSF and document the results.
	(If one is writing tests, it is relatively easy to comment
	them.)
ATE_FUN.1.2D The developer shall provide test documentation.
	(...and write some pages about their structure.)
ATE_FUN.1.1C The test documentation shall consist of test plans, test
	procedure descriptions, expected test results and actual test
	results.
	(The test script with comments, plus an overview doc is the test
	plan, and the proc. description, the expected test results are
	some files with the expected output, and actual test results
	can be gained by running the test suite.)
ATE_FUN.1.2C The test plans shall identify the security functions to be
	tested and describe the goal of the tests to be performed.
	(Nice objective for the comments of the test scripts.)
ATE_FUN.1.3C The test procedure descriptions shall identify the tests to
	be performed and describe the scenarios for testing each
	security function. These scenarios shall include any ordering
	dependencies on the results of other tests.
	(The code does that, the scenario is "run make;make tests":)
ATE_FUN.1.4C The expected test results shall show the anticipated
	outputs from a successful execution of the tests.
	(Easy, if we talk about test scripts.)
ATE_FUN.1.5C The test results from the developer execution of the tests
	shall demonstrate that each tested security function behaved as
	specified.
	(the diff command helps a lot:)

ATE_IND.3 Independent testing - complete (EAL7)
ATE_IND.3.1D   The developer shall provide the TOE for testing.
	(apt source)
ATE_IND.3.1C The TOE shall be suitable for testing.
	(seems easy, if the other requirements are fulfilled)
ATE_IND.3.2C The developer shall provide an equivalent set of resources
	to those that were used in the developer's functional testing of
	the TSF.
	(an account on the designated devel machine)

--=20
GNU GPL: csak tiszta forr=E1sb=F3l