Re: Re: Upcoming new test-suite release

Stuart D. Gathman ha scritto:
> On Mon, 18 Aug 2008, Frank Ellermann wrote:
> 
>>> Next stop: 2008.08 release announcement.
> 
> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible.  I am running
> against pyspf right now.  Any new failures need to be diagnosed
> as an implementation or test-suite problem.  It is not enough just
> to eyeball the changes.

I tested 2008.08 against jSPF and everything works.
I also tested current head for the test suite and we fail the macromania 
test.

We didn't correctly expanded the %_ and %- (they was explanded to "_" 
and "-" :-( ).

I have already an easy fix for that (It's very good to have this test 
suite!!) ready to be committed.

At Apache JAMES project we are also planning to extract the test suite 
for jSPF and publish it as a separate project (we currently have it in 
our tests).

The jSPF tester have the ability to load a zone configuration from the 
yaml file, set up a "fake" dsn server on a given port, read tests, run 
the spf checker and check the results.

One issue I've found when trying our tester against other 
implementations is that it is not easy to tell other implementations to 
use a specific dns server/port for the resolution.

IMHO the jSPF tester could be a good way for OpenSPF to certify the 
implementations, otherwise you will have to manually check if the test 
suite for each implementation is correct or not.

Stefano