Re: Re: Upcoming new test-suite release

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Stefano Bagnara wrote:
>> Stefano Bagnara ha scritto:
>>> Julian Mehnle ha scritto:
>>>> You'll see that this isn't exactly good proof that an implementation
>>>> passes the test suite, but it at least demonstrates that the test
>>>> suite has been run in some way.
>>> We run it as part of our continuos integration build.
>>> We run it at least 200 times per day and at least once per each
>>> commit.
>>>
>>> If you just need a log of the test run to certificate jSPF I'm
>>> happy.. I'll send you the next output.
>> Here is the last successful build test result. I think it's more fancy
>> than your perl log, isn't it?
> 
> This isn't about being fancy.
> 
>> http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/
> 
> That looks acceptable (though as I said it isn't exactly good proof).  Is
> there a _release_ version of jSPF that passes _that_release_ of the test
> suite?  And which release of the test suite was that?  2007.05?

Out last release (jSPF 0.9.6) has been done when 2007.05 was out and the 
tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08, it has 
1 issue in macromania that you added after 2007.08 in trunk.

Our macromania bug has been fixed in our trunk and the above report is 
for the trunk version and your trunk testsuite including the macromania 
test and our fix.

>> The test above is for our latest trunk tested against the latest yaml
>> testsuite from your trunk.
> 
> Please don't use "trunk" versions of our test suite (i.e., do not use the
> file named "rfc4408-tests.yml").  That's the development version.  Please
> use the files named "rfc4408-tests-YYYY.MM.yml" instead.

I use trunk in our trunk, so I can spot any issue as soon as possible. 
In past it happened that openspf added tests that we didn't pass and we 
complained for that and you finally removed them because they was 
controversial, so I think it is good to test against trunk in our dev 
snapshot.

I'll take care to add both the latest official and the latest trunk in 
our test tree, so that we always check both suites.

Stefano