Re: Re: Upcoming new test-suite release

Stefano Bagnara ha scritto:
> Julian Mehnle ha scritto:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Stefano Bagnara wrote:
>>> Probably running java applications is difficult for you as running perl
>>> applications for me.
>>> How easy is for me (let's say I only understand java) to run the
>>> testsuite for other implementations?? Hard. It's not jSPF that is hard,
>>> it is simply that the test suite for a library is usually written in
>>> the library own language and using that very language testing
>>> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
>>> tests, so it is really easy to run our test suite graphically in any
>>> jUnit compatible environment.
>>>
>>>> So please don't be mad, but let's now try to get this resolved
>>>> instead.
>>> My complain is that either you publish a procedure that I will try to
>>> follow or you say that jSPF is compliant. It doesn't work and it's not
>>> fair for me that you say "I don't know Java so your implementation will
>>> not be listed".
>>
>> We don't have a formal procedure, and I don't think establishing one 
>> will be worthwhile to any extent.  We should simply strive to be 
>> fair.  If we aren't, please hit us on the head.
>>
>> The issue with your implementation wasn't that I don't _speak_ Java, 
>> but that there was no good documentation on how to set it up (cf. 
>> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you 
>> will) for someone who isn't familiar with know how to _use_ Java, 
>> Maven, and jUnit, and, even more importantly, the test output was 
>> incomprehensible to me.  Perhaps that has since changed, but back then 
>> that was just how it was.
> 
> In every distribution we have BUILDING.txt:
> -----
> ############################################################################# 
> 
> # BUILDING JSPF
> ############################################################################# 
> 
> 
> You will need to install maven2 as well as acquire jSPF source from 
> subversion
> or a source tarball.
> 
> Steps:
> 
> 1) Install maven2 (v2.0.7 as of the time of this writing)
> 
> 2) Add maven2 to your path.  For me, I do the following:
>    $ tar zxvf maven-2.0.7.tar.gz
>    $ mv maven-2.0.7 /usr/local
>    $ ln -sf /usr/local/maven-2.0.7/bin/mvn /usr/local/bin/mvn
> 
> 3) Change directory to jspf source dir
>    $ cd jspf-x/
> 
> 4) Run the build (the "-Plocal" restrict maven to use bundled dependencies)
>    $ mvn -Plocal package
> -----------------
> 
> It doesn't seems more difficult than the instructions you give for the 
> perl implementation.
> 
>> Can you produce somethink like the following?
>>
>>   http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log
> 
> A list of tests with an "ok" on their side??
> The above "mvn package" command will run tests. The output will be more 
> verbose, but if any failure happens the package is not build and a 
> "BUILD FAILURE" appears.
> 
>> You'll see that this isn't exactly good proof that an implementation 
>> passes the test suite, but it at least demonstrates that the test 
>> suite has been run in some way.
> 
> We run it as part of our continuos integration build.
> We run it at least 200 times per day and at least once per each commit.
> 
> If you just need a log of the test run to certificate jSPF I'm happy.. 
> I'll send you the next output.

Here is the last successful build test result. I think it's more fancy 
than your perl log, isn't it?

http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/

We also run more tests as you can see here:
http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/

Because we also have an asynchronous resolver.

The test above is for our latest trunk tested against the latest yaml 
testsuite from your trunk.

We only make releases when our product passes all of the last yaml suite 
tests.

In fact our build tools reject to build jSPF at all if there are failures.

Stefano