Re: Re: Upcoming new test-suite release

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Stefano Bagnara wrote:
>> I tested 2008.08 against jSPF and everything works.
> 
> Very good.
> 
>> I also tested current head for the test suite and we fail the
>> macromania test.
>>
>> We didn't correctly expanded the %_ and %- (they was explanded to "_"
>> and "-" :-( ).
>>
>> I have already an easy fix for that (It's very good to have this test
>> suite!!) ready to be committed.
> 
> Even better!  Good to know that the test suite helped expose those 
> mistakes.  Mail::SPF 2.005 had a similar bug by expanding "%-" to "-", by 
> the way.
> 
>> The jSPF tester have the ability to load a zone configuration from the
>> yaml file, set up a "fake" dsn server on a given port, read tests, run
>> the spf checker and check the results.
>>
>> One issue I've found when trying our tester against other
>> implementations is that it is not easy to tell other implementations to
>> use a specific dns server/port for the resolution.
> 
> FWIW, it's all but trivial with Mail::SPF.  One can specify a custom DNS 
> resolver object, e.g., a Net::DNS::Resolver object configured to always 
> use a specific resolving server.

I'm not a perl programmer, so I won't do this. A configuration option to 
override the dns servers would do the trick.

>> IMHO the jSPF tester could be a good way for OpenSPF to certify the
>> implementations, otherwise you will have to manually check if the test
>> suite for each implementation is correct or not.
> 
> How will this help getting rid of any manual procedures in testing the 
> conformity of a specification?

This won't get rid of *any* manual procedure, but this at least save you 
from checking the code quality correctness for the testsuite runner.

> Besides, running an SPF implementations self-tests could be made (by 
> authors) as easy as `make test`.  It even almost works with Mail::SPF -- 
> you just have to run `perl Makefile.PL` beforehand.

The difference between this test and "selftests" is big, IMHO.
Most "self-tests" I've seen around don't actually run a real UDP/TCP dns 
query, but they mock the whole resolution mechanism at one or another layer.
Otherwise we could have a "selftest" returning "ok" for any test without 
checking the real result.

e.g: jSPF supports synchronous, asynchronous and SEDA (Staged 
Event-Driven Architecture) resolvers and testing against a real dns 
server raised much more issues than our previous testsuite.

Having an external testsuite allow you to check bugs in the testsuites 
too. Often people that wrote the code is the same that wrote the tester 
code, this can result in obvious bugs to be left there because of 
misunderstandings from the author.

BTW I don't care if you will or won't use the jSPF tester once we'll 
publish it, but I'm a bit annoyed that OpenSPF page keep saying that the 
only compliant implementations are pyspf 2 and Mail::SPF only because 
OpenSPF members wrote that while jSPF has an "the jSPF library is 
currently being evaluated by the project for RFC 4408 compliance" since 
years. At the end of 2006 jSPF was listed as "The jSPF library is also 
reported to pass all tests in the test suite, however this has not yet 
been officially confirmed" and then on 2006-12-10 you removed the whole 
sentence. On 2007-02-01 then the "currently being evaluated" has been 
written there... 18 months passed since that...

IMHO it is bad for OpenSPF to fail to tell how implementations can be 
listed between compliant implementation and ignore anyone but your own 
implementations.

Stefano