Re: Implementation certification procedure

Stuart D. Gathman ha scritto:
> On Tue, 19 Aug 2008, Stefano Bagnara wrote:
> 
>> Having an external testsuite allow you to check bugs in the testsuites too.
>> Often people that wrote the code is the same that wrote the tester code, this
>> can result in obvious bugs to be left there because of misunderstandings from
>> the author.
> 
> I have already put running your tester on my TODO list (although I don't
> know when I'll get to it).  Effective testing is an area of software
> that is often ignored because of its difficulty once you get beyond
> the simple input/process/output library routine.
> 
>> BTW I don't care if you will or won't use the jSPF tester once we'll publish
>> it, but I'm a bit annoyed that OpenSPF page keep saying that the only
>> compliant implementations are pyspf 2 and Mail::SPF only because OpenSPF
>> members wrote that while jSPF has an "the jSPF library is currently being
>> evaluated by the project for RFC 4408 compliance" since years. At the end of
>> 2006 jSPF was listed as "The jSPF library is also reported to pass all tests
>> in the test suite, however this has not yet been officially confirmed" and
>> then on 2006-12-10 you removed the whole sentence. On 2007-02-01 then the
>> "currently being evaluated" has been written there... 18 months passed since
>> that...
> 
>> IMHO it is bad for OpenSPF to fail to tell how implementations can be listed
>> between compliant implementation and ignore anyone but your own
>> implementations.
> 
> Agreed.  But I'm pretty sure it is due to not having come up with an
> actual procedure rather than self promotion.  Maybe you could volunteer
> to test implementations using your live DNS test framework?  That would
> at least get yours done :-)

I'm already testing jSPF with my tester :-)

My effort to make the tester "implementation agnostic" is really to gain 
some more trust from you when you see that my tester correctly check 
your implementations.

It is OpenSPF that have to tell how an implementation is certified. What 
tests have Mail::SPF and pyspf 2.0 passed in order to be listed there?? 
I bet jSPF passed the same tests.

In fact we always had selftests based on the yaml file you publish and 
we have some more unit test in place to increase the coverage, too.

I'd happily complete the live dns tester tool but in order to check 
implementations they have to return an "spfquery" like result, 4 lines 
where the 1st is the result, the 2nd is the explanation, the 4th is the 
Received-SPF: header. Then they also have the ability to use a specific 
dns server for their queries (e.g: --dnsserver|-s <IP>[:PORT]).
I can make parameters configurable, but that dnsserver option is needed 
in order to run my tester. AFAIK no implementation currently support 
this (jSPF will support this in the next release).

Using the "commandline interface" is the only way I found to test 
different implementations using a single tester.

Stefano