Re: Upcoming new test-suite release

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Julian Mehnle ha scritto:
> > Stefano Bagnara wrote:
> > > The jSPF tester have the ability to load a zone configuration from
> > > the yaml file, set up a "fake" dsn server on a given port, read
> > > tests, run the spf checker and check the results.
> > >
> > > One issue I've found when trying our tester against other
> > > implementations is that it is not easy to tell other implementations
> > > to use a specific dns server/port for the resolution.
> >
> > FWIW, it's all but trivial with Mail::SPF.  One can specify a custom
> > DNS resolver object, e.g., a Net::DNS::Resolver object configured to
> > always use a specific resolving server.
>
> I'm not a perl programmer, so I won't do this. A configuration option
> to override the dns servers would do the trick.

Actually, I'd rather have the online version of the test suite driver 
append a given zone name to every lookup performed, e.g., 
2008_08.rfc4408.test-suite.openspf.org.  That way you wouldn't have to 
specify a custom resolver server.  That zone can then be delegated to 
your fake DNS server.

> > Besides, running an SPF implementations self-tests could be made (by
> > authors) as easy as `make test`.  It even almost works with Mail::SPF
> > -- you just have to run `perl Makefile.PL` beforehand.
>
> The difference between this test and "selftests" is big, IMHO.
> Most "self-tests" I've seen around don't actually run a real UDP/TCP
> dns query, but they mock the whole resolution mechanism at one or
> another layer. Otherwise we could have a "selftest" returning "ok" for
> any test without checking the real result.

I don't see the difference as being all that big, given that the test 
suite drivers are probably rather trivial, but I certainly don't object 
to have SPF implementations tested against live DNS.  If you're willing 
to help with setting this up, or even lead the effort, all the better!

> BTW I don't care if you will or won't use the jSPF tester once we'll
> publish it, but I'm a bit annoyed that OpenSPF page keep saying that
> the only compliant implementations are pyspf 2 and Mail::SPF only
> because OpenSPF members wrote that while jSPF has an "the jSPF library
> is currently being evaluated by the project for RFC 4408 compliance"
> since years. At the end of 2006 jSPF was listed as "The jSPF library is
> also reported to pass all tests in the test suite, however this has not
> yet been officially confirmed" and then on 2006-12-10 you removed the
> whole sentence. On 2007-02-01 then the "currently being evaluated" has
> been written there... 18 months passed since that...
>
> IMHO it is bad for OpenSPF to fail to tell how implementations can be
> listed between compliant implementation and ignore anyone but your own
> implementations.

As Stuart said, this isn't in any way due to a desire for self-promotion.  
The reason that the website still says "The jSPF library is currently 
being evaluated by the project for RFC 4408 compliance" simply is that 
when I tried to run the (offline) test-suite against jSPF over a year 
ago, it was very complicated to get it to work for someone who isn't an 
experienced Java/Maven developer, and that the generated output was 
practically unreadable, and no one else capable of editing the website 
has since tried checking out jSPF's compliance.  I admit that it was my 
fault not to have announced that I had given up on trying to get useful 
output from my jSPF test-suite runs, and hadn't asked that someone else 
give it a try.

So please don't be mad, but let's now try to get this resolved instead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir3rIACgkQwL7PKlBZWjt+hQCgvaUqjODGqBHoBGCj6DbWJIzo
qQUAn1lV0LJR/sbXPowmNdrKSfu1YRG1
=jZrW
-----END PGP SIGNATURE-----