Re: Possible patch to Test::Pod

* Russ Allbery <[email protected]> [2012-06-06T01:10:01]
> Grant McLean <[email protected]> writes:
> 
> > While my patch adds a warning, in combination with Test::Pod it is
> > effectively elevated to a fatal error which blocks a clean installation
> > of affected distributions.
> 
> > The "correct" answer is for people who use Test::Pod to only run those
> > tests on the author's system - i.e. pre release rather than pre install.
> > (And ideally add the missing -=encoding too). Of course it might be a
> > bit inconvenient for some maintainers to rush out a new release for that
> > reason alone.
> 
> I don't think I agree with this, and I'm not sure why that would be a
> recommendation.  Surely, the tests should fail if there are bugs in the
> documentation?  I don't see what's different about this than any other bug
> in edge functionality or minor APIs, which may not affect any particular
> user but which are routinely diagnosed as bugs by the test suite.

If the author is shipping his Pod tests to run for all users, and to prevent
installation when the Pod is detected as might-be-broken, and if the tools for
detecting might-be-broken Pod get better, this is exactly what should be
expected.

Almost certainly, as Grant said, the authors should have limited the Pod tests
to their release process.

The biggest question for me is: if we add this exception now, will we also add
exceptions for every future warning?  Are only the current warnings as of
2012-05-01 the ones that should be allowed to make Test::Pod fail?

Otherwise, what's the transition plan from passing on this warning to not
passing on this warning?  We can't say "only fail if it's the author running
the tests" because we don't know.  In fact, the author should be telling us
this by making the tests only run for him or her, which gets us back to that
problem.

So, maybe we'll say "well, this warning is not fatal for a year, by which time
you should really have gotten around to this."  Except only x% of authors will,
and then we're back to a bunch of breakage, and this argument again about
whether we extend the grace period or allow the test failures to break those
modules that were otherwise so stable that they needed no fixes in the last
year.

DBI is already fixed.  Dancer releases frequently already, so the notion that
we're forcing them to "rush out a new release" doesn't seem too troublesome.
Do we have a list of affected dists?

> Why would an end-user have trouble with failing tests at install time?  If
> they don't care about the failure, just ignore it and install the module
> anyway.
> 
> Is there some tool involved that's refusing to install modules if the test
> suite fails?  If so, I think that tool is what's buggy here, and that tool
> should be fixed so that it's possible to install modules with failing
> tests, since that's very common, for many reasons other than this.

Basically every tool that does CPAN installs will refuse to install if the test
suite fails, and this isn't going to be viewed as buggy or changed.  End users
don't always know whether the test failure is significant or not.  This is
another reason for authors to disable insignificant tests in their shipped
distribution.

-- 
rjbs

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBAgAGBQJP0K21AAoJEOYby6cMccU55cEH/irYCA5liCz72VYHibXunK3d
s0eDC1WN7UaGUotT1je5lHjW9iHiDbWs9kAP4x85phMEkLFwkNsYnsIq6b90lZbS
6xRH/3IDIWEX7M/k2AnGvYTpAFEGXilkunj1yC2qaSL0cD6pRnp72vxEXWNLAL9Q
HS9StfXUqlyaPbyfy36MiYTrypXVbnp2uObf229GBltiWboFUd9vP7rxXolX0Si1
rcLDt5awOP5RXUCyvAUAXO7ICiTixSg/J8pBGSVm7GCoywiTWRKl+/eUxlnIdv2Y
YSaMjr3qzc81Gryi+1nk880gPXXal++4pFttQQAP5bqpYPf3NbgrAM7vcb/lfrA=
=Cmgl
-----END PGP SIGNATURE-----