Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

"L. David Baron" <[email protected]> Tue, 27 Sep 2016 11:57:04 -0700
Newsgroups	gmane.comp.mozilla.devel.layout
Message-ID	<[email protected]>
On Tuesday 2016-09-27 19:36 +0100, James Graham wrote:
> On 27/09/16 19:20, L. David Baron wrote:
> 
> >I'd like to see us run the reftests with our existing reftest
> >harness, which is optimized to do image comparison in compiled code,
> >and which also has a bunch of useful failure-annotation features [1]
> >and features that test for other sorts of failures such as
> >NS_ASSERTIONs.
> 
> I think there are a number of ways that this is non-trivial; for example I'm
> not aware that we have the infrastructure needed to generate reftest
> manifests from the source tree, I don't think the existing reftest harness
> can handle the possibility of a test causing a crash cleanly,

I don't think this is important.  Crashes are rare enough that, in
the rare case that we do have a non-intermittent crash, we can
simply skip the test until it's fixed.

> I don't think
> we have the ability to autogenerate expectation data to account for the fact
> that many tests will fail, and so on. I would rather start with wptrunner,
> which already solves these problems, and improve the harness with better
> performance and error reporting since these can be added incrementally
> whereas the other things are prerequisites for any frequently updated
> testsuite from upstream.

I don't think it's acceptable to convert any existing reftest
coverage in wptrunner's current state, or to ask our engineers to
write tests for new features targeting wptrunner rather than the
existing reftest harness.  The existing reftest harness has had a
significant amount of investment in the ability for engineers to
debug local failures and get useful information out of failures in
continuous integration.  Since we haven't really had a lot of that
experience with wptrunner, and the experience we have had has been
ignored rather than issues being fixed (see, say,
https://bugzil.la/1265575 and https://bugzil.la/1265582 , which are
completely unacceptable), and as a result of such basic things being
unfixed and the general lack of use, I don't think you have even
remotely close to the full list of issues that need to be fixed in
order to bring wptrunner up to parity.

I think you'd be better off trying to solve the autogeneration and
manifest issues in the existing harness.

-David

-- 
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

_______________________________________________
dev-tech-layout mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-tech-layout
signature.asc (application/pgp-signature, 819 B)