Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

On 27/09/16 19:57, L. David Baron wrote:

> I don't think it's acceptable to convert any existing reftest
> coverage in wptrunner's current state, or to ask our engineers to
> write tests for new features targeting wptrunner rather than the
> existing reftest harness.  The existing reftest harness has had a
> significant amount of investment in the ability for engineers to
> debug local failures and get useful information out of failures in
> continuous integration.  Since we haven't really had a lot of that
> experience with wptrunner, and the experience we have had has been
> ignored rather than issues being fixed (see, say,
> https://bugzil.la/1265575 and https://bugzil.la/1265582 , which are
> completely unacceptable), and as a result of such basic things being
> unfixed and the general lack of use, I don't think you have even
> remotely close to the full list of issues that need to be fixed in
> order to bring wptrunner up to parity.
>
> I think you'd be better off trying to solve the autogeneration and
> manifest issues in the existing harness.

I think this argument is backwards. Today we don't run a recent copy of 
the CSS tests because there are technical issues preventing it from 
working with the reftest harness. It is not clear to me why this 
reorganisation would suddenly cause us to fix those issues when we could 
have done at any previous time with similar effort. On the other hand 
the change will allow us to run a copy of the CSS tests in wptrunner 
immediately for "free". Running more tests in wptrunner inevitably 
increases the chance of fixing issues with that harness both because it 
will be easier to prioritise the work and more people will be motivated 
to fix the issues.

Apologies for missing the bugs you pointed to at the time; I believe one 
of them has a solution and I will need to investigate the second issue 
(with debugger-related bugs I find it very helpful to get a set of steps 
to reproduce the problem).