Re: Fwd: PROPOSED RESOLUTION: merge csswg-test into web-platform-tests

On 27/09/16 20:24, L. David Baron wrote:

>> On the other hand the change will allow
>> us to run a copy of the CSS tests in wptrunner immediately for "free".
>> Running more tests in wptrunner inevitably increases the chance of fixing
>> issues with that harness both because it will be easier to prioritise the
>> work and more people will be motivated to fix the issues.
>
> I think running a large number of reftests in wptrunner will be an
> unacceptable load on our CI infrastructure due to
> https://bugzil.la/1265586 .

That, at least, seems like a testable hypothesis once the merge is made. 
I don't think the performance concerns there are insurmountable even in 
the wpt architecture; as I said in the bug we can add better primitives 
to marionette to do this work.

>> Apologies for missing the bugs you pointed to at the time; I believe one of
>> them has a solution and I will need to investigate the second issue (with
>> debugger-related bugs I find it very helpful to get a set of steps to
>> reproduce the problem).
>
> I think those bugs are the very tip of an iceberg that involves many
> person-months of work.

Perhaps, but I think there is also significant work in making the 
reftest harness suitable for running these tests in a way that allows 
them to be updated as frequently and painlessly as possible. The 
difference is that I see an incremental path to victory with wptrunner 
(start with running just the highest value tests for features we are 
working on today, optimise the performance, fix ergonomics issues, 
switch on more tests), and the improvements we make along the way will 
not just help with these tests but also with all other 
web-platform-tests in both gecko and servo.