Re: revisiting disabling Linux32 testing

On 10/03/2016 21:09, Nicolas B. Pierron wrote:
> On 03/10/2016 03:29 PM, [email protected] wrote:
>> Last summer I posted a note requesting feedback regarding disabling
>> tests on Linux32. To summarize, the feedback was
>>
>> * We use Linux32 tests for sanity checking tests that aren't running
>> on b2g
>> * Linux64 tests don't have parity with the test coverage on Linux32
>> * Linux32 is a very small user base compared to Windows yet we
>> allocate expensive resources to it.
>> * Was mentioned that 25% of our contributor community run Linux,
>> however it is unclear if this is Linux32 or Linux64
>> * Most people don't use our Linux32 builds, but rather the ones from
>> their distro
>> * We do not have reliable telemetry numbers for Linux32 from distros
>> for various reasons
>>
>> https://groups.google.com/forum/#!searchin/mozilla.dev.planning/linux32/mozilla.dev.planning/wBgLRXCTlaw/-YzXkHxyBgAJ
>>
>>
>> I didn't proceed with the change at that time because of the test
>> parity and b2g issues.  However, since that time we have disabled b2g
>> builds and tests and achieved test parity on linux64.  Thus at this
>> time, I'm looking to proceed to disable linux32 tests.
>
> Does this include the JS Shell test suite, as well as the arm simulator?
> From what I understand we have no substitute yet.
>

Could we transition to having taskcluster-based builds and only the test 
suites that have no equivalent (so dump e.g. all of mochitest(-*), but 
not js shell as Nicolas brought up) ? Or would that be an inordinate 
amount of work? It seems like that might lead to faster + cheaper 
builds, plus a good amount of gain from being able to switch off the 
majority of the testing immediately.

I have no particular insight in expenses here, but I do know that the 
linux build results tend to be the fastest to show up as a result of a 
(try) push, and esp. for 32-bit build failures having to wait for the 
windows builds would delay when we notice that there are problems.

~ Gijs