Re: Getting a Tier-X for localized builds. X=1

On Wed, Jan 13, 2016 at 05:00:36PM -0800, Gregory Szorc wrote:
> We absolutely need visibility of l10n automation in TreeHerder.
> 
> Bustage in l10n automation should be subject to the same backout policy as
> everything else. I suppose this means making it a Tier 1 supported job (or
> whatever terminology we need to use).
> 
> A major obstacle to moving forward is that many l10n jobs only run
> periodically. e.g. l10n Nightlies. What we really need to start doing is
> running these jobs (or at least a representative subset of them) on every
> build (or at least intelligently scheduled). We produce an l10n nightly, we
> just don't publish it until the actual Nightly build. This gives us the
> automation coverage and confidence that l10n automation is working
> properly. We should also extend this same strategy to other
> release-oriented jobs, such as partner repacks. We can't be waiting until
> the next uplift or even the next Nightly to discover a regression in l10n
> or packaging. Regressions need to be detected soon after the commit that
> introduced them.

Worse, we can't wait to discover regressions when things hit beta (that
happens a *lot*)

Either way, saying that things breaking l10n builds should be backed out
is nice, but the problem is that in many cases, the answer to "what
should be backed out?" is not trivial, and you don't know if the backout
worked until the next nightly...

This, to me, is what makes l10n builds hard to make tier-1 with the
current state of things.

Mike