Re: Getting a Tier-X for localized builds. X=1

On 01/13/2016 07:09 PM, Mike Hommey wrote:
> On Wed, Jan 13, 2016 at 05:00:36PM -0800, Gregory Szorc wrote:
>> We absolutely need visibility of l10n automation in TreeHerder.
>>
>> Bustage in l10n automation should be subject to the same backout policy as
>> everything else. I suppose this means making it a Tier 1 supported job (or
>> whatever terminology we need to use).
>>
>> A major obstacle to moving forward is that many l10n jobs only run
>> periodically. e.g. l10n Nightlies. What we really need to start doing is
>> running these jobs (or at least a representative subset of them) on every
>> build (or at least intelligently scheduled). We produce an l10n nightly, we
>> just don't publish it until the actual Nightly build. This gives us the
>> automation coverage and confidence that l10n automation is working
>> properly. We should also extend this same strategy to other
>> release-oriented jobs, such as partner repacks. We can't be waiting until
>> the next uplift or even the next Nightly to discover a regression in l10n
>> or packaging. Regressions need to be detected soon after the commit that
>> introduced them.
> Worse, we can't wait to discover regressions when things hit beta (that
> happens a *lot*)
>
> Either way, saying that things breaking l10n builds should be backed out
> is nice, but the problem is that in many cases, the answer to "what
> should be backed out?" is not trivial, and you don't know if the backout
> worked until the next nightly...
>
> This, to me, is what makes l10n builds hard to make tier-1 with the
> current state of things.

Yes. I don't see what declaring l10n builds to be "tier 1" would do 
right now, other than weakening the meaning of tier 1.

The main meaning of "tier 1" in my mind is whether things get backed out 
for breaking them. Which only makes sense if it's possible to detect 
that a push has in fact broken them. A once per day build really does 
not allow that, imho; you'd need to train all the sheriffs to be experts 
in diagnosing l10n breakages. So I would think it would be more 
productive to figure out what parts of l10n testing *can* be made tier 1.

Products don't belong to a tier. Specific *jobs* do.

We totally want to get better at this. I've maintained non tier 1 code, 
and it's amazing how many ways jobs can start breaking if you're not 
keeping on top of them all the time. (Or rather, if you don't have 
sheriffs keeping on top of them!) For something we're shipping, it's 
crazy to be in that state.

What testing is feasible to run per-checkin? What useful jobs can we 
carve out?

As RyanVM said, is there more checkin automation that would prevent some 
classes from reaching the tree?

How can we identify what change broke things? (eg, would a bumper bot work?)

Are l10n breakages distinguishable from the logs? It's not about the 
logs being nicely usable by humans -- we're far from that -- but rather 
that the sheriffs don't have to open up each and every failing job's log 
and manually read through it to figure out what's going on. For build 
failures, you're already good -- treeherder will display those failures 
properly. But if, for example, &some; xml entity breaks things, is the 
error message in a pattern recognizable by the current log scanners and 
if not, does the error message need to change or is there a pattern we 
can check for in the scanner?

Going beyond, I have long speculated about having a tier 2 (tier 1.5?) 
classification for jobs that *don't* run on every push, using some 
additional mechanisms beyond what we have now to make them visible 
without being distracting. But I'd rather not divert the focus with that.