Re: Getting a Tier-X for localized builds. X=1

I think having a separate l10n repack per push for a small set of
relatively stable locales would go a long way towards detecting bustage
early.

https://bugzilla.mozilla.org/show_bug.cgi?id=848284 was filed a while back
to get l10n jobs available on Try and Inbound. I've taken a quick poke at
this, but haven't had a chance to test it. If we want per-push repacks on
inbound, we may want to take a different approach than the one I've taken
anyway.

Is there someone who could help debug any issues that come up if we decide
to tackle this?

Cheers,
Chris

On 14 January 2016 at 13:23, Steve Fink <[email protected]> wrote:

> On 01/13/2016 07:09 PM, Mike Hommey wrote:
>
>> On Wed, Jan 13, 2016 at 05:00:36PM -0800, Gregory Szorc wrote:
>>
>>> We absolutely need visibility of l10n automation in TreeHerder.
>>>
>>> Bustage in l10n automation should be subject to the same backout policy
>>> as
>>> everything else. I suppose this means making it a Tier 1 supported job
>>> (or
>>> whatever terminology we need to use).
>>>
>>> A major obstacle to moving forward is that many l10n jobs only run
>>> periodically. e.g. l10n Nightlies. What we really need to start doing is
>>> running these jobs (or at least a representative subset of them) on every
>>> build (or at least intelligently scheduled). We produce an l10n nightly,
>>> we
>>> just don't publish it until the actual Nightly build. This gives us the
>>> automation coverage and confidence that l10n automation is working
>>> properly. We should also extend this same strategy to other
>>> release-oriented jobs, such as partner repacks. We can't be waiting until
>>> the next uplift or even the next Nightly to discover a regression in l10n
>>> or packaging. Regressions need to be detected soon after the commit that
>>> introduced them.
>>>
>> Worse, we can't wait to discover regressions when things hit beta (that
>> happens a *lot*)
>>
>> Either way, saying that things breaking l10n builds should be backed out
>> is nice, but the problem is that in many cases, the answer to "what
>> should be backed out?" is not trivial, and you don't know if the backout
>> worked until the next nightly...
>>
>> This, to me, is what makes l10n builds hard to make tier-1 with the
>> current state of things.
>>
>
> Yes. I don't see what declaring l10n builds to be "tier 1" would do right
> now, other than weakening the meaning of tier 1.
>
> The main meaning of "tier 1" in my mind is whether things get backed out
> for breaking them. Which only makes sense if it's possible to detect that a
> push has in fact broken them. A once per day build really does not allow
> that, imho; you'd need to train all the sheriffs to be experts in
> diagnosing l10n breakages. So I would think it would be more productive to
> figure out what parts of l10n testing *can* be made tier 1.
>
> Products don't belong to a tier. Specific *jobs* do.
>
> We totally want to get better at this. I've maintained non tier 1 code,
> and it's amazing how many ways jobs can start breaking if you're not
> keeping on top of them all the time. (Or rather, if you don't have sheriffs
> keeping on top of them!) For something we're shipping, it's crazy to be in
> that state.
>
> What testing is feasible to run per-checkin? What useful jobs can we carve
> out?
>
> As RyanVM said, is there more checkin automation that would prevent some
> classes from reaching the tree?
>
> How can we identify what change broke things? (eg, would a bumper bot
> work?)
>
> Are l10n breakages distinguishable from the logs? It's not about the logs
> being nicely usable by humans -- we're far from that -- but rather that the
> sheriffs don't have to open up each and every failing job's log and
> manually read through it to figure out what's going on. For build failures,
> you're already good -- treeherder will display those failures properly. But
> if, for example, &some; xml entity breaks things, is the error message in a
> pattern recognizable by the current log scanners and if not, does the error
> message need to change or is there a pattern we can check for in the
> scanner?
>
> Going beyond, I have long speculated about having a tier 2 (tier 1.5?)
> classification for jobs that *don't* run on every push, using some
> additional mechanisms beyond what we have now to make them visible without
> being distracting. But I'd rather not divert the focus with that.
>
>
> _______________________________________________
> dev-planning mailing list
> [email protected]
> https://lists.mozilla.org/listinfo/dev-planning
>