Re: Getting a Tier-X for localized builds. X=1

On 14/01/16 19:34, Chris AtLee wrote:
> I think having a separate l10n repack per push for a small set of
> relatively stable locales would go a long way towards detecting bustage
> early.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=848284 was filed a while back
> to get l10n jobs available on Try and Inbound. I've taken a quick poke at
> this, but haven't had a chance to test it. If we want per-push repacks on
> inbound, we may want to take a different approach than the one I've taken
> anyway.
>
> Is there someone who could help debug any issues that come up if we decide
> to tackle this?

I believe that we should tackle this. And I'm happy to help.

I've also proven that I can be only of limited help, given how often 
I've broken the tree for two days past week. Finally even backing out my 
code. More on that in a separate reply.

Broadly speaking:

I think there would be two things to tackle first, one is to break out 
from repacking Nightlies. It sounded like a good idea at the time, but 
not any more. The other is to have repacks for just a select locale or 
two, and not all locales.

The good news is that we're having artifact builds now, which is really 
what repacks are, or can be. Have someone else do the compile, unwrap 
it, do your thing, be happy. I'm not so sure on the choice if routes 
(with fx-team first), mac being mac64..., requiring pushlog data. But 
that should be stuff we can refactor.

Given that I feel like hyatt on a t-shirt today (and I'm not sure if 
that was actually hyatt or someone else), I'm wondering if it'd make 
sense to start an alternative script on try/inbound, and then slowly 
deploy that to nightly/beta/release?

In particular when we get to PRETTY_PACKAGE_NAMES, the current code in 
artifacts.py probably breaks?

Axel

>
> Cheers,
> Chris
>
> On 14 January 2016 at 13:23, Steve Fink <[email protected]> wrote:
>
>> On 01/13/2016 07:09 PM, Mike Hommey wrote:
>>
>>> On Wed, Jan 13, 2016 at 05:00:36PM -0800, Gregory Szorc wrote:
>>>
>>>> We absolutely need visibility of l10n automation in TreeHerder.
>>>>
>>>> Bustage in l10n automation should be subject to the same backout policy
>>>> as
>>>> everything else. I suppose this means making it a Tier 1 supported job
>>>> (or
>>>> whatever terminology we need to use).
>>>>
>>>> A major obstacle to moving forward is that many l10n jobs only run
>>>> periodically. e.g. l10n Nightlies. What we really need to start doing is
>>>> running these jobs (or at least a representative subset of them) on every
>>>> build (or at least intelligently scheduled). We produce an l10n nightly,
>>>> we
>>>> just don't publish it until the actual Nightly build. This gives us the
>>>> automation coverage and confidence that l10n automation is working
>>>> properly. We should also extend this same strategy to other
>>>> release-oriented jobs, such as partner repacks. We can't be waiting until
>>>> the next uplift or even the next Nightly to discover a regression in l10n
>>>> or packaging. Regressions need to be detected soon after the commit that
>>>> introduced them.
>>>>
>>> Worse, we can't wait to discover regressions when things hit beta (that
>>> happens a *lot*)
>>>
>>> Either way, saying that things breaking l10n builds should be backed out
>>> is nice, but the problem is that in many cases, the answer to "what
>>> should be backed out?" is not trivial, and you don't know if the backout
>>> worked until the next nightly...
>>>
>>> This, to me, is what makes l10n builds hard to make tier-1 with the
>>> current state of things.
>>>
>>
>> Yes. I don't see what declaring l10n builds to be "tier 1" would do right
>> now, other than weakening the meaning of tier 1.
>>
>> The main meaning of "tier 1" in my mind is whether things get backed out
>> for breaking them. Which only makes sense if it's possible to detect that a
>> push has in fact broken them. A once per day build really does not allow
>> that, imho; you'd need to train all the sheriffs to be experts in
>> diagnosing l10n breakages. So I would think it would be more productive to
>> figure out what parts of l10n testing *can* be made tier 1.
>>
>> Products don't belong to a tier. Specific *jobs* do.
>>
>> We totally want to get better at this. I've maintained non tier 1 code,
>> and it's amazing how many ways jobs can start breaking if you're not
>> keeping on top of them all the time. (Or rather, if you don't have sheriffs
>> keeping on top of them!) For something we're shipping, it's crazy to be in
>> that state.
>>
>> What testing is feasible to run per-checkin? What useful jobs can we carve
>> out?
>>
>> As RyanVM said, is there more checkin automation that would prevent some
>> classes from reaching the tree?
>>
>> How can we identify what change broke things? (eg, would a bumper bot
>> work?)
>>
>> Are l10n breakages distinguishable from the logs? It's not about the logs
>> being nicely usable by humans -- we're far from that -- but rather that the
>> sheriffs don't have to open up each and every failing job's log and
>> manually read through it to figure out what's going on. For build failures,
>> you're already good -- treeherder will display those failures properly. But
>> if, for example, &some; xml entity breaks things, is the error message in a
>> pattern recognizable by the current log scanners and if not, does the error
>> message need to change or is there a pattern we can check for in the
>> scanner?
>>
>> Going beyond, I have long speculated about having a tier 2 (tier 1.5?)
>> classification for jobs that *don't* run on every push, using some
>> additional mechanisms beyond what we have now to make them visible without
>> being distracting. But I'd rather not divert the focus with that.
>>
>>
>> _______________________________________________
>> dev-planning mailing list
>> [email protected]
>> https://lists.mozilla.org/listinfo/dev-planning
>>