Re: Getting a Tier-X for localized builds. X=1

On Wed, Jan 13, 2016 at 4:16 PM, Axel Hecht <[email protected]> wrote:

> Hi,
>
> right now, localized builds, and in particular localized Nightlies, are
> part of our job exclusion profile, i.e., they're hidden from any sight.
>
> On the release channel, 60% of our users use those builds, and that's a
> weird combination. Even on aurora, it's 50%.
>
> I think we should make our localized nightlies Tier 1 builds.
>
> There's benefit from doing that today, in terms of tracking infrastructure
> failures, and regressing landings. More on that at the end.
>
> Reading through https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy,
> I recognize some short comings. Some of them are actual challenges in our
> status quo. Some of them might just call for bending our policies, I think.
>
> The first one on the list there is "has an owner".
>
> And this is the first and foremost we just need to bend. (I'm dismissing
> the "active)
>
> L10n builds have several owners, and not accommodating for that might be
> the prime reason why we haven't assigned a tier to l10n builds.
>
> I think we need to deal with the distributed ownership, and find peace
> with that.
>
> The current windows nightly bustage is a great example,
> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update
> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS, but (we
> hope) it's missing '\' in buildbot configs. Finding this out was a joint
> effort of some 5 folks from various groups,
> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466
> .
>
> We should see strength in that shared ownership instead of a blocker.
>
> The next section is titled "Usable job logs". My brain is at odds with
> this one, as I can't decipher the current logs. But I also strongly believe
> that that's tied to how my brain works with written text. We need way more
> stakeholders than me to make sure what this means.
>
> The remaining sections are about what to do when things break.
>
> I do think that we need to back out stuff if it breaks localized builds in
> general.
>
> I also think that our history shows that that only happens really rarely,
> so I'm not convinced that we win by making things like inbound or try a
> prerequisite. There's a bunch of constructive things we can do right now
> with shared ownership that will make progress.
>
> That said, I do think that getting try and inbounds to expose l10n
> failures (beyond what l10n-check does) is good.
>
> That's why I think we should make our l10n nightlies tier 1 now.
>
> I really expect to get some help from sheriffs immediately.
> I also think that we're not getting our requirements figured out and
> assigned unless we actually commit to a particular tier.
>
> To show some common failures these days and their respective actions:
>
>
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
> shows some infra failures and some broken builds for desktop. Some of those
> got retriggered, and for hg time-out reasons. I think this is something
> that'd be OK to handle for Sheriffs, in particular if it'd be part of the
> standard display.
> It also shows some per-locale bustages for Android, which are somewhat
> easy to detect from the logs. It's easy to see that you don't want to
> retrigger, though I have a hard time to detect what the actual problem is.
> But those can be fixed by the localizers, as
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
> shows. N4 going green on Android is related to Romansh fixing stuff. Tough
> to find out, but the initial error clearly looked like "locale busted".
>
> The thing that's sad is that there are some jobs here that should have
> been retriggered, and weren't. And that I wonder if there's an infra
> problem that should've been flagged when it started, and not when I
> happened to look at it. hg timeouts worry me, tbh.
>
> And then there are rare occasions where the builds just break.
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
> is the most recent example, which is showing a "windows nightlies are
> broken". The logs show that configure fails.
>
> Those are hard to resolve, but they're also easy to distinguish from infra
> or locale problems, and for those, filing them as early as we can is key.
> And getting them attention by a shared ownership will certainly help.
> Sheriffs can help here initially with other bugs they've seen. And then
> loop in various stakeholders, or trigger IRC/mailing list conversations.
>
> We've been doing l10n builds without tier for some 10 years, and with our
> focus on quality, getting them a tier can only help. I strongly favor
> getting them a tier and then fix the problems.
>

We absolutely need visibility of l10n automation in TreeHerder.

Bustage in l10n automation should be subject to the same backout policy as
everything else. I suppose this means making it a Tier 1 supported job (or
whatever terminology we need to use).

A major obstacle to moving forward is that many l10n jobs only run
periodically. e.g. l10n Nightlies. What we really need to start doing is
running these jobs (or at least a representative subset of them) on every
build (or at least intelligently scheduled). We produce an l10n nightly, we
just don't publish it until the actual Nightly build. This gives us the
automation coverage and confidence that l10n automation is working
properly. We should also extend this same strategy to other
release-oriented jobs, such as partner repacks. We can't be waiting until
the next uplift or even the next Nightly to discover a regression in l10n
or packaging. Regressions need to be detected soon after the commit that
introduced them.