Getting a Tier-X for localized builds. X=1

Hi,

right now, localized builds, and in particular localized Nightlies, are 
part of our job exclusion profile, i.e., they're hidden from any sight.

On the release channel, 60% of our users use those builds, and that's a 
weird combination. Even on aurora, it's 50%.

I think we should make our localized nightlies Tier 1 builds.

There's benefit from doing that today, in terms of tracking 
infrastructure failures, and regressing landings. More on that at the end.

Reading through 
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy, I recognize 
some short comings. Some of them are actual challenges in our status 
quo. Some of them might just call for bending our policies, I think.

The first one on the list there is "has an owner".

And this is the first and foremost we just need to bend. (I'm dismissing 
the "active)

L10n builds have several owners, and not accommodating for that might be 
the prime reason why we haven't assigned a tier to l10n builds.

I think we need to deal with the distributed ownership, and find peace 
with that.

The current windows nightly bustage is a great example, 
https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update 
to NSIS, and Nightlies broke. The fix has nothing to do with NSIS, but 
(we hope) it's missing '\' in buildbot configs. Finding this out was a 
joint effort of some 5 folks from various groups, 
http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466.

We should see strength in that shared ownership instead of a blocker.

The next section is titled "Usable job logs". My brain is at odds with 
this one, as I can't decipher the current logs. But I also strongly 
believe that that's tied to how my brain works with written text. We 
need way more stakeholders than me to make sure what this means.

The remaining sections are about what to do when things break.

I do think that we need to back out stuff if it breaks localized builds 
in general.

I also think that our history shows that that only happens really 
rarely, so I'm not convinced that we win by making things like inbound 
or try a prerequisite. There's a bunch of constructive things we can do 
right now with shared ownership that will make progress.

That said, I do think that getting try and inbounds to expose l10n 
failures (beyond what l10n-check does) is good.

That's why I think we should make our l10n nightlies tier 1 now.

I really expect to get some help from sheriffs immediately.
I also think that we're not getting our requirements figured out and 
assigned unless we actually commit to a particular tier.

To show some common failures these days and their respective actions:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false 
shows some infra failures and some broken builds for desktop. Some of 
those got retriggered, and for hg time-out reasons. I think this is 
something that'd be OK to handle for Sheriffs, in particular if it'd be 
part of the standard display.
It also shows some per-locale bustages for Android, which are somewhat 
easy to detect from the logs. It's easy to see that you don't want to 
retrigger, though I have a hard time to detect what the actual problem 
is. But those can be fixed by the localizers, as 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false 
shows. N4 going green on Android is related to Romansh fixing stuff. 
Tough to find out, but the initial error clearly looked like "locale 
busted".

The thing that's sad is that there are some jobs here that should have 
been retriggered, and weren't. And that I wonder if there's an infra 
problem that should've been flagged when it started, and not when I 
happened to look at it. hg timeouts worry me, tbh.

And then there are rare occasions where the builds just break. 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n 
is the most recent example, which is showing a "windows nightlies are 
broken". The logs show that configure fails.

Those are hard to resolve, but they're also easy to distinguish from 
infra or locale problems, and for those, filing them as early as we can 
is key. And getting them attention by a shared ownership will certainly 
help. Sheriffs can help here initially with other bugs they've seen. And 
then loop in various stakeholders, or trigger IRC/mailing list 
conversations.

We've been doing l10n builds without tier for some 10 years, and with 
our focus on quality, getting them a tier can only help. I strongly 
favor getting them a tier and then fix the problems.

Axel