Re: Getting a Tier-X for localized builds. X=1

catlee, I've reread this thread, and my takeaways would be:

- there's stuff to fix
- we probably have a pretty good list based on this thread
- tier-1 is a good milestone

Do you agree?

If so, what's a good way to get to a plan? My idea "fake it 'til we make 
it" didn't work ;-)

Axel

On 14/01/16 01:16, Axel Hecht wrote:
> Hi,
>
> right now, localized builds, and in particular localized Nightlies, are
> part of our job exclusion profile, i.e., they're hidden from any sight.
>
> On the release channel, 60% of our users use those builds, and that's a
> weird combination. Even on aurora, it's 50%.
>
> I think we should make our localized nightlies Tier 1 builds.
>
> There's benefit from doing that today, in terms of tracking
> infrastructure failures, and regressing landings. More on that at the end.
>
> Reading through
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy, I recognize
> some short comings. Some of them are actual challenges in our status
> quo. Some of them might just call for bending our policies, I think.
>
> The first one on the list there is "has an owner".
>
> And this is the first and foremost we just need to bend. (I'm dismissing
> the "active)
>
> L10n builds have several owners, and not accommodating for that might be
> the prime reason why we haven't assigned a tier to l10n builds.
>
> I think we need to deal with the distributed ownership, and find peace
> with that.
>
> The current windows nightly bustage is a great example,
> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update
> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS, but
> (we hope) it's missing '\' in buildbot configs. Finding this out was a
> joint effort of some 5 folks from various groups,
> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466.
>
>
> We should see strength in that shared ownership instead of a blocker.
>
> The next section is titled "Usable job logs". My brain is at odds with
> this one, as I can't decipher the current logs. But I also strongly
> believe that that's tied to how my brain works with written text. We
> need way more stakeholders than me to make sure what this means.
>
> The remaining sections are about what to do when things break.
>
> I do think that we need to back out stuff if it breaks localized builds
> in general.
>
> I also think that our history shows that that only happens really
> rarely, so I'm not convinced that we win by making things like inbound
> or try a prerequisite. There's a bunch of constructive things we can do
> right now with shared ownership that will make progress.
>
> That said, I do think that getting try and inbounds to expose l10n
> failures (beyond what l10n-check does) is good.
>
> That's why I think we should make our l10n nightlies tier 1 now.
>
> I really expect to get some help from sheriffs immediately.
> I also think that we're not getting our requirements figured out and
> assigned unless we actually commit to a particular tier.
>
> To show some common failures these days and their respective actions:
>
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
> shows some infra failures and some broken builds for desktop. Some of
> those got retriggered, and for hg time-out reasons. I think this is
> something that'd be OK to handle for Sheriffs, in particular if it'd be
> part of the standard display.
> It also shows some per-locale bustages for Android, which are somewhat
> easy to detect from the logs. It's easy to see that you don't want to
> retrigger, though I have a hard time to detect what the actual problem
> is. But those can be fixed by the localizers, as
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
> shows. N4 going green on Android is related to Romansh fixing stuff.
> Tough to find out, but the initial error clearly looked like "locale
> busted".
>
> The thing that's sad is that there are some jobs here that should have
> been retriggered, and weren't. And that I wonder if there's an infra
> problem that should've been flagged when it started, and not when I
> happened to look at it. hg timeouts worry me, tbh.
>
> And then there are rare occasions where the builds just break.
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
> is the most recent example, which is showing a "windows nightlies are
> broken". The logs show that configure fails.
>
> Those are hard to resolve, but they're also easy to distinguish from
> infra or locale problems, and for those, filing them as early as we can
> is key. And getting them attention by a shared ownership will certainly
> help. Sheriffs can help here initially with other bugs they've seen. And
> then loop in various stakeholders, or trigger IRC/mailing list
> conversations.
>
> We've been doing l10n builds without tier for some 10 years, and with
> our focus on quality, getting them a tier can only help. I strongly
> favor getting them a tier and then fix the problems.
>
> Axel