Re: Getting a Tier-X for localized builds. X=1

IMO in the most daunting obstacle is one that you do not mention at all:
l10n builds are not scheduled on every push.

- Kyle

On Wed, Jan 13, 2016 at 4:16 PM, Axel Hecht <[email protected]> wrote:

> Hi,
>
> right now, localized builds, and in particular localized Nightlies, are
> part of our job exclusion profile, i.e., they're hidden from any sight.
>
> On the release channel, 60% of our users use those builds, and that's a
> weird combination. Even on aurora, it's 50%.
>
> I think we should make our localized nightlies Tier 1 builds.
>
> There's benefit from doing that today, in terms of tracking infrastructure
> failures, and regressing landings. More on that at the end.
>
> Reading through https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy,
> I recognize some short comings. Some of them are actual challenges in our
> status quo. Some of them might just call for bending our policies, I think.
>
> The first one on the list there is "has an owner".
>
> And this is the first and foremost we just need to bend. (I'm dismissing
> the "active)
>
> L10n builds have several owners, and not accommodating for that might be
> the prime reason why we haven't assigned a tier to l10n builds.
>
> I think we need to deal with the distributed ownership, and find peace
> with that.
>
> The current windows nightly bustage is a great example,
> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update
> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS, but (we
> hope) it's missing '\' in buildbot configs. Finding this out was a joint
> effort of some 5 folks from various groups,
> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466
> .
>
> We should see strength in that shared ownership instead of a blocker.
>
> The next section is titled "Usable job logs". My brain is at odds with
> this one, as I can't decipher the current logs. But I also strongly believe
> that that's tied to how my brain works with written text. We need way more
> stakeholders than me to make sure what this means.
>
> The remaining sections are about what to do when things break.
>
> I do think that we need to back out stuff if it breaks localized builds in
> general.
>
> I also think that our history shows that that only happens really rarely,
> so I'm not convinced that we win by making things like inbound or try a
> prerequisite. There's a bunch of constructive things we can do right now
> with shared ownership that will make progress.
>
> That said, I do think that getting try and inbounds to expose l10n
> failures (beyond what l10n-check does) is good.
>
> That's why I think we should make our l10n nightlies tier 1 now.
>
> I really expect to get some help from sheriffs immediately.
> I also think that we're not getting our requirements figured out and
> assigned unless we actually commit to a particular tier.
>
> To show some common failures these days and their respective actions:
>
>
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
> shows some infra failures and some broken builds for desktop. Some of those
> got retriggered, and for hg time-out reasons. I think this is something
> that'd be OK to handle for Sheriffs, in particular if it'd be part of the
> standard display.
> It also shows some per-locale bustages for Android, which are somewhat
> easy to detect from the logs. It's easy to see that you don't want to
> retrigger, though I have a hard time to detect what the actual problem is.
> But those can be fixed by the localizers, as
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
> shows. N4 going green on Android is related to Romansh fixing stuff. Tough
> to find out, but the initial error clearly looked like "locale busted".
>
> The thing that's sad is that there are some jobs here that should have
> been retriggered, and weren't. And that I wonder if there's an infra
> problem that should've been flagged when it started, and not when I
> happened to look at it. hg timeouts worry me, tbh.
>
> And then there are rare occasions where the builds just break.
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
> is the most recent example, which is showing a "windows nightlies are
> broken". The logs show that configure fails.
>
> Those are hard to resolve, but they're also easy to distinguish from infra
> or locale problems, and for those, filing them as early as we can is key.
> And getting them attention by a shared ownership will certainly help.
> Sheriffs can help here initially with other bugs they've seen. And then
> loop in various stakeholders, or trigger IRC/mailing list conversations.
>
> We've been doing l10n builds without tier for some 10 years, and with our
> focus on quality, getting them a tier can only help. I strongly favor
> getting them a tier and then fix the problems.
>
> Axel
> _______________________________________________
> dev-planning mailing list
> [email protected]
> https://lists.mozilla.org/listinfo/dev-planning
>