Re: Getting a Tier-X for localized builds. X=1

The other big issue I see is that l10n builds can break at any time due 
to changes from the multitude of external repos where the strings are 
actually stored (i.e. someone lands a busted string in some locale's 
repo, next day's l10n nightlies break as a result). This was an issue we 
pushed back hard against in the early B2G days because it makes 
sheriffing a huge pain. The solution for B2G was in-tree manifests 
automatically updated via the B2G Bumper Bot. Not sure what we could do 
for l10n builds.

-Ryan

On 1/13/2016 7:41 PM, Axel Hecht wrote:
> On 14/01/16 01:22, Kyle Huey wrote:
>> IMO in the most daunting obstacle is one that you do not mention at all:
>> l10n builds are not scheduled on every push.
>
> I actually don't think this is an obstacle. Or should be.
>
> I also don't agree that we're not doing l10n builds per push. We do,
> they're the l10n-check target in browser/locales, and they're run on try
> and other automation,
> https://dxr.mozilla.org/mozilla-central/source/build/moz-automation.mk#37.
>
> They're catching some errors, but not others. For example, the try runs
> on https://bugzilla.mozilla.org/show_bug.cgi?id=1215694 failed for much
> of its development cycle. And once they were green, the only thing that
> was left busted were manifests in language packs.
>
> In contrast, we have dozens of locales/platform combinations failing
> each day for mercurial timeouts. Like, 3 failed chunks is easily that.
> All that's required to get those builds out to people is to retrigger in
> the treeherder UI.
>
> So we can benefit today.
>
> Having a meaningful test suite for localized builds would be great. But
> blocking on that hasn't proven to be constructive for the past ten year.
> https://groups.google.com/d/msg/mozilla.dev.automation/RbgbaShQb_Y/u7Tsnhj5FgAJ
> has thoughts on that, too.
>
> Axel
>
>>
>> - Kyle
>>
>> On Wed, Jan 13, 2016 at 4:16 PM, Axel Hecht <[email protected]> wrote:
>>
>>> Hi,
>>>
>>> right now, localized builds, and in particular localized Nightlies, are
>>> part of our job exclusion profile, i.e., they're hidden from any sight.
>>>
>>> On the release channel, 60% of our users use those builds, and that's a
>>> weird combination. Even on aurora, it's 50%.
>>>
>>> I think we should make our localized nightlies Tier 1 builds.
>>>
>>> There's benefit from doing that today, in terms of tracking
>>> infrastructure
>>> failures, and regressing landings. More on that at the end.
>>>
>>> Reading through
>>> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy,
>>> I recognize some short comings. Some of them are actual challenges in
>>> our
>>> status quo. Some of them might just call for bending our policies, I
>>> think.
>>>
>>> The first one on the list there is "has an owner".
>>>
>>> And this is the first and foremost we just need to bend. (I'm dismissing
>>> the "active)
>>>
>>> L10n builds have several owners, and not accommodating for that might be
>>> the prime reason why we haven't assigned a tier to l10n builds.
>>>
>>> I think we need to deal with the distributed ownership, and find peace
>>> with that.
>>>
>>> The current windows nightly bustage is a great example,
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update
>>> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS,
>>> but (we
>>> hope) it's missing '\' in buildbot configs. Finding this out was a joint
>>> effort of some 5 folks from various groups,
>>> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466
>>>
>>> .
>>>
>>> We should see strength in that shared ownership instead of a blocker.
>>>
>>> The next section is titled "Usable job logs". My brain is at odds with
>>> this one, as I can't decipher the current logs. But I also strongly
>>> believe
>>> that that's tied to how my brain works with written text. We need way
>>> more
>>> stakeholders than me to make sure what this means.
>>>
>>> The remaining sections are about what to do when things break.
>>>
>>> I do think that we need to back out stuff if it breaks localized
>>> builds in
>>> general.
>>>
>>> I also think that our history shows that that only happens really
>>> rarely,
>>> so I'm not convinced that we win by making things like inbound or try a
>>> prerequisite. There's a bunch of constructive things we can do right now
>>> with shared ownership that will make progress.
>>>
>>> That said, I do think that getting try and inbounds to expose l10n
>>> failures (beyond what l10n-check does) is good.
>>>
>>> That's why I think we should make our l10n nightlies tier 1 now.
>>>
>>> I really expect to get some help from sheriffs immediately.
>>> I also think that we're not getting our requirements figured out and
>>> assigned unless we actually commit to a particular tier.
>>>
>>> To show some common failures these days and their respective actions:
>>>
>>>
>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
>>>
>>> shows some infra failures and some broken builds for desktop. Some of
>>> those
>>> got retriggered, and for hg time-out reasons. I think this is something
>>> that'd be OK to handle for Sheriffs, in particular if it'd be part of
>>> the
>>> standard display.
>>> It also shows some per-locale bustages for Android, which are somewhat
>>> easy to detect from the logs. It's easy to see that you don't want to
>>> retrigger, though I have a hard time to detect what the actual
>>> problem is.
>>> But those can be fixed by the localizers, as
>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
>>>
>>> shows. N4 going green on Android is related to Romansh fixing stuff.
>>> Tough
>>> to find out, but the initial error clearly looked like "locale busted".
>>>
>>> The thing that's sad is that there are some jobs here that should have
>>> been retriggered, and weren't. And that I wonder if there's an infra
>>> problem that should've been flagged when it started, and not when I
>>> happened to look at it. hg timeouts worry me, tbh.
>>>
>>> And then there are rare occasions where the builds just break.
>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
>>>
>>> is the most recent example, which is showing a "windows nightlies are
>>> broken". The logs show that configure fails.
>>>
>>> Those are hard to resolve, but they're also easy to distinguish from
>>> infra
>>> or locale problems, and for those, filing them as early as we can is
>>> key.
>>> And getting them attention by a shared ownership will certainly help.
>>> Sheriffs can help here initially with other bugs they've seen. And then
>>> loop in various stakeholders, or trigger IRC/mailing list conversations.
>>>
>>> We've been doing l10n builds without tier for some 10 years, and with
>>> our
>>> focus on quality, getting them a tier can only help. I strongly favor
>>> getting them a tier and then fix the problems.
>>>
>>> Axel
>>> _______________________________________________
>>> dev-planning mailing list
>>> [email protected]
>>> https://lists.mozilla.org/listinfo/dev-planning
>>>
>