Re: Getting a Tier-X for localized builds. X=1

And I guess there's a related question there as to why can't we 
better-validate string changes before they get checked into the repo (or 
at least immediately after)?

On 1/13/2016 8:00 PM, Ryan VanderMeulen wrote:
> The other big issue I see is that l10n builds can break at any time due
> to changes from the multitude of external repos where the strings are
> actually stored (i.e. someone lands a busted string in some locale's
> repo, next day's l10n nightlies break as a result). This was an issue we
> pushed back hard against in the early B2G days because it makes
> sheriffing a huge pain. The solution for B2G was in-tree manifests
> automatically updated via the B2G Bumper Bot. Not sure what we could do
> for l10n builds.
>
> -Ryan
>
> On 1/13/2016 7:41 PM, Axel Hecht wrote:
>> On 14/01/16 01:22, Kyle Huey wrote:
>>> IMO in the most daunting obstacle is one that you do not mention at all:
>>> l10n builds are not scheduled on every push.
>>
>> I actually don't think this is an obstacle. Or should be.
>>
>> I also don't agree that we're not doing l10n builds per push. We do,
>> they're the l10n-check target in browser/locales, and they're run on try
>> and other automation,
>> https://dxr.mozilla.org/mozilla-central/source/build/moz-automation.mk#37.
>>
>>
>> They're catching some errors, but not others. For example, the try runs
>> on https://bugzilla.mozilla.org/show_bug.cgi?id=1215694 failed for much
>> of its development cycle. And once they were green, the only thing that
>> was left busted were manifests in language packs.
>>
>> In contrast, we have dozens of locales/platform combinations failing
>> each day for mercurial timeouts. Like, 3 failed chunks is easily that.
>> All that's required to get those builds out to people is to retrigger in
>> the treeherder UI.
>>
>> So we can benefit today.
>>
>> Having a meaningful test suite for localized builds would be great. But
>> blocking on that hasn't proven to be constructive for the past ten year.
>> https://groups.google.com/d/msg/mozilla.dev.automation/RbgbaShQb_Y/u7Tsnhj5FgAJ
>>
>> has thoughts on that, too.
>>
>> Axel
>>
>>>
>>> - Kyle
>>>
>>> On Wed, Jan 13, 2016 at 4:16 PM, Axel Hecht <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> right now, localized builds, and in particular localized Nightlies, are
>>>> part of our job exclusion profile, i.e., they're hidden from any sight.
>>>>
>>>> On the release channel, 60% of our users use those builds, and that's a
>>>> weird combination. Even on aurora, it's 50%.
>>>>
>>>> I think we should make our localized nightlies Tier 1 builds.
>>>>
>>>> There's benefit from doing that today, in terms of tracking
>>>> infrastructure
>>>> failures, and regressing landings. More on that at the end.
>>>>
>>>> Reading through
>>>> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy,
>>>> I recognize some short comings. Some of them are actual challenges in
>>>> our
>>>> status quo. Some of them might just call for bending our policies, I
>>>> think.
>>>>
>>>> The first one on the list there is "has an owner".
>>>>
>>>> And this is the first and foremost we just need to bend. (I'm
>>>> dismissing
>>>> the "active)
>>>>
>>>> L10n builds have several owners, and not accommodating for that
>>>> might be
>>>> the prime reason why we haven't assigned a tier to l10n builds.
>>>>
>>>> I think we need to deal with the distributed ownership, and find peace
>>>> with that.
>>>>
>>>> The current windows nightly bustage is a great example,
>>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an update
>>>> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS,
>>>> but (we
>>>> hope) it's missing '\' in buildbot configs. Finding this out was a
>>>> joint
>>>> effort of some 5 folks from various groups,
>>>> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466
>>>>
>>>>
>>>> .
>>>>
>>>> We should see strength in that shared ownership instead of a blocker.
>>>>
>>>> The next section is titled "Usable job logs". My brain is at odds with
>>>> this one, as I can't decipher the current logs. But I also strongly
>>>> believe
>>>> that that's tied to how my brain works with written text. We need way
>>>> more
>>>> stakeholders than me to make sure what this means.
>>>>
>>>> The remaining sections are about what to do when things break.
>>>>
>>>> I do think that we need to back out stuff if it breaks localized
>>>> builds in
>>>> general.
>>>>
>>>> I also think that our history shows that that only happens really
>>>> rarely,
>>>> so I'm not convinced that we win by making things like inbound or try a
>>>> prerequisite. There's a bunch of constructive things we can do right
>>>> now
>>>> with shared ownership that will make progress.
>>>>
>>>> That said, I do think that getting try and inbounds to expose l10n
>>>> failures (beyond what l10n-check does) is good.
>>>>
>>>> That's why I think we should make our l10n nightlies tier 1 now.
>>>>
>>>> I really expect to get some help from sheriffs immediately.
>>>> I also think that we're not getting our requirements figured out and
>>>> assigned unless we actually commit to a particular tier.
>>>>
>>>> To show some common failures these days and their respective actions:
>>>>
>>>>
>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
>>>>
>>>>
>>>> shows some infra failures and some broken builds for desktop. Some of
>>>> those
>>>> got retriggered, and for hg time-out reasons. I think this is something
>>>> that'd be OK to handle for Sheriffs, in particular if it'd be part of
>>>> the
>>>> standard display.
>>>> It also shows some per-locale bustages for Android, which are somewhat
>>>> easy to detect from the logs. It's easy to see that you don't want to
>>>> retrigger, though I have a hard time to detect what the actual
>>>> problem is.
>>>> But those can be fixed by the localizers, as
>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
>>>>
>>>>
>>>> shows. N4 going green on Android is related to Romansh fixing stuff.
>>>> Tough
>>>> to find out, but the initial error clearly looked like "locale busted".
>>>>
>>>> The thing that's sad is that there are some jobs here that should have
>>>> been retriggered, and weren't. And that I wonder if there's an infra
>>>> problem that should've been flagged when it started, and not when I
>>>> happened to look at it. hg timeouts worry me, tbh.
>>>>
>>>> And then there are rare occasions where the builds just break.
>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
>>>>
>>>>
>>>> is the most recent example, which is showing a "windows nightlies are
>>>> broken". The logs show that configure fails.
>>>>
>>>> Those are hard to resolve, but they're also easy to distinguish from
>>>> infra
>>>> or locale problems, and for those, filing them as early as we can is
>>>> key.
>>>> And getting them attention by a shared ownership will certainly help.
>>>> Sheriffs can help here initially with other bugs they've seen. And then
>>>> loop in various stakeholders, or trigger IRC/mailing list
>>>> conversations.
>>>>
>>>> We've been doing l10n builds without tier for some 10 years, and with
>>>> our
>>>> focus on quality, getting them a tier can only help. I strongly favor
>>>> getting them a tier and then fix the problems.
>>>>
>>>> Axel
>>>> _______________________________________________
>>>> dev-planning mailing list
>>>> [email protected]
>>>> https://lists.mozilla.org/listinfo/dev-planning
>>>>
>>