Re: Getting a Tier-X for localized builds. X=1

In general, yes, we should make localizations infallible, and I wish we 
could.

In my considerations for the really-next-big-shift for l10n, that'll 
come. Or the second. Updates soon here.

In the particular case of the failing android builds, this was the 
following scenario:

<!ENTITY my-bad "This is &something; aweful">

The sad truth is, I haven't found out a reliable way to say whether 
&something; is resolving to something in the document that's including 
the DTD, or if it's gonna produce an xml parsing error.

In the case of Romansh, the localizer was using &apos;, which resolves 
in HTML docs, but not in the plain XML that the strings.xml on Android is.

Right now, these are "Warnings". Some of those are OK, some of these are 
not. Like, if a localizer uses &brandShortName; instead of 
&brandShorterName; is that bad enough to drop their localization from 
the build? This is a problem of one edge case vs the other, and I don't 
have strong data to educate us if there's one side to pick.

I don't expect that we'll advance off of this particular problem unless we

- move to a localization infrastructure which has stronger semantics of 
context of localization
- move to a localization infrastructure which can recover from unknown 
references at runtime (extra candy for doing so on native platforms)

Also, if we got meaningful runtime testing for localized builds, we 
could do stuff. "Meaningful" being both in scope and in educating to a 
non-technical localizer as to what the fix is.

The cases where localizers introduce problems are actually pretty rare 
these days, though.

....

On 14/01/16 02:06, Ryan VanderMeulen wrote:
> And I guess there's a related question there as to why can't we
> better-validate string changes before they get checked into the repo (or
> at least immediately after)?

The validation part that we can do is actually part of l10n-merge, which 
in cases where a bot can make the call that a string is bad, removes it 
from the localized file.

So a thing like

<!ENTITY oh-my "&doh">

is just gonna be dropped, and if "oh-my" was a string that en-US has, 
it'd be added to the localized file at build time.

Axel

> On 1/13/2016 8:00 PM, Ryan VanderMeulen wrote:
>> The other big issue I see is that l10n builds can break at any time due
>> to changes from the multitude of external repos where the strings are
>> actually stored (i.e. someone lands a busted string in some locale's
>> repo, next day's l10n nightlies break as a result). This was an issue we
>> pushed back hard against in the early B2G days because it makes
>> sheriffing a huge pain. The solution for B2G was in-tree manifests
>> automatically updated via the B2G Bumper Bot. Not sure what we could do
>> for l10n builds.
>>
>> -Ryan
>>
>> On 1/13/2016 7:41 PM, Axel Hecht wrote:
>>> On 14/01/16 01:22, Kyle Huey wrote:
>>>> IMO in the most daunting obstacle is one that you do not mention at
>>>> all:
>>>> l10n builds are not scheduled on every push.
>>>
>>> I actually don't think this is an obstacle. Or should be.
>>>
>>> I also don't agree that we're not doing l10n builds per push. We do,
>>> they're the l10n-check target in browser/locales, and they're run on try
>>> and other automation,
>>> https://dxr.mozilla.org/mozilla-central/source/build/moz-automation.mk#37.
>>>
>>>
>>>
>>> They're catching some errors, but not others. For example, the try runs
>>> on https://bugzilla.mozilla.org/show_bug.cgi?id=1215694 failed for much
>>> of its development cycle. And once they were green, the only thing that
>>> was left busted were manifests in language packs.
>>>
>>> In contrast, we have dozens of locales/platform combinations failing
>>> each day for mercurial timeouts. Like, 3 failed chunks is easily that.
>>> All that's required to get those builds out to people is to retrigger in
>>> the treeherder UI.
>>>
>>> So we can benefit today.
>>>
>>> Having a meaningful test suite for localized builds would be great. But
>>> blocking on that hasn't proven to be constructive for the past ten year.
>>> https://groups.google.com/d/msg/mozilla.dev.automation/RbgbaShQb_Y/u7Tsnhj5FgAJ
>>>
>>>
>>> has thoughts on that, too.
>>>
>>> Axel
>>>
>>>>
>>>> - Kyle
>>>>
>>>> On Wed, Jan 13, 2016 at 4:16 PM, Axel Hecht <[email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> right now, localized builds, and in particular localized Nightlies,
>>>>> are
>>>>> part of our job exclusion profile, i.e., they're hidden from any
>>>>> sight.
>>>>>
>>>>> On the release channel, 60% of our users use those builds, and
>>>>> that's a
>>>>> weird combination. Even on aurora, it's 50%.
>>>>>
>>>>> I think we should make our localized nightlies Tier 1 builds.
>>>>>
>>>>> There's benefit from doing that today, in terms of tracking
>>>>> infrastructure
>>>>> failures, and regressing landings. More on that at the end.
>>>>>
>>>>> Reading through
>>>>> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy,
>>>>> I recognize some short comings. Some of them are actual challenges in
>>>>> our
>>>>> status quo. Some of them might just call for bending our policies, I
>>>>> think.
>>>>>
>>>>> The first one on the list there is "has an owner".
>>>>>
>>>>> And this is the first and foremost we just need to bend. (I'm
>>>>> dismissing
>>>>> the "active)
>>>>>
>>>>> L10n builds have several owners, and not accommodating for that
>>>>> might be
>>>>> the prime reason why we haven't assigned a tier to l10n builds.
>>>>>
>>>>> I think we need to deal with the distributed ownership, and find peace
>>>>> with that.
>>>>>
>>>>> The current windows nightly bustage is a great example,
>>>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1239074. There's an
>>>>> update
>>>>> to NSIS, and Nightlies broke. The fix has nothing to do with NSIS,
>>>>> but (we
>>>>> hope) it's missing '\' in buildbot configs. Finding this out was a
>>>>> joint
>>>>> effort of some 5 folks from various groups,
>>>>> http://logs.glob.uno/?c=mozilla%23releng&s=12+Jan+2016&e=12+Jan+2016&h=pike#c217466
>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>> We should see strength in that shared ownership instead of a blocker.
>>>>>
>>>>> The next section is titled "Usable job logs". My brain is at odds with
>>>>> this one, as I can't decipher the current logs. But I also strongly
>>>>> believe
>>>>> that that's tied to how my brain works with written text. We need way
>>>>> more
>>>>> stakeholders than me to make sure what this means.
>>>>>
>>>>> The remaining sections are about what to do when things break.
>>>>>
>>>>> I do think that we need to back out stuff if it breaks localized
>>>>> builds in
>>>>> general.
>>>>>
>>>>> I also think that our history shows that that only happens really
>>>>> rarely,
>>>>> so I'm not convinced that we win by making things like inbound or
>>>>> try a
>>>>> prerequisite. There's a bunch of constructive things we can do right
>>>>> now
>>>>> with shared ownership that will make progress.
>>>>>
>>>>> That said, I do think that getting try and inbounds to expose l10n
>>>>> failures (beyond what l10n-check does) is good.
>>>>>
>>>>> That's why I think we should make our l10n nightlies tier 1 now.
>>>>>
>>>>> I really expect to get some help from sheriffs immediately.
>>>>> I also think that we're not getting our requirements figured out and
>>>>> assigned unless we actually commit to a particular tier.
>>>>>
>>>>> To show some common failures these days and their respective actions:
>>>>>
>>>>>
>>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=7f6dc6f3589b&filter-job_group_symbol=L10n&exclusion_profile=false
>>>>>
>>>>>
>>>>>
>>>>> shows some infra failures and some broken builds for desktop. Some of
>>>>> those
>>>>> got retriggered, and for hg time-out reasons. I think this is
>>>>> something
>>>>> that'd be OK to handle for Sheriffs, in particular if it'd be part of
>>>>> the
>>>>> standard display.
>>>>> It also shows some per-locale bustages for Android, which are somewhat
>>>>> easy to detect from the logs. It's easy to see that you don't want to
>>>>> retrigger, though I have a hard time to detect what the actual
>>>>> problem is.
>>>>> But those can be fixed by the localizers, as
>>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=cb11e926ac33&filter-job_group_symbol=L10n&exclusion_profile=false
>>>>>
>>>>>
>>>>>
>>>>> shows. N4 going green on Android is related to Romansh fixing stuff.
>>>>> Tough
>>>>> to find out, but the initial error clearly looked like "locale
>>>>> busted".
>>>>>
>>>>> The thing that's sad is that there are some jobs here that should have
>>>>> been retriggered, and weren't. And that I wonder if there's an infra
>>>>> problem that should've been flagged when it started, and not when I
>>>>> happened to look at it. hg timeouts worry me, tbh.
>>>>>
>>>>> And then there are rare occasions where the builds just break.
>>>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=531d1f6d1cde&exclusion_profile=false&filter-job_group_symbol=L10n
>>>>>
>>>>>
>>>>>
>>>>> is the most recent example, which is showing a "windows nightlies are
>>>>> broken". The logs show that configure fails.
>>>>>
>>>>> Those are hard to resolve, but they're also easy to distinguish from
>>>>> infra
>>>>> or locale problems, and for those, filing them as early as we can is
>>>>> key.
>>>>> And getting them attention by a shared ownership will certainly help.
>>>>> Sheriffs can help here initially with other bugs they've seen. And
>>>>> then
>>>>> loop in various stakeholders, or trigger IRC/mailing list
>>>>> conversations.
>>>>>
>>>>> We've been doing l10n builds without tier for some 10 years, and with
>>>>> our
>>>>> focus on quality, getting them a tier can only help. I strongly favor
>>>>> getting them a tier and then fix the problems.
>>>>>
>>>>> Axel
>>>>> _______________________________________________
>>>>> dev-planning mailing list
>>>>> [email protected]
>>>>> https://lists.mozilla.org/listinfo/dev-planning
>>>>>
>>>
>