[tor-qa] Pre-build announcement for 3.6-beta-1

[anonym]

13/03/14 20:59, Mike Perry wrote:
> anonym:
[...]
>> To make problem 1 less of a problem we could use a tool which can sync
>> all non-English DTD:s to the English one. It be invoked via something
>> like `make update-translations`, making it possible refresh the DTD
>> files so one can package a Tor Launcher that works for all locales
>> without having to involve Transifex (that can wait until right before
>> release time). I'm quite ignorant of XML/DTD stuff, but clearly
>> Transifex must make use of a tool like that. Any ideas about this?
> 
> Yes, however, it takes a few hours, and has had some issues (which they
> are in the process of resolving. See
> https://trac.torproject.org/projects/tor/ticket/9144).
> 
> I am not sure if their DTD merge tool is open source/available.
> Probably?
>  
>> Problem 2: Another issue is that `chrome.manifest` doesn't list English
>> as the first locale. If it did, English would be the fallback for
>> locales not listed in `chrome.manifest` (note that this would not solve
>> problem 1!). For the record, the current fallback is `af`, which doesn't
>> even have a directory in `chrome/locale`, so all unlisted locales are
>> broken.
> 
> Ah, I was actually not aware that the first locale was the fallback.

To be honest I was a bit frustrated by the lack of in-depth
documentation about this from Mozilla (or if it exists, that they've
hidden it so well), so I just went with what I had seen working through
experimentation. On second thought, we shouldn't rely on the order. I
believe it's just a coincidence that this works at the moment.

I've yet to see it properly documented (except on a third-party site
[1]) but it seems that "en-US" is first tried as a fallback. Tor
Launcher currently registers "en", which isn't good enough, apparently.
Renaming it to "en-US" makes it the fallback no matter in which order
the locales are registered.

Because of this, you may want to rename "en" to "en-US" in your
translation branches (possibly in Torbutton and any other
Firefox/xulrunner repo too, didn't check).

[1]
http://babelwiki.babelzilla.org/index.php?title=Create_a_Locale_Support_for_your_Extension#Why_your_extension_needs_locale_support

> Is it per-entity, or is it only used as a fallback if a locale is entirely
> absent? It sounds like only if a locale is absent?

Yes, only if the locale is absent. There's no per-entity fallback, which
is the root cause of problem 1.

>> Problem 3: However, `chrome.manifest` has a pretty exhaustive list of
>> locales, so the broken fallback isn't used in many cases. Unfortunately
>> most of them are pointing either to non-existing directories in
>> `chrome/locale` (fatal error similar to the one from problem 1), and to
>> existing directories which contain translations that are pretty much
>> guaranteed to be outdated (and hence probably hit by problem 1) because
>> they're not updated by `localization/import-translations.sh`. Hence all
>> these locales will experience similar errors to what lunar reported.
>>
>> For problems 2 and 3 I think some `make` automation can improve the
>> situation quite a bit. See the attached patch for something like that.
> 
> Right. This sounds like a good start. I would prefer only one list of
> locales to update though.. Ideally in a form that makes it easy to
> cut+paste from the TBB list (ie an env var updated in two places like we
> have now).
>
> In either case, we should only list locales supported in transifex
> though, it sounds like.

To avoid problem 3, yes.

An alternative approach to maintaining a static list of supported
locales (which is error prone, see commit f9759b6 :)) would be to import
and bundle/register all locales available in Transifex:

1. In `import-translations.sh` we import all locales available in
Transifex into `chrome/locale`.
2. When packaging we register (in `chrome.manifest`) each locale (i.e.
folder) present in `chrome/locale` that has all required translation
files (i.e. the same files as the English locale). At the moment e.g.
`ca_ES` and `ms` do not have all, so this check is necessary to not
break those locales.

This would result in Tor Launcher bundling translations with lots of
untranslated strings (possibly 100% untranslated in some) which I'm not
sure how you feel about. OTOH, your current process doesn't guarantee
anything about that even for the explicitly listed supported locales
(except if you manually check). Still, one can easily make it so that
only the locales specified by some env var are packaged, which seems to
be what you want.

[...]
> I am very interested in arriving at a solution that works for both of us
> out of the box, but I also want to maintain the ability to make it
> straightforward to verify that the list of locales in TBB, Torbutton,
> and Tor Launcher are the same. This is most easily done if we have a
> single line of locales in an env var.
> 
> Otherwise, your patch looks like a good direction to head in.

See the `locale_fix` branch on git://git.tails.boum.org/tor-launcher for
an improvement using the "alternative approach" mentioned above.

In that branch, building a working package that includes all locales
from Transifex' translations amounts to:

    make import-translations
    git add src/chrome/locale
    git commit -m "Import translations."
    make package

Note that we have to assume that Transifex' translation repos are
up-to-date; we only ensure that each bundled locale has the necessary
.dtd and .properties files, not that they have all required
entities/strings. That would be a nice fail-safe against problem 1 but
we'll have to find the appropriate tools first.

To only bundle selected locales, as you requested, one exports
`BUNDLE_LOCALES` appropriately, e.g. with your old list:

    export BUNDLE_LOCALES="ar de es fa fr it ko nl pl pt ru vi zh-CN"
    make package

Unlike the rather optimistic "bundle everything" case, this will throw
an error if any of them couldn't be bundled for whatever reason, as a
fail-safe. Also note that the "en-US" locale is always automatically
included since it's the fallback.

Hopefully this suits your needs.

Cheers!