Fix locale mapping (#4969)

* Fix locale mapping

Now, Angular>=9.0.5 supports `zh-Hant-HK` and `zh-Hans-SG`, so we can build for them.
Also, `Match` of `golang.org/x/text/language` returns `zh-u-rg-cnzzzz` for `zh-CN` and
`zh-u-rg-txzzzz` for `zh-TW`, so we need to handle them properly before.

Since `Match` returns tag as `zh-Hant-HK` for `zh-HK`, and `zh-Hans-SG` for `zh-SG`,
so dashboard just use them.

However due to following reason, remove `zh-Hant-SG` and `zh` support for now.
* Chinese is not official in Singapore
* `zh` is not used often
* Translation files for `zh` and `zh-Hans-SG` is same as `zh-Hant` for now.

In conclusion, dashboard supports strictly matched locale IDs as followings:
* en (default)
* de
* fr
* ja
* ko
* zh-CN (zh-Hans)
* zh-TW (zh-Hant)
* zh-HK (zh-Hant-HK)

Also, support locales in lowercase and clean up `state="new"` in some translation files.

* Simplify locale matching logic
Co-authored-by: NSebastian Florek <sebastian.florek@loodse.com>

.gitignore

@@ -4,9 +4,6 @@ src/app/frontend/environments/version.ts
 # Temporary translation file:
 i18n/messages.new.xlf
 
-# Local setting for locales not to build
-i18n/locale_not_for_build_local
-
 # Local copies of dependencies that should stay on developers' local machines:
 node_modules/
 

angular.json

@@ -29,10 +29,6 @@
             "translation": "i18n/ko/messages.ko.xlf",
             "baseHref": ""
           },
-          "zh": {
-            "translation": "i18n/zh/messages.zh.xlf",
-            "baseHref": ""
-          },
           "zh-Hans": {
             "translation": "i18n/zh-Hans/messages.zh-Hans.xlf",
             "baseHref": ""
@@ -40,6 +36,10 @@
           "zh-Hant": {
             "translation": "i18n/zh-Hant/messages.zh-Hant.xlf",
             "baseHref": ""
+          },
+          "zh-Hant-HK": {
+            "translation": "i18n/zh-Hant-HK/messages.zh-Hant-HK.xlf",
+            "baseHref": ""
           }
         }
       },

docs/developer/internationalization.md

@@ -2,18 +2,16 @@
 
 Based on current browser locale the Dashboard can be displayed in one of the supported languages listed below. In case it does not work, make sure that your browser's locale is identified with correct language code. In more details, Dashboard determines requested language based on HTTP `Accept-Language` header from browser. We can check which language codes are requested by browser on `Network` tab in developer tool of browser.
 
-| Language            | Code    | Remarks    |
-|---------------------|---------|------------|
-| English (default)   | en      | -          |
-| French              | fr      | -          |
-| German              | de      | -          |
-| Japanese            | ja      | -          |
-| Korean              | ko      | -          |
-| Simplified Chinese  | zh      | -          |
-| Chinese (PRC)       | zh-cn   | Same as zh |
-| Chinese (Hong Kong) | zh-hk   | -          |
-| Chinese (Singapore) | zh-sg   | -          |
-| Chinese (Taiwan)    | zh-tw   | -          |
+| Language            | Code       | Remarks         |
+|---------------------|------------|-----------------|
+| English (default)   | en         | -               |
+| French              | fr         | -               |
+| German              | de         | -               |
+| Japanese            | ja         | -               |
+| Korean              | ko         | -               |
+| Simplified Chinese  | zh-Hans    | -               |
+| Traditional Chinese | zh-Hant    | -               |
+| Traditional Chinese (Hong Kong) | zh-Hant-HK | -               |
 
 ## Building localized dashboard
 
@@ -45,16 +43,17 @@ Find new localizable texts in `i18n/[locale]/messages.[locale].xlf` file and tra
 Since dashboard team can not review translation files in your language, so dashboard team transfers authority to review and approve for updating your translation file. At first, you need to organize translation team for your language that manages dashboard translation file.
 
 1. Create your locale directory under `i18n` directory, e.g. `i18n/fr` or `i18n/ja`.
-2. Add your locale, e.g. `fr` or `ja`, into `"languages"` array of `"xliffmergeOptions"` in `package.json` file. If you want to add only locale and use an existing translation file for it, i.e. add `zh-cn` but use existing `i18n/zh/messages.zh.xlf` file for it, skip this step and go step 5.
-  **Important: Locales should be written in lower case to be handled by Dashboard, e.g. `zh-cn`, not `zh-CN`**
+2. Add your locale, e.g. `fr` or `ja`, into `"languages"` array of `"xliffmergeOptions"` in `package.json` file. If you want to add only locale using an existing translation file, i.e. add `zh` but use existing `i18n/zh-Hans/messages.zh-Hans.xlf` file for it, skip this step and go step 5.
 3. Run `npm run fix:i18n`. Then translation file for your language, e.g. `i18n/fr/messages.fr.xlf`, would be generated in your locale directory.
-  If `i18n/[locale]/messages.[locale].xlf` is not normal file type, our script ignores `xliffmerge` for the locale.
-4. Open your translation file and translate texts in `<target>` element into your language.
-5. If you want to use an existing translation file for the locale, create symbolic link `messages.[locale].xlf` to the existing translation file like follow:
+4. Open your translation file and translate texts in `<target>` element into your language, and remove `state="new"` to mark it as translated.
+5. To build dashboard for your language, add your locale into `locales` in `angular.json` like follow:
   ```
-  cd i18n/zh-cn
-  ln -s ../zh/messages.zh.xlf messages.zh-cn.xlf
+          "ja": {
+            "translation": "i18n/ja/messages.ja.xlf",
+            "baseHref": ""
+          },
   ```
+  If you want to add only locale using an existing translation file, specify existing translation file to `"translation"`.
 
 After preparation of new translation file, configure `i18n/locale_conf.json` file to support translated dashboard as follows:
 
@@ -68,13 +67,6 @@ To add Japanese translation file, add `"ja"` into `"translations"` array in alph
 {"translations": [ "en", "fr", "ja", "ko", "zh" ]}
 ```
 
-To save time for building localized version in your develop environment, you can set locales not to build by creating `i18n/locale_not_for_build_local` and adding into it like below:
-
-```
-fr
-ko
-```
-
 Then you can build your localized dashboard with `npm run build`.
 
 Before submit Pull Request, add `i18n/[locale]/OWNERS` file for your translation team like below:

i18n/ko/messages.ko.xlf

@@ -2789,7 +2789,7 @@
       <trans-unit id="192867803de476e3137425685702cb44b3bb9981" datatype="html">
         <source><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </source>
-        <target state="new"><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
+        <target><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/chrome/nav/pinner/template.html</context>
@@ -5152,7 +5152,7 @@
       </trans-unit>
       <trans-unit id="a4f3393592e5ebfd34408e85c3bd576d88839191" datatype="html">
         <source>Disable access denied notification</source>
-        <target state="new">접근 거부 알림 해제</target>
+        <target>접근 거부 알림 해제</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">100</context>
@@ -5160,7 +5160,7 @@
       </trans-unit>
       <trans-unit id="b6484080715a11bad65395e51d42e6ae4aa3c856" datatype="html">
         <source>Hides all access denied warnings in the notification panel.</source>
-        <target state="new">알림 패널에서 접근 거부 경고 모두 숨김.</target>
+        <target>알림 패널에서 접근 거부 경고 모두 숨김.</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">102</context>

i18n/locale_conf.json

-{"translations": [ "de", "en", "fr", "ja", "ko", "zh", "zh-Hans", "zh-Hant-HK", "zh-Hans-SG", "zh-Hant" ]}
+{"translations": [ "de", "en", "fr", "ja", "ko", "zh-Hans", "zh-Hant", "zh-Hant-HK" ]}

i18n/zh-Hans-SG/OWNERS

-reviewers:
-  - chenrui333
-  - zehuaiWANG
-
-approvers:
-  - hwdef
-  - tanjunchen
-
-labels:
-- language/zh

i18n/zh-Hans/messages.zh-Hans.xlf

@@ -2789,7 +2789,7 @@
       <trans-unit id="192867803de476e3137425685702cb44b3bb9981" datatype="html">
         <source><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </source>
-        <target state="new"><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
+        <target><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/chrome/nav/pinner/template.html</context>
@@ -5155,7 +5155,7 @@
       </trans-unit>
       <trans-unit id="a4f3393592e5ebfd34408e85c3bd576d88839191" datatype="html">
         <source>Disable access denied notification</source>
-        <target state="new">禁止拒绝访问的通知</target>
+        <target>禁止拒绝访问的通知</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">100</context>
@@ -5163,7 +5163,7 @@
       </trans-unit>
       <trans-unit id="b6484080715a11bad65395e51d42e6ae4aa3c856" datatype="html">
         <source>Hides all access denied warnings in the notification panel.</source>
-        <target state="new">在通知面板中隐藏所有拒绝访问的警告。</target>
+        <target>在通知面板中隐藏所有拒绝访问的警告。</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">102</context>

i18n/zh-Hant-HK/messages.zh-Hant-HK.xlf

@@ -2793,7 +2793,7 @@
       <trans-unit id="192867803de476e3137425685702cb44b3bb9981" datatype="html">
         <source><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </source>
-        <target state="new"><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
+        <target><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/chrome/nav/pinner/template.html</context>
@@ -5155,7 +5155,7 @@
       </trans-unit>
       <trans-unit id="a4f3393592e5ebfd34408e85c3bd576d88839191" datatype="html">
         <source>Disable access denied notification</source>
-        <target state="new">禁止拒絕訪問的通知</target>
+        <target>禁止拒絕訪問的通知</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">100</context>
@@ -5163,7 +5163,7 @@
       </trans-unit>
       <trans-unit id="b6484080715a11bad65395e51d42e6ae4aa3c856" datatype="html">
         <source>Hides all access denied warnings in the notification panel.</source>
-        <target state="new">在通知面板中隱藏所有拒絕訪問的警告。</target>
+        <target>在通知面板中隱藏所有拒絕訪問的警告。</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">102</context>

i18n/zh-Hant/messages.zh-Hant.xlf

@@ -2793,7 +2793,7 @@
       <trans-unit id="192867803de476e3137425685702cb44b3bb9981" datatype="html">
         <source><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </source>
-        <target state="new"><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
+        <target><x id="INTERPOLATION" equiv-text="{{resource.displayName}}"/>
   </target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/chrome/nav/pinner/template.html</context>
@@ -5155,7 +5155,7 @@
       </trans-unit>
       <trans-unit id="a4f3393592e5ebfd34408e85c3bd576d88839191" datatype="html">
         <source>Disable access denied notification</source>
-        <target state="new">禁止拒絕訪問的通知</target>
+        <target>禁止拒絕訪問的通知</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">100</context>
@@ -5163,7 +5163,7 @@
       </trans-unit>
       <trans-unit id="b6484080715a11bad65395e51d42e6ae4aa3c856" datatype="html">
         <source>Hides all access denied warnings in the notification panel.</source>
-        <target state="new">在通知面板中隱藏所有拒絕訪問的警告。</target>
+        <target>在通知面板中隱藏所有拒絕訪問的警告。</target>
         <context-group purpose="location">
           <context context-type="sourcefile">../src/app/frontend/settings/global/template.html</context>
           <context context-type="linenumber">102</context>

i18n/zh/OWNERS

-reviewers:
-  - chenrui333
-  - zehuaiWANG
-
-approvers:
-  - hwdef
-  - tanjunchen
-
-labels:
-- language/zh

package.json

@@ -88,9 +88,7 @@
       "fr",
       "ja",
       "ko",
-      "zh",
       "zh-Hans",
-      "zh-Hans-SG",
       "zh-Hant",
       "zh-Hant-HK"
     ],

src/app/backend/handler/localehandler.go

@@ -20,6 +20,7 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
+	"strings"
 
 	"github.com/golang/glog"
 	"golang.org/x/text/language"
@@ -27,15 +28,6 @@ import (
 	"github.com/kubernetes/dashboard/src/app/backend/args"
 )
 
-// TODO(floreks): Remove that once new locale codes are supported by the browsers.
-// For backward compatibility only.
-var localeMap = map[string]string{
-	"zh-cn": "zh-Hans",
-	"zh-sg": "zh-Hans-SG",
-	"zh-tw": "zh-Hant",
-	"zh-hk": "zh-Hant-HK",
-}
-
 const defaultLocaleDir = "en"
 const assetsDir = "public"
 
@@ -95,16 +87,6 @@ func getAssetsDir() string {
 	return filepath.Join(filepath.Dir(path), assetsDir)
 }
 
-func dirExists(name string) bool {
-	if _, err := os.Stat(name); err != nil {
-		if os.IsNotExist(err) {
-			glog.Warningf(name)
-			return false
-		}
-	}
-	return true
-}
-
 // LocaleHandler serves different html versions based on the Accept-Language header.
 func (handler *LocaleHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	if r.URL.EscapedPath() == "/" || r.URL.EscapedPath() == "/index.html" {
@@ -116,61 +98,62 @@ func (handler *LocaleHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)
 	if acceptLanguage == "" {
 		acceptLanguage = r.Header.Get("Accept-Language")
 	}
+
 	dirName := handler.determineLocalizedDir(acceptLanguage)
 	http.FileServer(http.Dir(dirName)).ServeHTTP(w, r)
 }
 
 func (handler *LocaleHandler) determineLocalizedDir(locale string) string {
-	assetsDir := getAssetsDir()
-	defaultDir := filepath.Join(assetsDir, defaultLocaleDir)
-	tags, _, err := language.ParseAcceptLanguage(locale)
-	if err != nil || len(tags) == 0 {
-		return defaultDir
-	}
+	// TODO(floreks): Remove that once new locale codes are supported by the browsers.
+	// For backward compatibility only.
+	localeMap := strings.NewReplacer(
+		"zh-CN", "zh-Hans",
+		"zh-cn", "zh-Hans",
+		"zh-TW", "zh-Hant",
+		"zh-tw", "zh-Hant",
+		"zh-hk", "zh-Hant-HK",
+		"zh-HK", "zh-Hant-HK",
+	)
+
+	return handler.getLocaleDir(localeMap.Replace(locale))
+}
 
-	locales := handler.SupportedLocales
-	tag, _, confidence := language.NewMatcher(locales).Match(tags...)
+func (handler *LocaleHandler) getLocaleDir(locale string) string {
+	localeDir := ""
+	assetsDir := getAssetsDir()
+	tags, _, _ := language.ParseAcceptLanguage(locale)
+	localeMap := handler.getLocaleMap()
 
-	if confidence < language.Exact {
-		tag, confidence, err = mapLocale(locale, locales)
-		if err != nil {
-			return defaultDir
+	for _, tag := range tags {
+		if _, exists := localeMap[tag.String()]; exists {
+			localeDir = filepath.Join(assetsDir, tag.String())
+			break
 		}
 	}
 
-	matchedLocale := tag.String()
-	// If locale match is exact, then we have to manually look for proper locale code as language
-	// library contains a bug that returns invalid locale string.
-	// Related issue: https://github.com/golang/go/issues/24211
-	if confidence == language.Exact {
-		matchedLocale = ""
-		for _, l := range locales {
-			base, _ := tag.Base()
-			if l.String() == base.String() {
-				matchedLocale = l.String()
-			}
-		}
+	if handler.dirExists(localeDir) {
+		return localeDir
 	}
 
-	localeDir := filepath.Join(assetsDir, matchedLocale)
-	if matchedLocale != "" && dirExists(localeDir) {
-		return localeDir
+	return filepath.Join(assetsDir, defaultLocaleDir)
+}
+
+func (handler *LocaleHandler) getLocaleMap() map[string]struct{} {
+	result := map[string]struct{}{}
+	for _, tag := range handler.SupportedLocales {
+		result[tag.String()] = struct{}{}
 	}
-	return defaultDir
+
+	return result
 }
 
-// Used to map old locale codes to new ones, i.e. zh-cn -> zh-Hans
-func mapLocale(locale string, locales []language.Tag) (language.Tag, language.Confidence, error) {
-	if mappedLocale, ok := localeMap[locale]; ok {
-		locale = mappedLocale
-		tags, _, err := language.ParseAcceptLanguage(locale)
-		if (err != nil) || (len(tags) == 0) {
-			return language.Tag{}, language.No, err
+func (handler *LocaleHandler) dirExists(name string) bool {
+	if _, err := os.Stat(name); err != nil {
+		if os.IsNotExist(err) {
+			glog.Warningf(name)
+			return false
 		}
-
-		tag, _, confidence := language.NewMatcher(locales).Match(tags...)
-		return tag, confidence, nil
 	}
 
-	return language.Tag{}, language.No, nil
+	return true
 }

src/app/backend/handler/localehandler_test.go

@@ -141,7 +141,7 @@ func TestDetermineLocale(t *testing.T) {
 		},
 		{
 			&LocaleHandler{
-				SupportedLocales: languageMake([]string{"en", "zh-tw", "zh-hk", "zh", "ar-dz"}),
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hant", "zh-Hans", "ar-DZ"}),
 			},
 			true,
 			"en",
@@ -149,7 +149,7 @@ func TestDetermineLocale(t *testing.T) {
 		},
 		{
 			&LocaleHandler{
-				SupportedLocales: languageMake([]string{"en", "zh-tw", "zh-hk", "zh", "ar-dz"}),
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hant", "zh-Hans", "ar-DZ"}),
 			},
 			true,
 			"zh",
@@ -157,27 +157,51 @@ func TestDetermineLocale(t *testing.T) {
 		},
 		{
 			&LocaleHandler{
-				SupportedLocales: languageMake([]string{"en", "zh-tw", "zh-hk", "zh", "ar-dz"}),
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hant", "zh-Hans", "ar-DZ"}),
 			},
 			true,
-			"ar",
-			filepath.Join(assetsDir, "en"),
+			"ar-DZ",
+			filepath.Join(assetsDir, "ar-DZ"),
 		},
 		{
 			&LocaleHandler{
-				SupportedLocales: languageMake([]string{"en", "zh-tw", "zh-hk", "zh", "ar-dz"}),
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hant", "zh-Hans", "ar-DZ"}),
 			},
 			true,
-			"ar-bh",
-			filepath.Join(assetsDir, "en"),
+			"ar-BH",
+			defaultDir,
 		},
 		{
 			&LocaleHandler{
-				SupportedLocales: languageMake([]string{"en", "zh-tw", "zh", "ar-dz"}),
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hans", "zh-Hant", "zh-Hant-HK"}),
 			},
 			true,
 			"af,zh-HK,zh;q=0.8,en;q=0.6",
-			filepath.Join(assetsDir, "zh"),
+			filepath.Join(assetsDir, "zh-Hant-HK"),
+		},
+		{
+			&LocaleHandler{
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hans", "zh-Hant", "zh-Hant-HK"}),
+			},
+			true,
+			"af,zh-TW,zh;q=0.8,en;q=0.6",
+			filepath.Join(assetsDir, "zh-Hant"),
+		},
+		{
+			&LocaleHandler{
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hans", "zh-Hant", "zh-Hant-HK"}),
+			},
+			true,
+			"zh-tw",
+			filepath.Join(assetsDir, "zh-Hant"),
+		},
+		{
+			&LocaleHandler{
+				SupportedLocales: languageMake([]string{"en", "zh", "zh-Hans", "zh-Hant", "zh-Hant-HK"}),
+			},
+			true,
+			"zh-hant-hk",
+			filepath.Join(assetsDir, "zh-Hant-HK"),
 		},
 	}
 
@@ -198,7 +222,7 @@ func TestDetermineLocale(t *testing.T) {
 			}
 			actual := c.handler.determineLocalizedDir(c.acceptLanguageKey)
 			if !reflect.DeepEqual(actual, c.expected) {
-				t.Errorf("localeHandler.determineLocalizedDir() returns %#v, expected %#v", actual, c.expected)
+				t.Errorf("localeHandler.determineLocalizedDir(%#v) returns %#v, expected %#v", c.acceptLanguageKey, actual, c.expected)
 			}
 		}()
 	}