Proposed RFC Feature: Internationalization Support

[Shirley-yhm]

Summary:

This feature proposes to implement internationalization in O3DE to support other languages besides English on UI.

What is the relevance of this feature?

O3DE only supports English on UI currently, which is not so friendly to users from other countries. Especially for new users who had never done 3D product development before, support for their native language might help them start with O3DE and understand the terminologies more easily.

The proposed solution implements internationalization support in O3DE. After this work, developers only need to add a new language type and translate strings that have already been extracted to support any other language as they want.

Feature design description:

Mechanism of QT's internationalization support is used in this feature. The basic flow is as follows (Chinese translation for example).

1. Tag —— Tag all strings to be translated on UI in source code.

   // Before tagged
   AZ_Printf("Editor", "Hello");
   
   // After tagged
   AZ_Printf("Editor", tr("Hello"));
   AZ_Printf("Editor", QObject::tr("Hello"));
   AZ_Printf("Editor", QT_TRANSLATE_NOOP("xxx", "Hello"));

2. Extract —— Use lupdate in QT to extract tagged strings into translation files with .ts suffix.

   • Directory Structure:

       >Translations
           >en
               >Editor_en.ts
               >qtbase_en.ts
           >zh_CN
               >Editor_zh_CN.ts
               >qtbase_zh_CN.ts
     

   • Information of the tagged string with file path and line number will be added into TS file.

     <message>
         <location filename="../relative_path/Example.cpp" line="100"/>
         <source>Hello</source>
         <translation type="unfinished"></translation>
     </message>

3. Translate —— Use Linguist Tool in QT to translate tagged strings in translation files. Developers can also modify TS files directly.
   • Chinese translation will be added, and the type will be marked as finished.

     <message>
         <location filename="../relative_path/Example.cpp" line="100"/>
         <source>Hello</source>
         <translation>你好</translation>
     </message>

4. Compile —— Use lrelease in QT to compile the translation files with .qm suffix.

   • Directory Structure:

       >Translations
           >en
               >Editor_en.qm
               >qtbase_en.qm
           >zh_CN
               >Editor_zh_CN.qm
               >qtbase_zh_CN.qm
     

5. Set Translators —— Set translators for every module needs to be translated.
   • Translations are seperated into different files for every module. In particular, qtbase is added to locate translations for basic Qt modules, such as Qt Core, Qt GUI, Qt Network, and Qt Widgets

     QStringList translatorNames = { 
         "Editor",
         "qtbase",
         // More modules like AssetProcessor, AzToolsFramework, and every Gem...
     };

6. Load/Unload Translator —— Load translators before UI is initialized, and unload translators before UI exits.

Technical design description:

Internationalization with QT

Internationalization with QT supports most languages in use today, and and will work on all platforms as long as the system has fonts to render these writing systems installed.

Following APIs are mainly used:

API	description
QTranslator	Provides internationalization support for text output.
QLocale	Converts between numbers and their string representations in various languages.
QTextCodec	Provides conversions between text encodings.

Three tools in QT are used to generate translation files automatically:

• lupdate is used to generate translation source (TS) files with all the user-visible text (context, file path, line number etc.) but no translations.
• QT Linguist is used to add translations in the TS files.
• lrelease is used to read the TS files and produce the QM files used by the application at runtime.

Implementation in O3DE

Currently, some strings to be translated have been tagged using Object::tr(). Translators are loaded in EditorQtApplication::Initialize().

void EditorQtApplication::Initialize()
{
    // Install QTranslator
    InstallEditorTranslators();
}

void EditorQtApplication::InstallEditorTranslators()
{
    m_editorTranslator =        CreateAndInitializeTranslator("editor_en-us.qm", ":/Translations");
    m_assetBrowserTranslator =  CreateAndInitializeTranslator("assetbrowser_en-us.qm", ":/Translations");
}

QTranslator* EditorQtApplication::CreateAndInitializeTranslator(const QString& filename, const QString& directory) { }

void EditorQtApplication::UninstallEditorTranslators() { }

However, the current implementation only supports localization in Editor and Asset Browser, and strings not in subclasses of QObject can't be translated.

Step 1. Set Translators

Based on what has been done in O3DE, this proposal uses m_translators with type QVector to store translators.

QVector<QTranslator*> m_translators;

void EditorQtApplication::InstallTranslators()
{
    QTextCodec::setCodecForLocale(QTextCodec::codecForName("utf-8"));

    QStringList translatorNames = { "Editor", "qtbase", "AzToolsFramework" }; // Translators of Editor for example
    // QStringList translatorNames = { "qtbase", "AssetProcessor", "AzToolsFramework", "AzQtComponents" }; // Translators of Asset Processor for example

    for (QString trans : translatorNames)
    {
        m_translators.append(AzToolsFramework::CreateAndLoadTranslator(trans));
    }
}

Step 2. Load & Unload Translators

Since the startup time of each process are actually different, translators are loaded in every module separately.

• Common functions are defined in file Translation.h.

  enum class Language
  {
      English,
      Chinese
      // ... Other languages will be added later.
  };
  
  Language GetLanguage()
  {
      QSettings settings("O3DE", "O3DE Editor");
      int language = settings.value("Settings/Language").toInt();
      return static_cast<Language>(language);
  }
  
  inline QTranslator* CreateAndLoadTranslator(const QString& modulename)
  {
      QString filename;
      switch (GetLanguage())
      {
      case Language::Chinese:
          filename = "zh_CN/" + modulename + "_zh_CN.qm";
          break;
      // ... Other languages will be added later.
      default:
          break;
      }
      return LoadTranslator(filename);        
  }
  
  inline QTranslator* LoadTranslator(const QString& filename)
  {
      QString translationFilePath;
      QTranslator *translator = new QTranslator();
      translator->load(translationFilePath);
      qApp->installTranslator(translator);
      return translator;
  }
  
  inline void UnloadTranslator(QTranslator* translator)
  {
      qApp->removeTranslator(translator);
      delete translator;
      translator = nullptr;
  }

• Translators are loaded/unloaded in every module separately.

  // Editor: Code/Editor/CryEdit.cpp
  extern "C" int AZ_DLL_EXPORT CryEditMain(int argc, char* argv[])
  {
      // ...
      Editor::EditorQtApplication::instance()->InstallEditorTranslators();
      // ...
      Editor::EditorQtApplication::instance()->UninstallEditorTranslators();
  }
  
  // Asset Processor: Code/Tools/AssetProcessor/native/utilities/GUIApplicationManager.cpp
  bool GUIApplicationManager::Run()
  {
      // ...
      InstallTranslators();
      // ...
      UninstallTranslators();
  }

Step 3. Tag

There are three types of strings to translate:

• If the string is in a subclass of QObject, just wrap it with tr().
• If the string is in an environment of QT, it can be wrapped with QObject::tr() or SubclassOfQObject::tr(). In particular, QT can't be used in code of runtime.
• For other strings, tags should be added using QT_TRANSLATE_NOOP macro.

  #ifndef QT_TRANSLATE_NOOP
      #define QT_TRANSLATE_NOOP(scope, x) (x)
  #endif

Step 4. Extract

Strings tagged are extracted into corresponding TS files automatically.

• A cmake file is used to generate TS files automatically.

  function(add_translation_module target_source_dir module_name)
      if (PAL_TRAIT_BUILD_HOST_TOOLS)
          find_package(Qt5 COMPONENTS LinguistTools REQUIRED) # Linguist
          find_program(LUPDATE_EXECUTABLE lupdate PATHS "${Qt5_DIR}/../../../bin" REQUIRED) # lupdate
          set(LANGUAGES zh_CN) # Other languages will be added later.
          foreach(_language ${LANGUAGES})
              execute_process(COMMAND ${LUPDATE_EXECUTABLE} ${target_source_dir} -ts ${TS_FILE} -silent)
          endforeach()
      endif()
  endfunction()

• Then, call add_translation_module in cmake file of every module.

  add_translation_module(${CMAKE_CURRENT_SOURCE_DIR} Editor) // For Editor: Code/Editor/CMakeLists.txt
  add_translation_module(${CMAKE_CURRENT_SOURCE_DIR} AssetProcessor) // For Asset Processor: Code/Tools/AssetProcessor/CMakeLists.txt

• After built, information of the tagged string with file path and line number will be added into TS files like Editor_zh_CN.ts and AssetProcessor_zh_CN.ts.

  <message>
      <location filename="../relative_path/Example.cpp" line="100"/>
      <source>Hello</source>
      <translation type="unfinished"></translation>
  </message>

Step 5. Translate

TS files are in type of XML. Translations can be added manually or using Qt Linguist tool.

• Translation will be added into <translation> </translation>, and the type will be removed indicating that translation of this string has been finished.

  <message>
      <location filename="../relative_path/Example.cpp" line="100"/>
      <source>Hello</source>
      <translation>你好</translation>
  </message>

• Once a string in TS file is retranslated, the corresponding TS file needs to be reuploaded into the repository.
• Everytime the source code is recompiled, line numbers in TS files will be updated automatically. In this case, there is no need to reupload TS files.

After translation, TS files will be simplified and compiled into QM files with suffix .qm automatically. QM files are fast compact versions of TS files.

• A cmake file is used to generate QM files automatically.

  find_package(Qt5 COMPONENTS LinguistTools REQUIRED) # Linguist
  find_program(LRELEASE_EXECUTABLE lrelease PATHS "${Qt5_DIR}/../../../bin" REQUIRED) # lrelease
  set(LANGUAGES zh_CN) # Other languages will be added later.
  foreach(_language ${LANGUAGES})
      foreach(_ts_file ${TS_FILES})
          execute_process(COMMAND ${LRELEASE_EXECUTABLE} ${_ts_file} -qm "${QM_DIR}/${_language}/${QM_NAME}.qm" -silent)
      endforeach()
  endforeach()

• QM files are generated by build system and they don't need to be uploaded into the repository.

Step 6. Read

• Translations of strings tagged by tr() or Object::tr() can be read straightly.
• Translations of strings tagged by QT_TRANSLATE_NOOP macro need to be called by QCoreApplication::translate().

  // Tag text and tooltip of the button.
  void CEditorPreferencesPage_ViewportCamera::CameraMovementSettings::Reflect(AZ::SerializeContext& serialize)
  {
      editContext->Class<CameraMovementSettings>("Camera Movement Settings", "")
      ->DataElement(AZ::Edit::UIHandlers::Button, &CameraMovementSettings::m_resetButton, "", QT_TRANSLATE_NOOP("Reflect", "Restore camera movement settings to defaults"))
      ->Attribute(AZ::Edit::Attributes::ButtonText, QT_TRANSLATE_NOOP("Reflect", "Restore defaults"))
  }
  
  // Read translation of buttons' text.
  void PropertyButtonCtrl::SetButtonText(const char* text)
  {
      m_button->setText(QCoreApplication::translate("Reflect", text));
  }
  
  // Read translation of buttons' tooltip.
  void PropertyButtonCtrl::SetButtonToolTip(const char* description)
  {
      m_button->setToolTip(QCoreApplication::translate("Reflect", description));
  }

Language Setting

A new setting for language is also needed in Editor Settings for users to switch language. It's designed like the setting named Console Background.

• It's added in Editor Settings - General Editor Preferences - General Settings. Other languages will be added later.
• It's stored in Settings Registry.

Work Plan

1. Firt PR: Basic architeture of localization & Chinese translation of Editor and Asset Processor
   • Tag strings in Editor and Asset Processor.
   • Add cmake files to generate translation files (TS and QM) automatically.
   • Translate tagged strings into Chinese.
   • Provide back-end code to load the translation files at startup and unload them at teardown.
   • Add a setting for language in Editor Settings.
2. Follow-up documentation: Documents about how to use and how to contribute to this feature.
3. Incremental PRs: Chinese translations of other modules like Gems.
4. Translations in other languages contributed by the community.

What are the advantages of the feature?

• Users can choose their preferred language freely when use O3DE.
• The barrier to entry is lowered for those who had never done 3D product development before.
• Localization of other languages can be added easily in the future.

How will this be implemented or integrated into the O3DE environment?

CMake build files will be added in the first PR. No additional processing is required by developers.

How will users learn this feature?

Once this proposal is passed, documents will be provided for users and developers to learn this feature.

• For users, this feature can be added in USER GUIDE to let them know how to switch language on UI in O3DE.
• For developers, documents about how to add localization for other modules or other languages can be added in TOOLS UI.

Are there any open questions?

• How can developers add support for other languages?
  • Once we support localization for Chinese, there will be no need to tag strings in the future. Developers just need to add language type in settings and translations in corresponding TS files to support localization for any other language.
• Is parameterized template supported in strings to be translated?
  • Yes, it is. For example, QString str = tr("Loading level %1 ...").arg(levelName) will be extracted and translated in TS file as:

  <message>
      <location filename="../CryEdit.cpp" line="1240"/>
      <source>Loading level %1 ...</source>
      <translation>正在加载关卡%1...</translation>
  </message>

Potential User Experience Improvements

According to the user experience we have collected, there are two questions worth considered.

1. Editor needs to restart to change the display language currently. Since it takes a long time to launch Editor and Asset Processor, restarting may lead to poor user experience.

   • Is there any way to achieve hot reload in O3DE to avoid restart when change genaral settings?

2. It's hard to translate some terminologies accurately for developers, and inappropriate translations may confuse users.

   • Once hot reload is achieved, users wil be able to switch language to understand terminologies easily. But if not, is there any way to simplify users' operations when they want to know the meaning of the terminologies in other languages?
   • Some experiment has been done in this scenario: Add description in English into tooltips when the display language is set to Chinese. However, this may lead to overly long tooltips and poor user experience. This will also increase the burden on developers who add translations.

     // tooltip in English
     Hello, O3DE
     
     // tooltip in Chinese
     Hello, O3DE
     你好，ODE
     

   • Another option is to add a button in every component which allows users to change the language partially.

   Both of the above two questions need UX design and technical support.

3. Title of window's name is used as the key to store its layout. When the display language changed, key of the window will also change and information of its layout will become invalid.

   // Information of layout in English
   ["Asset Browser", struct_of_layout]
   
   // Information of layout in English
   ["资产浏览器", struct_of_layout]

   • Is it possible to decouple the identifier and title of every module?

[Shirley-yhm]

We've had some discussions in o3de/o3de#14287

[Ulrick28]

Awesome feature, thanks for the rfc! An option for terminology that doesn't translate well is to just not translate that particular item until an appropriate translation is determined. Many products follow this model.

Ideally I would want to see testing against at least 3 languages (english, chinese, and ???) but understand that may not be feasible. Also, there will be considerations for button sizing and alignment in the editor. Example german is known for long translations, japanese fonts tend to be taller. We will also need to test what kind of performance and memory impact this will have.

[nick-l-o3de]

I still think its a great idea to get the architecture in place and start making steps towards doing this. No reason not to start just because the road is long.

The only point I'd like to try to make here is that we have a bunch of code that isn't necessarily based on Qt and still emits english errors and other text (such as the runtime, other tools potentially, python scripts). in which case we'd need to think of whether we want a separate mechanism for translating those, since putting Qt in them would probably not be okay from a license (GPL-3 vs Apache) situation.

Its possible we could still use the same tags and macros, tooling (Lupdate, qt linguist) but use them without linking to qt.

[Shirley-yhm]

Thanks for your feedback :)

1. Test against at least 3 languages

• I'm sorry that no one in my team knowns other languages besides English and Chinese. We have tested other languages like Japanese in some modules, it can be added successfully like Chinese.

2. UI adaptation

• UI adaptation and UX are worth considering. I'm afraid we need professionals to help design it and it may take a lot of time. However, I don't think this problem should hinder our progress in this RFC, cause we have added Chinese translations in our branch and everything goes well. After the first PR, developers from other countries can follow the guidance to add translations of other languages, from which we can gather advices on UI and UX.

3. Performance and Memory impact

• We have added translations on our branch for every modules, including Editor, Asset Processor and Gems. The total size of TS files is around 4 MB, and the total size of QM files is around 1 MB. The built from our branch has been used in my team for a long time, and the impact on performance is also not obvious.

4. License of QT

• We have thought about it before. The tools (lupdate, Linguist, lrelease) are only used to generate QM files by developers when they add translations, and they won't link to the engine.

[Shirley-yhm]

I had some discussion with @yuyihsu, @bhanuja-s(from UX team) and @AMZN-daimini recently. Combined with the above in this RFC, here are some conclusion related.

• What we can do in the long term:

  1. Hot reload. We should display immediate changes in UI when change to a new language.
  2. Decouple the identifier and title of every module, so that keys used for layout won't be translated.
  3. Let other members in the community help with other languages, then make some adaptation on UI/UX.

• What I will adjust in the PR of this RFC in the short term:

  1. Provide an option in the installer to select language.
  2. Provide an option in Project Manager to select language, rather than in Editor.
  3. Prompt users to save the level/file when apply a new language, especially when Editor is open.

Should I edit this RFC directly? What should I do once the above 3 points are finished? Please let me know asap if there is any other suggestions. Thanks!

---

For the sake of future discussion, I put the main contents of our discussion here.

• From: Sanghavi, Bhanuja

  With the time (and steps) it takes to re-launch the Editor, users should ideally be provided with a solution that changes the language in the Editor. However, with that being a long-term goal, I believe there might be ways that we can improve this in the short-term.

  1. Users should be provided an option in the Installer to select a language. When the download and installation is complete, all the engine tools and windows should be in the new language.
  2. Users should be provided an option in Project Manager to select a language, rather than in the Editor – this way the changes are global and reflect across the different engine tools and windows (Project Manager, Editor, Asset Processor).
     • In the short-term, if the Editor is open they should be well instructed to save open levels/files, and to then relaunch the Editor for these changes to be applied.
     • In the long-term, we should display immediate changes in the UI to the new language.
       We should definitely avoid the use of tooltips to retranslate UI, and as you suggest in the short-term have a translate button temporarily, something similar to the image below. In the long-term, it seems essential to correctly handle language changes and present accurate information to the users.

• From: Aimini, Danilo

  I like the idea of picking the language early in the install / project management flow so we can set it before the Editor is started. I have a couple of technical notes related to Shirley’s questions:

  • Regarding the ability to switch language at runtime, it is theoretically possible. It would be a matter of defining a signal/bus, and have the UI widgets displaying labels listen to it and trigger a re-translation whenever that event is fired. Designs generated via UI files automatically fill in a retranslateUi function that can be used for this, so it sounds like it’s something Qt endorses.
    In practice, though, we would have to check whether it would be possible to embed such a refresh system directly into the UI Library components, so that all tools using them would just inherit the functionality without having to add repeated code ad-hoc in each tool. I’m sure some legacy tools like TrackView or the Lua Editor will still need some ad-hoc changes, but it may save us some time with the rest of the Editor. It is definitely the kind of change that could be worked on in development behind a flag, so we can test it piece by piece and only enable it by default when it's complete.
  • Regarding the tools/layout names being used in the backend code, I would guess it may still be possible to always use the English names in the backend as keys to the layout information, and only translate those keys in the instance they are displayed in the editor. I do agree that the underlying system should actually use a different key, to be fair, but a refactor of the entire system may increase the scope of the effort right now so I'd go for something less intrusive first, and maybe backlog a deeper re-architecture.
  • Finally, regarding a possible third language, I should be able to kickstart an Italian translation if it can help with testing the functionality. I know a few community members of the Discord would likely be interested in contributing the rest of the translations over time so it sounds like an easy win, and could provide the rest of the community with the proper info to also work on other languages.