[Design Discussion] i18n support

[ThumulaPerera]

Related Feature Issue

#876

Problem Summary

Thunder backend needs internationalization (i18n) support to:

1. Return translatable error messages and API responses
2. Allow org administrators to manage translations without code changes
3. Support multiple languages with fallback to defaults

Currently, all user-facing strings are hardcoded in English.

Proposed Approach

Overview

• Thunder backend developers define messages inline using i18n.Message{Key, DefaultValue}
• A build-time extraction script scans source code and generates a Go file with all defaults
• Database stores language specific translations
• I18n Bundle API merges embedded defaults with database translations
• API responses include an optional i18n metadata section with translation keys
• Frontend fetches bundles and translates using keys from the i18n section

Backward Compatible Response Format

All API responses (errors and non-errors) use an optional i18n metadata section:

Error response:

{
    "code": "USR-1001",
    "message": "User not found",
    "description": "The user with the specified ID does not exist",
    "i18n": {
        "message": "error.user.not_found",
        "description": "error.user.not_found.description"
    }
}

Non-error response:

{
    "id": "schema-123",
    "name": "Employee",
    "displayName": "Employee Schema",
    "i18n": {
        "displayName": "userschema.employee.display_name"
    }
}

Key benefits:

• Original fields remain unchanged (strings) - no breaking change
• i18n section mirrors parent structure (same field names)
• Fields not needing translation are omitted from i18n
• Old clients ignore i18n section, new clients use it for translation
• Modules can be migrated incrementally

How It Works

1. Developer defines an i18n message:

var ErrorUserNotFound = serviceerror.NewI18nError(
    serviceerror.ClientErrorType,
    "USR-1001",
    i18n.Message{
        Key:          "error.user.not_found",
        DefaultValue: "User not found",
    },
    i18n.Message{
        Key:          "error.user.not_found.description",
        DefaultValue: "The user with the specified ID does not exist",
    },
)

2. Build-time extraction generates defaults.go:

make generate-i18n

This runs an AST parser that finds all i18n.Message{} literals and generates:

// Code generated by i18n-extract. DO NOT EDIT.
package i18n

var DefaultMessages = map[string]string{
    "error.user.not_found":             "User not found",
    "error.user.not_found.description": "The user with the specified ID does not exist",
}

3. Handler builds response with i18n metadata:

errResp := apierror.ErrorResponse{
    Code:        svcErr.Code,
    Message:     svcErr.Error,
    Description: svcErr.ErrorDescription,
}
if svcErr.ErrorKey != "" {
    errResp.I18n = &apierror.I18nKeys{
        Message:     svcErr.ErrorKey,
        Description: svcErr.ErrorDescriptionKey,
    }
}

4. Frontend fetches language bundle and translates keys from i18n section.

Bundle Generation Logic

GetBundle(language) {
    1. Start with all keys from embedded defaults (English)
    2. Query DB for translations in requested language
    3. Overlay: DB values replace defaults where present
    4. Return merged bundle
}

Technical Questions for Discussion

1. i18n Key Format

With namespacing:

Option A - Function style:

{{ t(namespace:domain.hierarchy) }}

example : {{ t(registration:username.label) }}

Option B - Object style:

{{ i18n.namespace:domain.hierarchy }}

example : {{ i18n.registration:username.label }}

Without namespacing:

Option C - Function style:

{{ t(domain.hierarchy) }}

example : {{ t(registration.username.label) }}

Option D - Object style:

{{ i18n.domain.hierarchy }}

example : {{ i18n.registration.username.label }}

Questions:

• Is there a use case that requires namespacing and cannot be handled by domain hierarchy alone?
• Which key style should we use?

---

2. Templating Support in i18n Messages

Should we support variable interpolation in messages?

Example:

i18n.Message{
    Key:          "error.user.age.invalid",
    DefaultValue: "Age must be between {{min}} and {{max}}",
    Params: {
        min: "10",
        max: "18"
    }
}

Bundle response:

{
    "error.user.age.invalid": "Age must be between {{min}} and {{max}}"
}

API response:

{
    "message": "Age must be between 10 and 18",
    "i18n": {
        "message": {
            "key": "error.user.age.invalid",
            "params": {
                "min": "10",
                "max": "18"
            }
        }
    }
}

Frontend usage:

// read i18n key
// get translated message corresponding for i18n key (from the bundle)
// replace `{{ }}` values with corresponding param value

Questions:

• Do we need first class support for pluralizations? (e.g., "1 item" vs "5 items")
  • In most cases backend can programmatically handle this without any additional support as it can figure out whether message needs to be plural or not.

---

3. Custom Keys Support

Should we allow administrators to create translation keys that don't exist in the codebase?

Questions:

• What are the use cases for custom keys ?

Alternative Approaches

1. Server-Side Translation

• Server translates messages before sending response
• Requires Accept-Language header / query param processing

2. Translation Files (JSON/YAML)

• Store translations in static files instead of database

3. Alternative approach for bundle API

• Instead of a separate API, consolidate this with the branding resolver response

Architecture and Components

Package Structure

internal/system/i18n/       # Core infrastructure
├── message.go              # Message struct
├── constants.go            # DefaultLanguage = "en"
├── defaults.go             # Auto-generated defaults map
└── bundle.go               # BundleService (merges defaults + DB)

internal/i18n/              # Translation Management API
├── model.go                # Request/Response DTOs
├── handler.go              # HTTP handlers
├── service.go              # Business logic
├── store.go                # Database operations
└── init.go                 # Route registration

scripts/i18n-extract/       # Build-time extraction tool
├── main.go
├── extractor.go            # AST parser
└── generator.go            # defaults.go generator

Database Schema

New table to store translations in thunderdb:

API Endpoints

Method	Endpoint	Description
GET	/i18n/bundle/{language}	Get merged bundle for frontend (with domain/namespace based filtering)
GET	/translations	List all translations (paginated)
POST	/translations	Create translation
GET	/translations/{key}/{language}	Get single translation
PUT	/translations/{key}/{language}	Update translation
DELETE	/translations/{key}/{language}	Delete translation
POST	/translations/bulk	Bulk upsert translations

Security Considerations

1. Input Sanitization: All translation values should be sanitized before storage to prevent XSS
2. Access Control: Translation management APIs should require admin privileges
3. Key Validation: Message keys are validated against allowed patterns
4. Language Code Validation: Only valid ISO 639-1 codes accepted (e.g., en, es, fr-CA)

Implementation Complexity

Medium (3-6 weeks)

Areas of Thunder that will be impacted

System-wide

Questions for Community Input

1. Finalize key format (namespacing vs domain hierarchy)
2. Decide on templating support
3. Decide on custom keys support

[ThumulaPerera]

Thinking further about i18n section mirrors parent structure (same field names), rather than having a single (possibly nested) top level i18n metadata object, having flat i18n metadata object at each nesting level may be better.

{
    "id": "prompt_credentials",
    "type": "PROMPT",
    "meta": {
        "components": [
            {
                "type": "TEXT",
                "id": "text_001",
                "label": "sign in",
                "variant": "HEADING_01"
            },
            ...
    },
    ...
    "i18n": {
          "meta": {
              "components": [{
                  "label": "signin:heading.label"
              }]
          }
    }
} 

Is harder to deal with on the frontend than

{
    "id": "prompt_credentials",
    "type": "PROMPT",
    "meta": {
        "components": [
            {
                "type": "TEXT",
                "id": "text_001",
                "label": "sign in",
                "variant": "HEADING_01"
                "i18n": {
                      "label": "signin:heading.label"      
                }
            },
            ...
    },
    ...
}

[senthalan]

The overall design looks great.

Find my input for the Technical Questions

1. i18n Key Format

I think with namespacing will be more clear. Then I would expect this namespacing seperation will be API request and response as well. So that it can be easily managed.

For the Function style vs Object style, we need to align with the discussion in #812 (comment)

2. Templating Support in i18n Messages
   Do we need first class support for pluralizations? (e.g., "1 item" vs "5 items")

yes, we need to support it. As the saying goes in software design, “There are only three numbers: 0, 1, and many.”. So we should support pluralizations

3. Custom Keys Support

This is something nice to have. The use case might be if anyone want to use Thunder to manage their application content localisation too.

---

Regarding the Package Structure, can't we have Translation Management API also within the system package ? (as core and mgt sub packages)

We need to make sure that the scripts/i18n-extract/ files are not included in the final build binary or zip distribution. Also shall we rethink on this directory structure. I don't think naming this as scripts suites well.

[ThumulaPerera]

Implementation Update

Current Status

A base version of the i18n support has been implemented in PR #940.

Note: Templating and pluralization support are not yet implemented.

Implementation Deviations

The implementation deviates slightly from the original proposal in the following areas:

1. API Endpoints:
   Endpoints were renamed for better clarity. Reference: api/i18n.yaml.

2. System i18n Response Format:
   Each i18n message has been standardised to include both a key and default value, rather than having a separate i18n field.

   {
       "code": "USR-1001",
       "message": {
           "key": "error.user.not_found",
           "defaultValue": "User not found"
       },
       "description": {
           "key": "error.user.not_found.description",
           "defaultValue": "The user with the specified ID does not exist"
       }
   }

---

Message Categories

During implementation, we identified three distinct types of i18n messages:

1. Backend compile-time defined (e.g., System API error descriptions)
2. Frontend compile-time defined (e.g., Copyright notices in the Gate app)
3. Runtime defined (e.g., Display name of an application)

Handling Strategy:

• Types 1 & 2: Default English translations are defined in the code by the developer.
• Type 3: Translations are added by the Thunder organization admin at runtime.

Note: While the initial design prioritized frontend-side translation, Type 1 messages exist even when interacting with the backend purely via API. Therefore, the API must support returning default translations for these messages.

---

Ease of Adoption by Message Type

1. Backend Compile-time Messages

Status: ✅ Fully Covered

• Definition: Developers define keys and default values directly in the backend code.
• Extraction: A script extracts these keys/defaults into a Go constants file at compile time.
• Fallback: Values from the constants file (baked into the binary) are used if no translation is found in the DB.
• Management: Org admins can populate specific translations in the DB.
• Response: The API returns both the i18n key and the default translation.

2. Frontend Compile-time Messages

Status: ⚠️ Improvement Needed

• Definition: Developers define keys and defaults in the frontend code (via react-i18next).
• Extraction: Keys are extracted to a constants file at compile time (via i18next-cli).
• Missing Piece: A mechanism to populate these constants into the DB during setup (or via immutable configs).
• Management: Org admins can populate translations in the DB.
• Network: These messages are resolved client-side and do not traverse the network.

3. Runtime Messages

Status: ⚠️ Supported with Caveat

• Definition: Org admins use translatable string syntax in resource create/update requests.

  {
      "appName": "t{custom:app_1_name}"
  }

• Management: Org admins populate the translations in the DB.
• Caveat: No default translation is sent in the response. The API returns the translatable string key, and the client is responsible for resolution (or displaying the key/placeholder if missing).