feat(cli): python (#1088)

add python registration support: `t('hello, {name}', name='john',
_max_chars=10)`, also supports `declare_static()`/`declare_var()` syntax

<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR adds a new `@generaltranslation/python-extractor` package and
wires it into the CLI, enabling translation string registration for
Python projects using `gt-flask` and `gt-fastapi`. The extractor uses
`tree-sitter-python` to parse `.py` files and extract `t()` / `msg()`
calls (including `declare_static()` / `declare_var()` compound
expressions) into the same `ExtractionResult` format used by the
existing React pipeline.

Key changes:
- New `packages/python-extractor` package with a full AST-based
extraction pipeline (`extractImports` → `extractCalls` →
`parseStringExpression` → `resolveFunctionVariants`).
- CLI framework detection extended to check `pyproject.toml`,
`requirements.txt`, and `setup.py` for `gt-flask` / `gt-fastapi`
dependencies.
- Shared `postProcess.ts` utilities (`calculateHashes`, `dedupeUpdates`,
`linkStaticUpdates`) extracted from the React pipeline so both JS and
Python extraction paths share the same deduplication and static-ID
linking logic.
- `PythonLibrary` type and `isPythonLibrary` guard added;
`translation/parse.ts` routes Python libraries to the new
`createPythonInlineUpdates` pipeline.
- **Issue:** In `createPythonInlineUpdates.ts`, `fs.promises.readFile`
is called outside the per-file `try/catch`, so file-read errors
(permissions, broken symlinks) will abort the entire extraction loop
rather than being collected in the `errors` array.
- **Issue:** The module-level `crossFileCache` in
`resolveFunctionVariants.ts` keys by `filePath::functionName` but does
not include calling-context import aliases; two callers with different
`declare_static` aliases for the same helper will share a stale cache
entry.
- **Minor:** `linkStaticUpdates` in `postProcess.ts` calls `.sort()` on
the mapped hash array without filtering out potential `undefined`
values.

<details open><summary><h3>Confidence Score: 3/5</h3></summary>

- Safe to merge after fixing the uncaught file-read error in the Python
extraction pipeline; the logic and cross-file cache issues are edge
cases but worth addressing.
- The core extraction logic is sound and well-tested, but the
`fs.promises.readFile` call outside the try/catch in
`createPythonInlineUpdates.ts` is a real bug that can crash the CLI on
permission errors or broken symlinks. The cross-file cache issue in
`resolveFunctionVariants.ts` is an edge case that would only surface
when two calling files use different aliases for
`declare_static`/`declare_var`. The sort-on-undefined in
`postProcess.ts` is a minor type safety concern. Overall the feature is
substantial and correct for the common case, but the file-read bug
should be fixed before shipping.
- packages/cli/src/python/parse/createPythonInlineUpdates.ts (uncaught
file-read error),
packages/python-extractor/src/resolveFunctionVariants.ts (cache key
ignores import context)
</details>


<details><summary><h3>Important Files Changed</h3></summary>




| Filename | Overview |
|----------|----------|
| packages/python-extractor/src/index.ts | New entry point for the
python-extractor package; orchestrates parsing, import extraction, and
call extraction cleanly; cache-clear exports are absent from this file
(noted in prior threads). |
| packages/python-extractor/src/extractCalls.ts | Extracts and validates
t()/msg() calls from a Python AST; routes compound expressions
(f-strings, binary concat, declare_static/declare_var) through
parseStringExpression; straightforward and well-covered by tests. |
| packages/python-extractor/src/parseStringExpression.ts | Core
recursive parser that converts Python string expressions into StringNode
trees; handles f-strings, binary concat, ternary, and
declare_static/declare_var calls; uses extractImports for GT-package
filtering in extractImportsFromRoot, addressing prior concerns. |
| packages/python-extractor/src/resolveFunctionVariants.ts | Resolves
helper function return variants across files; module-level
crossFileCache key (filePath::functionName) does not encode
calling-context import aliases, which could produce stale results when
two callers use different declare_static/declare_var aliases for the
same helper file. |
| packages/cli/src/python/parse/createPythonInlineUpdates.ts | Python
extraction pipeline for the CLI; fs.promises.readFile is called outside
the per-file try/catch, so file-read errors (permissions, broken
symlinks) will propagate as uncaught exceptions instead of being
collected in the errors array. |
| packages/cli/src/extraction/postProcess.ts | Shared post-processing
utilities (hashing, dedup, static-ID linking) extracted from the React
pipeline; linkStaticUpdates sorts a potentially undefined hash array
without guarding against undefined values. |
| packages/cli/src/fs/determineFramework/matchSetupPyDependency.ts |
Parses setup.py install_requires/extras_require blocks for GT
dependencies; correctly handles string boundaries and nested brackets;
escaped-backslash fix from prior thread is present. |
| packages/cli/src/translation/parse.ts | Routes Python libraries to
createPythonInlineUpdates and React/JS libraries to createInlineUpdates
using the new isPythonLibrary guard. |

</details>


</details>


<details><summary><h3>Sequence Diagram</h3></summary>

```mermaid
sequenceDiagram
    participant CLI as CLI (translation/parse.ts)
    participant PY as createPythonInlineUpdates
    participant EXT as extractFromPythonSource
    participant IMP as extractImports
    participant CALLS as extractCalls
    participant PSE as parseStringExpression
    participant RFV as resolveFunctionVariants
    participant RI as resolveImport
    participant POST as postProcess

    CLI->>PY: isPythonLibrary(pkg) → true
    PY->>PY: matchFiles(cwd, patterns)
    loop each .py file
        PY->>EXT: extractFromPythonSource(code, filePath)
        EXT->>IMP: extractImports(rootNode)
        IMP-->>EXT: ImportAlias[]
        EXT->>CALLS: extractCalls(rootNode, imports, filePath)
        CALLS->>PSE: parseStringExpression(firstArg, ctx)
        alt declare_static / declare_var
            PSE->>RFV: resolveFunctionInCurrentFile / resolveFunctionInFile
            RFV->>RI: resolveImportPath(moduleName, filePath)
            RI-->>RFV: resolved file path
            RFV-->>PSE: StringNode
        end
        PSE-->>CALLS: StringNode
        CALLS-->>EXT: RawTranslationCall[]
        EXT-->>PY: ExtractionResult[]
    end
    PY->>POST: calculateHashes(updates)
    PY->>POST: dedupeUpdates(updates)
    PY->>POST: linkStaticUpdates(updates)
    PY-->>CLI: { updates, errors, warnings }
```
</details>


<!-- greptile_failed_comments -->
<details><summary><h3>Comments Outside Diff (1)</h3></summary>

1. `python-extractor-detailed-plan.md`, line 1
([link](https://github.com/generaltranslation/gt/blob/f98045879fbee352b039b80cbb96472e0e661862/python-extractor-detailed-plan.md#L1))

This detailed plan document is committed to the repo root and is now
stale. Its "Phase 2 (Future)" section lists `declare_static`,
`declare_var`, and `msg()` support as not yet implemented, but this PR
implements all three features.

Committed design documents tend to drift out of sync and can mislead
future contributors. Consider removing this file or relocating it to a
`docs/` or `.github/` directory if you want to preserve it as a living
design reference.

   <details><summary>Prompt To Fix With AI</summary>

   `````markdown
   This is a comment left during a code review.
   Path: python-extractor-detailed-plan.md
   Line: 1

   Comment:
This detailed plan document is committed to the repo root and is now
stale. Its "Phase 2 (Future)" section lists `declare_static`,
`declare_var`, and `msg()` support as not yet implemented, but this PR
implements all three features.

Committed design documents tend to drift out of sync and can mislead
future contributors. Consider removing this file or relocating it to a
`docs/` or `.github/` directory if you want to preserve it as a living
design reference.

   How can I resolve this? If you propose a fix, please make it concise.
   `````
   </details>
</details>

<!-- /greptile_failed_comments -->

<sub>Last reviewed commit: 57ce390</sub>

> Greptile also left **3 inline comments** on this PR.

<!-- /greptile_comment -->

Author: ErnestM1234

.changeset/wide-mirrors-invent.md

@@ -0,0 +1,6 @@
+---
+'@generaltranslation/python-extractor': minor
+'gt': minor
+---
+
+feat: add python support for registration

packages/cli/package.json

@@ -117,6 +117,7 @@
     "esbuild": "^0.27.2",
     "fast-glob": "^3.3.3",
     "fast-json-stable-stringify": "^2.1.0",
+    "@generaltranslation/python-extractor": "workspace:*",
     "generaltranslation": "workspace:*",
     "gt-remark": "workspace:*",
     "html-entities": "^2.6.0",
@@ -132,6 +133,7 @@
     "remark-parse": "^11.0.0",
     "remark-stringify": "^11.0.0",
     "resolve": "^1.22.10",
+    "smol-toml": "^1.3.1",
     "tsconfig-paths": "^4.2.0",
     "unified": "^11.0.5",
     "unist-util-visit": "^5.0.0",

packages/cli/src/cli/base.ts

@@ -66,7 +66,7 @@ import {
   getAgentInstructions,
   appendAgentInstructions,
 } from '../setup/agentInstructions.js';
-import { determineLibrary } from '../fs/determineFramework.js';
+import { determineLibrary } from '../fs/determineFramework/index.js';
 import { INLINE_LIBRARIES } from '../types/libraries.js';
 import { handleEnqueue } from './commands/enqueue.js';
 

packages/cli/src/cli/inline.ts

@@ -193,10 +193,14 @@ function fallbackToGtReact(library: SupportedLibraries): InlineLibrary {
     Libraries.GT_NEXT,
     Libraries.GT_NODE,
     Libraries.GT_REACT_NATIVE,
+    Libraries.GT_FLASK,
+    Libraries.GT_FASTAPI,
   ].includes(library as Libraries)
     ? (library as
         | typeof Libraries.GT_NEXT
         | typeof Libraries.GT_NODE
-        | typeof Libraries.GT_REACT_NATIVE)
+        | typeof Libraries.GT_REACT_NATIVE
+        | typeof Libraries.GT_FLASK
+        | typeof Libraries.GT_FASTAPI)
     : Libraries.GT_REACT;
 }

packages/cli/src/cli/python.ts

@@ -0,0 +1,17 @@
+import { Command } from 'commander';
+import { SupportedLibraries } from '../types/index.js';
+import { InlineCLI } from './inline.js';
+import { PythonLibrary } from '../types/libraries.js';
+
+/**
+ * CLI tool for managing translations with gt-flask and gt-fastapi
+ */
+export class PythonCLI extends InlineCLI {
+  constructor(
+    command: Command,
+    library: PythonLibrary,
+    additionalModules?: SupportedLibraries[]
+  ) {
+    super(command, library, additionalModules);
+  }
+}

packages/cli/src/config/generateSettings.ts

@@ -32,6 +32,17 @@ export const DEFAULT_SRC_PATTERNS = [
   'components/**/*.{js,jsx,ts,tsx}',
 ];
 
+export const DEFAULT_PYTHON_SRC_PATTERNS = ['**/*.py'];
+export const DEFAULT_PYTHON_SRC_EXCLUDES = [
+  'venv/**',
+  '.venv/**',
+  '__pycache__/**',
+  '**/migrations/**',
+  '**/tests/**',
+  '**/test_*.py',
+  '**/*_test.py',
+];
+
 /**
  * Generates settings from any
  * @param flags - The CLI flags to generate settings from
@@ -158,8 +169,8 @@ export async function generateSettings(
   // Add publish if not provided
   mergedOptions.publish = (gtConfig.publish || flags.publish) ?? false;
 
-  // Populate src if not provided
-  mergedOptions.src = mergedOptions.src || DEFAULT_SRC_PATTERNS;
+  // Don't default src here — each pipeline (JS/Python) has its own defaults.
+  // Only set src if the user explicitly provided it via flags or config.
 
   // Resolve all glob patterns in the files object
   const compositePatterns = [

packages/cli/src/extraction/__tests__/mapToUpdates.test.ts

@@ -0,0 +1,111 @@
+import { describe, it, expect } from 'vitest';
+import { mapExtractionResultsToUpdates } from '../mapToUpdates.js';
+import type { ExtractionResult } from '@generaltranslation/python-extractor';
+
+describe('mapExtractionResultsToUpdates', () => {
+  it('maps empty results to empty updates', () => {
+    const updates = mapExtractionResultsToUpdates([]);
+    expect(updates).toEqual([]);
+  });
+
+  it('maps single result with all metadata fields', () => {
+    const results: ExtractionResult[] = [
+      {
+        dataFormat: 'ICU',
+        source: 'Hello, {name}!',
+        metadata: {
+          id: 'greeting',
+          context: 'casual',
+          maxChars: 100,
+          filePaths: ['app.py'],
+          staticId: 'static-1',
+        },
+      },
+    ];
+
+    const updates = mapExtractionResultsToUpdates(results);
+
+    expect(updates).toHaveLength(1);
+    expect(updates[0]).toEqual({
+      dataFormat: 'ICU',
+      source: 'Hello, {name}!',
+      metadata: {
+        id: 'greeting',
+        context: 'casual',
+        maxChars: 100,
+        filePaths: ['app.py'],
+        staticId: 'static-1',
+      },
+    });
+  });
+
+  it('passes through dataFormat correctly', () => {
+    const results: ExtractionResult[] = [
+      {
+        dataFormat: 'JSX',
+        source: '<p>Hello</p>',
+        metadata: {},
+      },
+    ];
+
+    const updates = mapExtractionResultsToUpdates(results);
+    expect(updates[0].dataFormat).toBe('JSX');
+  });
+
+  it('handles missing optional metadata', () => {
+    const results: ExtractionResult[] = [
+      {
+        dataFormat: 'ICU',
+        source: 'Simple string',
+        metadata: {},
+      },
+    ];
+
+    const updates = mapExtractionResultsToUpdates(results);
+
+    expect(updates).toHaveLength(1);
+    expect(updates[0].metadata).toEqual({});
+    expect(updates[0].metadata.id).toBeUndefined();
+    expect(updates[0].metadata.context).toBeUndefined();
+    expect(updates[0].metadata.maxChars).toBeUndefined();
+  });
+
+  it('preserves filePaths array', () => {
+    const results: ExtractionResult[] = [
+      {
+        dataFormat: 'ICU',
+        source: 'Multi-file string',
+        metadata: {
+          filePaths: ['routes/index.py', 'routes/auth.py'],
+        },
+      },
+    ];
+
+    const updates = mapExtractionResultsToUpdates(results);
+    expect(updates[0].metadata.filePaths).toEqual([
+      'routes/index.py',
+      'routes/auth.py',
+    ]);
+  });
+
+  it('maps multiple results', () => {
+    const results: ExtractionResult[] = [
+      {
+        dataFormat: 'ICU',
+        source: 'Hello',
+        metadata: { id: 'hello' },
+      },
+      {
+        dataFormat: 'ICU',
+        source: 'Goodbye',
+        metadata: { id: 'goodbye', context: 'farewell' },
+      },
+    ];
+
+    const updates = mapExtractionResultsToUpdates(results);
+    expect(updates).toHaveLength(2);
+    expect(updates[0].source).toBe('Hello');
+    expect(updates[1].source).toBe('Goodbye');
+    expect(updates[1].metadata.context).toBe('farewell');
+  });
+});

packages/cli/src/extraction/__tests__/postProcess.test.ts

@@ -0,0 +1,125 @@
+import { describe, it, expect } from 'vitest';
+import {
+  calculateHashes,
+  dedupeUpdates,
+  linkStaticUpdates,
+} from '../postProcess.js';
+import type { Updates } from '../../types/index.js';
+
+describe('calculateHashes', () => {
+  it('generates consistent hashes for same input', async () => {
+    const updates1: Updates = [
+      { dataFormat: 'ICU', source: 'hello', metadata: {} },
+    ];
+    const updates2: Updates = [
+      { dataFormat: 'ICU', source: 'hello', metadata: {} },
+    ];
+
+    await calculateHashes(updates1);
+    await calculateHashes(updates2);
+
+    expect(updates1[0].metadata.hash).toBeDefined();
+    expect(updates1[0].metadata.hash).toBe(updates2[0].metadata.hash);
+  });
+
+  it('generates different hashes for different sources', async () => {
+    const updates: Updates = [
+      { dataFormat: 'ICU', source: 'hello', metadata: {} },
+      { dataFormat: 'ICU', source: 'world', metadata: {} },
+    ];
+
+    await calculateHashes(updates);
+
+    expect(updates[0].metadata.hash).not.toBe(updates[1].metadata.hash);
+  });
+});
+
+describe('dedupeUpdates', () => {
+  it('removes duplicates with same hash, merges filePaths', () => {
+    const updates: Updates = [
+      {
+        dataFormat: 'ICU',
+        source: 'hello',
+        metadata: { hash: 'h1', filePaths: ['pathA'] },
+      },
+      {
+        dataFormat: 'ICU',
+        source: 'hello',
+        metadata: { hash: 'h1', filePaths: ['pathB'] },
+      },
+    ];
+
+    dedupeUpdates(updates);
+
+    expect(updates).toHaveLength(1);
+    expect(updates[0].metadata.filePaths).toEqual(['pathA', 'pathB']);
+  });
+
+  it('keeps distinct entries with different hashes', () => {
+    const updates: Updates = [
+      {
+        dataFormat: 'ICU',
+        source: 'hello',
+        metadata: { hash: 'h1', filePaths: ['pathA'] },
+      },
+      {
+        dataFormat: 'ICU',
+        source: 'world',
+        metadata: { hash: 'h2', filePaths: ['pathB'] },
+      },
+    ];
+
+    dedupeUpdates(updates);
+
+    expect(updates).toHaveLength(2);
+  });
+
+  it('handles entries without hashes', () => {
+    const updates: Updates = [
+      { dataFormat: 'ICU', source: 'no-hash', metadata: {} },
+      {
+        dataFormat: 'ICU',
+        source: 'has-hash',
+        metadata: { hash: 'h1', filePaths: ['pathA'] },
+      },
+    ];
+
+    dedupeUpdates(updates);
+
+    expect(updates).toHaveLength(2);
+  });
+});
+
+describe('linkStaticUpdates', () => {
+  it('groups entries by temporary staticId and assigns shared hash', () => {
+    const updates: Updates = [
+      {
+        dataFormat: 'ICU',
+        source: 'variant-a',
+        metadata: { hash: 'ha', staticId: 'temp-static' },
+      },
+      {
+        dataFormat: 'ICU',
+        source: 'variant-b',
+        metadata: { hash: 'hb', staticId: 'temp-static' },
+      },
+    ];
+
+    linkStaticUpdates(updates);
+
+    // Both should now share the same staticId (derived from their hashes)
+    expect(updates[0].metadata.staticId).toBe(updates[1].metadata.staticId);
+    // The staticId should have been replaced (no longer the temporary value)
+    expect(updates[0].metadata.staticId).not.toBe('temp-static');
+  });
+
+  it('does not modify entries without staticId', () => {
+    const updates: Updates = [
+      { dataFormat: 'ICU', source: 'no-static', metadata: { hash: 'h1' } },
+    ];
+
+    linkStaticUpdates(updates);
+
+    expect(updates[0].metadata.staticId).toBeUndefined();
+  });
+});

packages/cli/src/extraction/index.ts

@@ -0,0 +1,7 @@
+export type { ExtractionResult, ExtractionMetadata } from './types.js';
+export { mapExtractionResultsToUpdates } from './mapToUpdates.js';
+export {
+  calculateHashes,
+  dedupeUpdates,
+  linkStaticUpdates,
+} from './postProcess.js';

packages/cli/src/extraction/mapToUpdates.ts

@@ -0,0 +1,25 @@
+import type { ExtractionResult } from '@generaltranslation/python-extractor';
+import type { Updates } from '../types/index.js';
+
+/**
+ * Maps ExtractionResult[] to Updates[] format used by the CLI pipeline
+ */
+export function mapExtractionResultsToUpdates(
+  results: ExtractionResult[]
+): Updates {
+  return results.map((result) => ({
+    dataFormat: result.dataFormat,
+    source: result.source,
+    metadata: {
+      ...(result.metadata.id && { id: result.metadata.id }),
+      ...(result.metadata.context && { context: result.metadata.context }),
+      ...(result.metadata.maxChars != null && {
+        maxChars: result.metadata.maxChars,
+      }),
+      ...(result.metadata.filePaths && {
+        filePaths: result.metadata.filePaths,
+      }),
+      ...(result.metadata.staticId && { staticId: result.metadata.staticId }),
+    },
+  }));
+}