Publish a Python client + types for the Project Engine API (transport parity with the TS client)

[rainer-friederich]

Summary

Provide a published, CI-verified Python client + types for the Project Engine API in @adobe/spacecat-shared-project-engine-client, at parity with the existing TypeScript client's transport behaviour. Today the package generates Pydantic models only — there is no Python transport (auth, retry, base-URL handling), the generated Python is not published so no other repo can depend on it, and it is not regenerated or drift-checked in CI. As a result Python consumers hand-roll the whole client, and at least one such hand-rolled client has a real correctness gap the shared TS transport already avoids.

What exists today (v1.10.0)

The generation pipeline lives in packages/spacecat-shared-project-engine-client:

• Source spec: spec/projectengine_swagger_public.yaml (Swagger 2.0, "Project Engine API" v1.0) → swagger2openapi → build/openapi3.json → overlay spec/overlays/corrections.yaml.
• generate:ts → src/generated/types.ts (openapi-typescript). The TS client (src/client.js, ~135 lines, plus src/internal.js) is a thin openapi-fetch wrapper over those types that OWNS the transport: auth middleware, method-aware retry, base-URL normalisation.
• generate:pydantic → python/serenity_project_engine/ (model.py, http_server.py, aiseo.py, __init__.py) via datamodel-codegen — Pydantic v2 models ONLY.

The gap

1. No Python transport client. datamodel-codegen emits data models, not a client. There is no Python equivalent of src/client.js — no Bearer-token auth injection, no retry/backoff, no base-URL handling. A Python consumer gets typed request/response shapes but must write and maintain all the HTTP plumbing itself.
2. The Python artifacts are not consumable. package.json files: ['src'] excludes python/, and there is no PyPI publish or git-branch distribution for it. So the committed Pydantic models cannot be installed or git+-depended-on by another repo today. (Contrast: mysticat-data-service distributes its own generated Python types via a types-py orphan git branch — a proven pattern in this org.)
3. No CI regeneration / drift-check. generate:pydantic requires Python and is deliberately kept manual (see the comments in .github/workflows/project-engine-mock-e2e.yaml and .github/workflows/user-manager-mock-e2e.yaml). The committed Python surface can therefore silently lag the spec and the TS types, with nothing to catch it — whereas generate:ts is verified in CI.

Motivating consumer (and a concrete defect it demonstrates)

mysticat-data-service scripts/serenity_migration/semrush_write.py defines SemrushWriteClient, a bespoke httpx client against this exact API (/enterprise/projects/api/..., IMS Bearer). It re-implements auth, retry/backoff, pagination, and the wire shapes by hand.

Its retry is not method-aware: _RETRY_STATUSES = {429, 502, 503, 504} are retried for every verb, including POST, with no Retry-After handling. A POST that returns 502 after the server already applied it gets replayed → a potential duplicate write (e.g. create_project, create_benchmarks). The TS transport avoids exactly this by retrying 5xx/network errors only for idempotent methods (see src/internal.js). A shared Python client would give every Python consumer that correctness for free instead of each re-deriving it (often wrong).

The dimension-root tag migration program is the immediate driver; this migration CLI is a living tool (--reconcile is used for ongoing customer onboarding), so removing its hand-rolled transport layer has lasting value, not one-shot.

Scope / tasks

• Add a thin Python transport client (over httpx) in this package that consumes the generated Pydantic models and replicates the TS client's cross-cutting behaviour (see "Transport contract" below).
• Decide and implement a distribution mechanism for the Python package (PyPI, or a types-py-style orphan git branch, or a git+https subdirectory dependency) and stop excluding it from what ships. Document how consumers depend on it.
• Wire generate:pydantic (and the new client) into CI as a drift-check — regenerate and fail on diff, mirroring how the TS surface is guarded — so the committed Python can't lag the spec.
• Provide minimal tests for the transport client (auth injection, method-aware retry incl. Retry-After, base-URL normalisation) and, ideally, an e2e smoke against the existing stateful mock (mock/, served under /enterprise/projects/api).
• Acceptance / first consumer: mysticat-data-service SemrushWriteClient is re-backed onto (or replaced by) the shared Python client for the Project Engine surface, dropping its hand-rolled wire + retry layer.

Transport contract the Python client must replicate

From the TS client (src/client.js, src/internal.js):

• Base URL: hard-code API_PREFIX = /enterprise/projects/api; from the caller's baseUrl keep only protocol//host (drop any path/credentials); reject non-http(s).
• Auth: Authorization: Bearer <token>; token source is a value or a callable (sync/async), resolved once per logical request and reused across that request's retries; fail fast on empty/invalid source. The raw IMS JWT is forwarded verbatim — no token exchange/minting.
• Retry: retry 429 for all methods; retry 5xx and network errors only for idempotent methods (GET/HEAD/PUT/DELETE/OPTIONS). Equal-jitter exponential backoff (base * 2**attempt * rand[0.5,1)), honour Retry-After (seconds or HTTP-date), clamp to 20000 ms. Defaults: maxRetries 2, base delay 200 ms. Replay bodied requests safely (per-attempt body handling). Best-effort onRetry hook.
• Model nuances: tag create vs read use different shapes (TreeNode* request vs AIOTag response); AIOTag.path is an ordered array of {id, name, parent_id} leaves in snake_case (not a string, no client-side parentId/dimension invention); prompts/tags/benchmarks/brand_urls live on v2 /aio/..., with some deprecated v1 equivalents.
• Error model: the TS client does not throw on HTTP errors (returns {data, error}). The Python client should pick a deliberate, documented error model (the existing SemrushWriteClient raises a typed SemrushError — a reasonable Pythonic choice; just make it intentional).

Explicitly out of scope (separate follow-ups, noted so no one conflates them)

• The dimension-root DOMAIN model is NOT in this package and this issue does not add it. Neither the TS nor the Python client encodes dimension = path[0], DIMENSION_ROOTS, the closed intent/source/type taxonomies, or buildTagsOf — those are re-implemented independently in spacecat-api-service, project-elmo-ui, and mysticat-data-service (scripts/serenity_migration/tags.py). Generating a Python client does not dedupe that taxonomy. A single source of truth for the taxonomy is a distinct, higher-value follow-up and should be tracked separately.
• Workspace provisioning is a different API. The migration CLI also drives workspace create/status/transfer/delete/rename on /enterprise/users/api/..., which belongs to @adobe/spacecat-shared-user-manager-client (which has its own generate:pydantic in the same state). A full de-hand-rolling of the migration CLI needs the equivalent Python client there too — companion work, same pattern.

Pointers

• Package: packages/spacecat-shared-project-engine-client (@adobe/spacecat-shared-project-engine-client, v1.10.0).
• Generation: package.json scripts spec:convert / spec:overlay / generate:ts / generate:pydantic / generate.
• TS transport to mirror: src/client.js, src/internal.js.
• Stateful mock for e2e: mock/ (Counterfact, /enterprise/projects/api prefix).
• Consumer to migrate: mysticat-data-service → scripts/serenity_migration/semrush_write.py (SemrushWriteClient).
• Companion package: @adobe/spacecat-shared-user-manager-client (same Python-client gap, for the workspace/users surface).

[rainer-friederich]

Companion issue for the TypeScript side (the domain-model half): #1806 — make the TS client a real domain client and establish the taxonomy single-source-of-truth this issue's Python client should mirror.

[rainer-friederich]

User Manager twin of this issue (the migration CLI's hand-rolled client spans both APIs, so both are needed to retire it): #1807