chore(release): provision Testnet staging and an end-to-end market smoke test

[Joaco2603]

Pre-flight checks

• I reviewed deployments, workflows, and open issues across all four repositories.
• No deployed Backend/UI environment or cross-repository smoke test currently proves the market journey.

Priority

P0 — release evidence and product demos currently depend on local state.

Problem

Backend and UI have no GitHub deployments, the Backend pipeline is red, and no automated environment proves wallet auth -> catalog -> purchase -> ledger confirmation -> encrypted delivery. Unit tests passing in isolation do not prove a product exists.

Proposed solution

Provision one production-like Testnet staging environment with:

• Backend, PostgreSQL migrations, encrypted object storage/KMS boundary, delivery worker, and health checks.
• UI configured through environment variables.
• Fresh verified Testnet contract IDs and funded test identities.
• A curated private prompt fixture created through the real publication boundary.
• Automated smoke coverage for an authorized buyer and negative unauthorized/replay cases.
• Redacted observability, rollback/reconciliation instructions, and secret rotation ownership.

Acceptance criteria

• A clean deploy applies migrations and starts with DB_SYNCHRONIZE=false.
• /api/health verifies database and required runtime dependencies, not only process liveness.
• The deployed UI authenticates a Freighter wallet against the deployed Backend.
• A buyer completes a real Testnet private-prompt purchase and retrieves an encrypted delivery result.
• Unauthorized access, duplicate confirmation, pending RPC, and replay cases fail safely.
• Deployment URLs, contract IDs, artifact hashes, and smoke-test evidence are recorded in a release runbook.
• No long-lived private key is committed or printed in CI logs.

Dependencies

• Backend feat(marketplace): add authenticated Testnet purchase-to-access MVP flow #9.
• Green Backend CI.
• Curated private-prompt publication.
• UI auth and purchase/delivery work.

[dostertags]

I can provision a production-like Testnet staging environment and automated smoke test suite to validate the full marketplace journey. This configures the deployed Backend, UI, PostgreSQL migrations, KMS boundaries, and health checks. It verifies the entire buyer flow—from Freighter wallet authentication to Stellar ledger confirmation and encrypted prompt delivery—while ensuring replay safeguards and generating release runbook evidence.

[Hollujay]

I would like to take on this issue, I will get it done efficiently as soon as possible.

[grantfox-oss]

🦊 GrantFox — @dostertags has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #17)
2. Your PR will be reviewed by the Stellar-AgentVerse maintainers

Good luck! Track your progress on GrantFox.

[dostertags]

Opened #20 for this — Refs #17, not Closes, because three criteria are still open.

I could not provision the environment itself: there is no hosting account a contributor
can reach, and neither Backend nor UI has a GitHub deployment. So I built the environment
locally against a real PostgreSQL 16.14 — the version ci runs — and tried to drive the
market journey through it. That is where the useful part came from, and it is the reason
this comment is long: a clean deploy provisioned by migrations boots, reports itself
healthy, and then fails every write.

Everything below I reproduced myself. If any of it is wrong, say so and I will correct it
rather than defend it.

The finding that motivated the rest

All fifteen uuid primary keys are created without a default. The entities use
@PrimaryGeneratedColumn('uuid'), which the TypeORM Postgres driver implements through the
column default rather than generating in the application, but every migration declares
"id" uuid PRIMARY KEY with nothing after it.

QueryFailedError: null value in column "id" of relation "assets" violates not-null constraint

AssetsService.create() never supplies an id (src/assets/assets.service.ts:54), so
POST /api/assets 500s on any migration-built database — and so does every other write path.

Two different reasons kept it hidden. docker-compose.yml ran with
DB_SYNCHRONIZE=true, where auto-synchronize adds the defaults on boot and has been
silently repairing what the migrations build wrong. And ci never touched a database at
all: npm test is rootDir: src, the e2e specs mock their data layer, and test:e2e is
not in the workflow — its postgres service has never had a query run against it. Criterion 1
is exactly the configuration that stops the first, and the new workflow closes the second.

Fixed in #20

Twelve findings, each reproduced first — eleven fixed, one reported only

#	Defect	How I confirmed it
1	All 15 uuid primary keys lack DEFAULT uuid_generate_v4()	information_schema query returns 15 uuid PKs with column_default IS NULL; a TypeORM save() with no id fails as above
2	asset_type_enum has 4 of the 6 values AssetType declares	Insert probe: PROMPT OK, MODEL and ORACLE → invalid input value for enum asset_type_enum
3	purchases."transactionHash" is varchar(128); the entity says varchar(64)	Reported, not changed. I narrowed it at first, then backed it out: ConfirmPurchaseDto accepts 32-128 characters with no hex or length check, so the narrower column turns a malformed hash into a 500 inside confirm(). Narrow it together with the DTO
4	data-source.ts imports the three delivery entities but omits them from entities	Read; a migration:generate would have planned DROP TABLE on delivery_commands, delivery_outbox, delivery_results
5	npm run build && npm run start:prod cannot start	The committed root-level demo-prompt.ts / test-kms.ts widen the TS root dir, so nest build emits dist/src/main.js while start:prod runs node dist/main → MODULE_NOT_FOUND
6	Nothing applies migrations at deploy	entrypoint.sh went from the pg_isready loop straight to exec node dist/main
7	The app cannot boot without STELLAR_ADMIN_SECRET_KEY	TokensService.validateConfig() warns the key is missing, then the next line calls Keypair.fromSecret(''), which throws. This is in direct tension with criterion 7
8	/api/health could not fail	It caught the DB error, set db: 'error', returned status: 'ok' / HTTP 200 anyway — its own spec asserted that behaviour
9	Compose builds its schema by auto-sync while declaring NODE_ENV=production	DB_SYNCHRONIZE: ${DB_SYNCHRONIZE:-true} next to NODE_ENV: ${NODE_ENV:-production}
10	The api container waits on MinIO	depends_on: minio: service_healthy, but grep -rniE "client-s3|S3Client" src/ returns 0 — nothing reads it, so the gate can only delay startup
11	AWS_KMS_KEY_ID is required in production and absent from .env.example	Filling in the example file still produced a container that aborts at bootstrap. Compose supplies only 9 of the 16 required variables itself
12	The README env table had 8 rows stranded below the marketplace section	An earlier edit split the table; they are back inside it

Found and deliberately not fixed

These are real and reproduced, but they belong to other issues or to decisions that are
yours. I would rather report them than quietly fix them inside a release PR.

Blocking criteria 4 and 5 — #9 and #15 territory

• Pending RPC is unsafe, and it is criterion 5's fourth case.
  purchases.service.ts:283-288 returns false for any status that is not SUCCESS,
  including PENDING and NOT_FOUND. :185-190 then writes FAILED, persists the
  transaction hash, and throws. The hash is now burned into the partial unique index, so the
  buyer can never retry that purchase. I did not assert this behaviour in the smoke suite,
  because asserting it would lock it in.
• Confirming a purchase never enqueues a delivery. acceptVerifiedEvent has exactly one
  caller, PromptExecutionService, and nothing in src/marketplace/ calls that. The
  purchase-to-delivery chain is not connected.
• No AI provider adapter is ever registered. AiProviderService exposes register()
  and holds an empty Map; there are zero .register( call sites in src/. Every delivery
  would end TERMINAL_FAILURE with PROVIDER_NOT_CONFIGURED.
• Nothing publishes an asset over HTTP. POST /api/assets creates a DRAFT
  (asset.entity.ts:44) and no route promotes it, while createIntent rejects anything not
  PUBLISHED. So the "curated private prompt fixture created through the real publication
  boundary" this issue asks for cannot exist yet. The smoke suite promotes its fixture
  directly in the database and says so in the file rather than dressing it up.
• The seed produces nothing purchasable. seed.ts sets status only on activity-log
  rows; assets are saved without one and default to DRAFT.

Together these are why I marked criterion 4 blocked rather than partially met.

Wallet sign-in: the signature scheme, and a 500 where a 401 belongs

AuthService.verifyWallet verifies a raw Ed25519 signature over the challenge bytes,
hex decoded:

keypair.verify(Buffer.from(entry.challenge, 'utf-8'), Buffer.from(signature, 'hex'))

A wallet that signs the SEP-53 payload instead — SHA-256("Stellar Signed Message: " + message),
returned base64 — cannot authenticate against that. I measured all three cases against a
running instance of this branch:

What the client sends	Result
Raw challenge signature, hex	200 + JWT
SEP-53 payload signature, hex	401 Invalid signature
SEP-53 payload signature, base64	500 Internal Server Error

The last row is a Backend defect on its own terms. Buffer.from(<base64>, 'hex') yields a
zero-length buffer, keypair.verify throws signature of length 64 expected, got 0, and
nothing catches it — so a client sending a well-formed-but-wrongly-encoded signature gets a
500 instead of a 401. The encoding half is bridgeable; the payload half is a hash preimage
and has to be handled server-side if SEP-53 wallets are the target.

Also worth noting for whoever wires the UI: verifyWallet deletes the challenge before
verifying, so every retry needs a fresh POST /api/auth/challenge.

I have not changed any of this — it is AuthModule, which #16 is in. Flagging it because
criterion 3 cannot be met until it is settled, and from the browser side it will look like a
UI bug.

Latent or operational — worth knowing before staging goes up

• token_transactions can never exist. src/indexer/entities/token-transaction.entity.ts:9
  declares the entity, but it appears in neither entity array and in no migration. Latent
  only because IndexerService has no HTTP caller; it would throw EntityMetadataNotFoundError
  on first use under any configuration.
• PROMPT_DELIVERY_WORKER_ENABLED is read raw from process.env with a strict
  === 'true' (prompt-delivery.worker.ts:43) and is absent from env validation. 1, TRUE
  and yes all silently leave the worker off — a staging deploy would stall at AUTHORIZED
  while looking healthy. The new /api/health now reports this flag for exactly that reason,
  but validating it belongs with the env work, which fix(wallet): authenticate wallet data and remove simulated production balances #16 is touching right now.
• Challenges are stored in memory. AuthModule wires InMemoryChallengeStore, so wallet
  authentication breaks the moment staging runs more than one instance: the challenge is
  issued by one replica and verified by another.
• MOCK_MODE can green a chainless run. NODE_ENV === 'test' plus an empty or
  PLACEHOLDER contract id makes verifyTransaction return true unconditionally. The
  smoke suite states this in its header and asserts the contract binding when a real contract
  is configured, so a configured environment cannot silently fall back to the mock builder.
• .env.development sets DB_DATABASE while env.validation.ts reads DB_NAME, so
  cp .env.development .env silently connects to the default agentverse database instead
  of agent_verse_stellar. One word; not a release PR's word.

CI — #14's, and why #20 does not touch it

npm run lint is eslint … --fix, so CI's lint step writes to the checkout. On a clean
checkout it still exits non-zero: 746 errors and 62 warnings measured without --fix.
429 of those errors are auto-fixable, so the 317 this issue's sibling #14 quotes is the
post---fix remainder the CI run reports, not the count on a clean tree. Lint runs first, so
Test and Build have been reported as skipped on every run of main.

ci.yml also sets DB_SYNCHRONIZE: true, which is precisely why none of the schema defects
above were ever caught there.

I left all of it alone. #20 adds a separate staging-smoke workflow instead, so release
evidence is not gated on a pipeline that belongs to another issue. #20 reduces repo-wide lint
to 739/61 as a side effect of its own files being clean; it does not try to close the gap.

Findings already reported in #19 — the unauthenticated creator parameter on
/api/dashboard/*, POST /api/payments returning 201 for a failed payment, the unread
S3_* and PROMPT_CONTENT_ENCRYPTION_KEY entries in .env.example, and the unreachable
stripe.service.ts / paypal.service.ts — I have not repeated here.

Criteria

#	Status	Why
1	Met, proven in CI	Migrations apply, DB_SYNCHRONIZE=false, the compiled app boots and serves. Not proven on a deployed host, because there isn't one
2	Met	/api/health reports database, applied migration, Soroban RPC, marketplace contract and delivery worker, and answers 503 when a required one is down. /api/health/live is the liveness probe
3	Blocked	No deployed UI, and the signature scheme does not line up — see the auth note below. The Backend half of the handshake is covered by the smoke suite: real Ed25519 signatures, wrong-signer and replayed-challenge both rejected
4	Blocked	See the #9/#15 section above. The suite marks the real-settlement case todo naming its blockers, rather than asserting a substitute for it
5	3 of 4	Unauthorized, duplicate confirmation and replay are asserted with exact status codes and messages. Pending RPC is not, for the reason given above
6	Structure met, values blocked	docs/release-runbook.md has the 16-variable environment contract, the deploy/verify/rollback procedure and secret ownership. URLs, contract ids and hashes are empty TODO(maintainer) fields next to the command that produces each. I did not invent any
7	Met	Nothing secret is committed. The workflow sets no signing key — it generates a throwaway keypair inside one step, never writes it to a file or $GITHUB_ENV, and never signs with it. A final step fails the run if a Stellar secret seed appears in a tracked file or the captured log

One more round after I first pushed

I ran an independent read-only review over the pushed branch before writing this, and it
found four defects in my own work — worth naming, since three of them were claims rather
than code. My docker-compose.yml default of DB_SYNCHRONIZE=false was inert, because
Compose interpolates the same .env the service loads and .env.example still said
true; the readiness check returned a permanent 503 on any synchronize-owned deployment,
including the README's own local flow; the transactionHash narrowing was unsafe as
described above; and the health probes were unbounded, so a partitioned database would have
made the endpoint hang rather than answer 503. All four are fixed in the branch, and the
workflow is green again on the same commit.

Evidence

The new workflow is green on a real runner, all 13 steps, first attempt. I ran it in my fork
because this repo holds workflows from first-time contributors:
https://github.com/dostertags/Backend/actions/runs/33278950877

I also checked that the suite is capable of failing, since a smoke suite that passes
vacuously is worse than none: four mutations — dropping assets from the uuid-default list,
dropping ORACLE from the enum backfill, changing the replay guard's message, and making
/api/health always answer 200 — each turned red exactly the assertion that claims to cover
it. Withholding the new migration entirely turns all 18 red on the null value in column "id"
error.

What I need from you

• Approve the workflow run on chore(release): provision Testnet staging and prove the market journey #20 so its checks execute here. The fork run above is the same commit.
• A deploy target and its secrets, to turn criterion 1's CI proof into a real environment and
  fill the runbook's TODO(maintainer) rows.
• A call on whether the pending-RPC fix belongs in feat(marketplace): add authenticated Testnet purchase-to-access MVP flow #9 or should come back here.

Happy to split #20 into stacked PRs — the seven commits are ordered for it — and happy to be
told any of the above is wrong.