Native Hashtag Indexing + Filecoin Image Uploads (Platform-First Microblog Track)

[snissn]

011: Native Hashtag Indexing + Filecoin Image Uploads (Platform-First Microblog Track)

Goal

Improve Token Host Builder as a platform, not just as a demo generator, by adding:

• native secondary query indexes with tokenizer support for hashtags,
• native generated-UI image upload support,
• a first-class upload provider abstraction with a Filecoin Onchain Cloud implementation,
• and a canonical microblog demo that proves the new platform surfaces end-to-end.

This ticket intentionally takes the "hard way" so the outcome is reusable across future apps and schemas rather than solving only one demo.

Sprint execution / source of truth

• This ticket is the source of truth for this sprint.
• The work should be implemented as a set of 8 stacked PRs, corresponding to the phases in this ticket.
• AI may prepare the PRs, but they are not to be merged until human approval.
• This ticket closes only when all 8 PRs are merged.

Recommended PR stack:

1. docs scaffolding + backlog/spec sync
2. THS extensions for native tokenized indexes
3. generator equality index support
4. generator native hashtag token index support + bounded behavior
5. bounded-cost tests and perf validation
6. upload abstraction + Filecoin Onchain Cloud adapter
7. generated UI native image upload support
8. canonical microblog example + end-to-end hardening/docs sync

Why this exists

Current builder reality is close, but still incomplete for the intended product shape:

• query indexes are modeled in THS but not fully implemented in the generator/runtime,
• image is a first-class schema field type but the generated UI still treats create/edit as plain text entry,
• Filecoin is a first-class deploy target, but uploads are not yet integrated as a first-class generated-app capability,
• hashtag-based feeds are possible only through app-specific modeling workarounds, not through a native indexing capability.

This ticket closes those gaps in a way that improves the long-term Token Host platform contract.

Spec alignment

• SPEC.md section 2.1 (deterministic builds)
• SPEC.md section 2.4 (CRUD-first, indexer-optional)
• SPEC.md section 2.5 (secure-by-default configuration)
• SPEC.md section 6.4 (field definitions)
• SPEC.md section 6.7 (indexes and constraints)
• SPEC.md section 7.5 (bounded list scanning)
• SPEC.md section 7.8 and 7.8.1 (on-chain indexes and key derivation)
• SPEC.md section 7.9 (events)
• SPEC.md section 8.3 (uploads)
• SPEC.md section 13.6 (upload signing / managed upload interface)

Problem statement

We need a canonical Token Host example app that is a microblog:

• text posts,
• image posts,
• hashtag-based feed navigation.

But the implementation must strengthen the platform itself:

• no demo-only image uploader,
• no browser-side foc-cli,
• no Tag/PostTag workaround as the primary architecture,
• no unbounded tokenization or indexing behavior that is "correct" but gas-hostile in practice.

Product outcome

After this ticket lands, Token Host Builder should support:

• a schema-defined native tokenized secondary index suitable for hashtags,
• deterministic and bounded-cost contract generation for that index type,
• generated UIs that treat image as a native uploadable field,
• a runtime upload-provider abstraction with a Filecoin Onchain Cloud adapter,
• and a canonical microblog demo schema/UI that exercises those capabilities.

Scope

1. THS / schema modeling

• Extend query-index modeling in a backward-compatible way.
• Preserve existing meaning of:
  • indexes.index: [{ field }] => equality index
• Add optional native tokenizer/index options for string fields, expected to support at least:
  • index mode/type
  • tokenizer kind
  • normalization policy
  • bounded token extraction settings where appropriate
• Add schema validation and linting for unsupported combinations.
• Add schema migration support for any new THS shape.

2. Generator / contract layer

• Implement missing equality index support in the Solidity generator.
• Implement tokenized hashtag index generation as a first-class on-chain secondary index.
• Keep index storage and access semantics aligned with SPEC 7.8:
  • candidate IDs only,
  • paginated accessors,
  • bounded limits,
  • deterministic key derivation,
  • correctness validated by follow-up getC / current-record reads.
• Define and enforce bounded token/scan behavior so gas remains predictable.
• Add events or event coverage sufficient for indexer/debugging parity where needed.

3. Runtime / client layer

• Add runtime helpers for equality-index and token-index lookups.
• Ensure generated UIs can query native hashtag feeds without client-side full scans of all posts.
• Keep runtime behavior compatible with indexer-optional operation.

4. Upload abstraction

• Introduce a first-class upload provider abstraction for generated apps.
• The browser/UI must talk to a stable Token Host upload interface, not directly to foc-cli.
• Implement a Filecoin Onchain Cloud provider behind a relay/adapter.
• The adapter may use foc-cli operationally on the server side.
• The generated UI should remain storage-provider-agnostic.
• The upload architecture should remain runner-agnostic so different deployments can choose different execution models without changing schema or UI behavior.

5. Generated UI

• Replace plain text handling for image create/edit with:
  • file pick,
  • upload progress,
  • preview,
  • retry,
  • replace/remove,
  • final URL/CID persistence in the normal field value.
• Improve list/detail rendering so image-backed records look native in generated apps.
• Keep this as base generated UI behavior, not only app-extension code.

6. Canonical demo app

• Add a canonical microblog schema and generated-app example that proves:
  • text posts,
  • image posts,
  • hashtag feeds using the native token index,
  • Filecoin Onchain Cloud-backed image upload flow.
• Initial target chain for the example should be filecoin_calibration.

Non-goals

• Building a general-purpose full-text search engine.
• Browser-side direct integration with foc-cli.
• Requiring a hosted Token Host control plane/backend to use the feature locally.
• Implementing every possible tokenizer or normalization mode in v1.
• Solving video uploads or arbitrary asset pipelines in this ticket.
• Rewriting the entire generated UI routing model.

Operating rules for this workstream

Spec sync rule

• Any merge that changes platform behavior must do one of:
  • update SPEC.md in the same change, or
  • record an explicit spec delta and follow-up reconciliation note.
• AGENTS.md must be updated as milestones land so the backlog remains authoritative.

Documentation rule

This ticket is not complete unless the following living artifacts are created and maintained:

• work log
  • chronological execution log
  • decisions, changes, test evidence, follow-ups
• working note
  • current architecture
  • open questions
  • spec delta tracker
  • unresolved tradeoffs
• blog post / memo draft
  • narrative summary of the problem, architecture, limits, and results
  • suitable for future product, engineering, and external communication reuse

The initial recommended locations are:

• docs/worklogs/native-hashtag-indexing-and-filecoin-image-uploads.md
• docs/spec-deltas/native-hashtag-indexing-and-filecoin-image-uploads.md
• docs/native-hashtag-indexing-and-filecoin-image-uploads-memo.md

Filecoin Onchain Cloud integration rules

• Use AgentHub material for foc-cli as the operator-facing source of truth while implementing the adapter.
• Prefer the recommended high-level upload flow, not low-level dataset management, unless a concrete requirement forces lower-level control.
• Keep Calibration (314159) versus mainnet (314) explicit in tooling, docs, and tests.
• Any credentials, wallet material, or funding configuration must remain environment-based and never be committed.

Proposed architecture

Native hashtag index

Recommended v1 direction:

• support native tokenized query indexes on string fields,
• first tokenizer implementation is hashtag extraction,
• normalization is deterministic and documented,
• index storage is append-only candidate buckets keyed by token hash,
• reads are paginated and bounded,
• callers validate current record state and current field contents on read.

This should be modeled as a platform index feature, not as app-specific relational materialization.

Upload provider abstraction

Recommended v1 direction:

• generated UI calls a Token Host upload interface,
• interface is implemented locally in preview/dev and can later map to managed hosted mode,
• Filecoin Onchain Cloud is one provider implementation,
• final on-chain value remains a URL/CID string in the normal image field.
• runner choice is a deployment concern, not a schema concern.

Recommended runner modes to preserve flexibility:

• process
  • local/dev or self-hosted Node service shells out to foc-cli
• remote
  • generated UI or local preview talks to an external HTTP upload service that runs the FOC adapter elsewhere
• worker
  • upload requests are handed off to a background worker where that is operationally preferable
• sdk
  • reserved for a future direct-library path if we replace CLI invocation later

The generated UI should not need to know which runner mode is being used. It should only depend on a stable upload request/response contract.

Bounded-cost and performance requirements

This ticket must explicitly protect against designs that are algorithmically "correct" but operationally too expensive.

Required limits

The implementation must define and enforce bounded limits for at least:

• maximum list scan steps,
• maximum index page size,
• maximum tokens extracted per indexed field on create/update,
• maximum token length,
• maximum normalized source field length considered for tokenization, if needed,
• maximum multicall page/query combinations used by generated UI feed paths.

Required validation

Add schema validation or lints for dangerous configurations, including when appropriate:

• tokenizer requested on unsupported field types,
• tokenizer requested while on-chain indexing is disabled,
• field/UI combinations that imply unsupported behavior,
• settings that exceed safe generator/runtime limits.

Required tests

Add tests that measure both correctness and boundedness.

Contract/generator tests:

• equality index generation and accessors compile and behave correctly,
• hashtag token index generation compiles and behaves correctly,
• over-limit tokenization inputs revert or truncate according to the defined contract behavior,
• candidate buckets remain paginated and bounded.

Gas/perf tests:

• gas snapshot or threshold tests for post creation with:
  • no hashtags,
  • typical hashtag count,
  • maximum allowed hashtag count
• gas snapshot or threshold tests for update paths with token changes,
• regression gate so a substantial gas increase fails CI or at least fails a dedicated perf suite.

Runtime/UI tests:

• generated UI does not fall back to full collection scans for hashtag feeds when native indexes are available,
• upload flow exposes progress/error states,
• image posts render correctly in list/detail surfaces.

Integration tests:

• end-to-end path for:
  1. upload image,
  2. create post,
  3. retrieve by hashtag feed,
  4. render feed entry with image.

Deliverables

• THS schema/type updates for native tokenized query indexes.
• THS validation/lint updates and migration support as needed.
• Solidity generator support for:
  • equality indexes,
  • native hashtag token indexes,
  • bounded accessors and token limits.
• Runtime helpers for index-based reads.
• Upload-provider abstraction and Filecoin Onchain Cloud adapter.
• Generated UI support for native image upload fields.
• Canonical microblog schema/example app.
• Work log, working note, and blog/memo draft.
• Backlog/spec updates reflecting the new feature set.

Acceptance criteria

• A schema can declare a native hashtag-capable token index without custom app-specific relational modeling.
• Generated contracts compile and expose bounded, paginated accessors for equality and hashtag token indexes.
• Generated UI can create text and image posts without asking users to paste image URLs manually.
• Filecoin Onchain Cloud upload works through the Token Host upload abstraction in local/dev flow.
• Filecoin Onchain Cloud upload architecture does not assume a single hosting model for the runner; at minimum, the design must support both:
  • local/self-hosted process execution
  • remote adapter service execution
• A canonical microblog example on filecoin_calibration demonstrates:
  • text post create,
  • image post create,
  • hashtag feed navigation.
• Spec deltas or spec updates are committed alongside behavior changes.
• Performance evidence exists for token/scan bounds and is recorded in repo docs.

Sequencing

Phase A: freeze design + process

• Create the work log, working note, and memo/blog draft.
• Record initial spec delta or update SPEC.md for any new THS/index semantics.
• Update AGENTS.md with the planned workstream.

Phase B: schema and generator foundations

• Extend THS query-index modeling.
• Add validation/linting/migration support.
• Implement equality indexes properly.
• Implement native hashtag token indexes with bounded behavior.

Phase C: runtime and UI

• Add runtime lookup helpers.
• Add upload-provider abstraction.
• Implement Filecoin Onchain Cloud upload adapter.
• Upgrade generated UI image field support.

Phase D: canonical app and hardening

• Add canonical microblog example.
• Add gas/perf test coverage and thresholds.
• Finalize documentation, memo/blog draft, and backlog/spec sync.

Risks and open questions

• Tokenizer semantics must be simple enough to keep deterministic behavior and bounded gas.
• String normalization choices can create compatibility or UX confusion if not specified clearly.
• foc-cli is an operator CLI, so local/dev ergonomics and credential handling must be designed carefully.
• Some performance assertions may need dedicated CI lanes if they are too slow or environment-sensitive for the default test suite.
• If we discover that on-chain token indexing is too expensive even with strict bounds on some chains, the feature must degrade cleanly under onChainIndexing=false.

Suggested implementation note

If a design choice is good engineering practice but not yet clearly represented in SPEC.md, prefer to write down the spec delta immediately instead of letting code become the de facto spec.

[snissn]

Progress update on Native Hashtag Indexing + Filecoin Image Uploads.

Completed so far:

• THS/generator/runtime support for native equality indexes and tokenized hashtag indexes
• bounded token/list behavior with validation and test coverage
• native generated-UI image upload flows
• local preview upload support with local and foc-cli process-runner modes
• remote upload runner contract with exact manifest endpoint metadata
• standalone remote upload adapter example
• canonical microblog example using native hashtag indexing and native image upload flows
• integration coverage for both local/self-hosted and remote-upload topologies
• SPEC.md reconciliation for the new upload/index contract surface

Still open before this sprint is truly finished:

• decide whether upload runner-mode names stay in the stable manifest surface or collapse behind capability flags later
• decide whether current token/list bounds remain generator defaults or move into chain-config-driven limits
• reconcile current lightweight event-hash behavior against the spec’s stronger full-record-hash preference
• cut the work into the intended stacked PR boundaries and prepare them for human review

The local sprint docs (work log, working note, memo) have been updated to reflect the spec sync point.

[snissn]

Additional progress update.

This slice hardened the bounded-cost story instead of leaving it as hidden generator constants:

• generateAppSolidity now accepts explicit generation limits
• th generate, th build, and th up now resolve chain-aware generation limits
• emitted chain-config artifacts now include list, multicall, and tokenized-index limits
• chain-config schema now documents limits.indexing.tokenized.maxTokens and maxTokenLength
• SPEC.md and the sprint docs were updated accordingly

Validation:

• pnpm build
• pnpm mocha test/testCliDev.js test/testCliVerify.js test/testQueryIndexes.js test/testCliBuildArtifacts.js

Note: a full pnpm test:integration rerun was blocked by an environmental ENOSPC failure in /tmp during a Next build. The specific upload-preview test that failed in the full run passed in isolation after cleanup, so this does not currently point to a functional regression in the feature work.

[snissn]

Additional progress update.

This slice closed the event-hash/spec gap:

• generated contracts now emit canonical per-collection record hashes for create events
• update events now emit the post-update record hash in the existing changedFieldsHash slot
• SPEC.md now explicitly allows that v1 interpretation of changedFieldsHash
• the benchmark-registry fixture still builds, so the stronger integrity hash did not reintroduce the earlier stack-pressure regression

Validation:

• pnpm build
• pnpm mocha test/testCrudGenerator.js test/testCliBenchmarkRegistryBuild.js test/testQueryIndexes.js

[snissn]

Packaging update.

The repo now has an explicit stacked PR review plan at:

• docs/plans/issue-62-stacked-pr-plan.md

That doc maps the sprint to 8 PRs with:

• purpose
• primary file ownership/scope
• acceptance criteria
• review order
• merge gate

I also linked it from:

• docs/tickets/011-native-hashtag-indexing-and-filecoin-image-uploads.md
• AGENTS.md

This slice is documentation/process only; no platform behavior changed.

[snissn]

Review handoff update.

I added two companion docs for the 8-PR stack:

• docs/plans/issue-62-pr-manifests.md
• docs/plans/issue-62-review-checklist.md

The first maps actual files and verification commands into each PR in the stack.
The second gives reviewers a short go/no-go checklist and reiterates the merge gate.

These are linked from docs/plans/issue-62-stacked-pr-plan.md.
This slice is docs/process only.

[snissn]

Operational hardening update.

I fixed the UI build temp-workspace path so th build no longer creates its temporary Next.js workspace under /tmp/tokenhost-ui-build-*.

What changed:

• the temporary UI build workspace now lives under the build output tree
• it is cleaned up after a successful build
• there is now a regression test proving th build leaves no .tokenhost-build-tmp directory behind

Validation:

• pnpm build
• pnpm mocha test/testCliBuildArtifacts.js

This directly targets the ENOSPC failure mode we saw when /tmp was tight during integration runs.

[snissn]

Stack-cutting update.

I added docs/plans/issue-62-hot-files-split-map.md.

Purpose:

• document how to cut the real 8-PR stack when a few files carry changes for multiple slices
• especially packages/cli/src/index.ts, packages/generator/src/solidity/generateAppSolidity.ts, and SPEC.md
• map those files by anchor/function/section instead of forcing whole-file ownership

It is linked from docs/plans/issue-62-stacked-pr-plan.md.
This slice is docs/process only.

[snissn]

Patch-export update.

I added a small helper so the current working tree can be turned into stack-shaped patch drafts:

• machine-readable manifest: docs/plans/issue-62-pr-manifests.json
• exporter: scripts/export-issue-62-stack.mjs
• package scripts:
  • pnpm issue62:stack:list
  • pnpm issue62:stack:export

What it does:

• lists the 8 PRs and their files
• exports one draft patch plus one metadata file per PR
• flags PRs that still require anchor-level manual splitting via splitRequired

Validation:

• node scripts/export-issue-62-stack.mjs --list
• node scripts/export-issue-62-stack.mjs --out /tmp/issue-62-stack-export

[snissn]

Patch-export materialization update.

The exporter is now carrying per-PR verification commands, and I generated a real local bundle at:

• artifacts/issue-62-stack

That bundle now contains:

• one draft patch per PR
• one metadata file per PR
• summary.json
• a generated README.md with verification commands and split-required warnings

Validation:

• pnpm issue62:stack:export
• inspected artifacts/issue-62-stack/README.md
• inspected artifacts/issue-62-stack/pr-04-bounded-cost-controls-and-chain-aware-limits.json

[snissn]

Path-readiness update from the exported stack bundle.

The current export bundle (artifacts/issue-62-stack) now computes pathReady and shared-file overlaps.
Result: every PR currently shows pathReady=false.

This is expected because the sprint accumulated in one working tree and a few hot files span multiple planned slices.

Recommended first real cuts:

1. PR 2 (THS query-index extensions)
2. PR 6 (Upload abstraction and Filecoin runner support)
3. PR 7 (Native generated-UI image field UX)

Highest-friction cuts:

• PR 4 (Bounded-cost controls and chain-aware limits)
• PR 5 (Event-hash integrity reconciliation)

Reason:
those two share the densest hot files, especially packages/generator/src/solidity/generateAppSolidity.ts, packages/cli/src/index.ts, and SPEC.md.

[snissn]

First curated cut update.

I materialized the first path-ready candidate from the broader stack bundle:

• artifacts/issue-62-stack/curated/pr-02-ths-query-index-extensions-ready.patch
• artifacts/issue-62-stack/curated/pr-02-ths-query-index-extensions-ready.json

What it keeps:

• THS query-index type changes
• THS lint changes
• both THS JSON Schemas
• THS schema tests
• only the normalizeStudioFormState query-index normalization hunk from packages/cli/src/index.ts

What it excludes:

• later CLI work that belongs to upload, bounded-cost, or preview/runtime slices

This is the first real path-ready candidate cut out of the larger draft export.

[snissn]

Curated the next low-friction stack slice.

PR 6 now has a validated path-ready candidate at:

• artifacts/issue-62-stack/curated/pr-06-upload-abstraction-and-filecoin-runner-support-ready.patch
• artifacts/issue-62-stack/curated/pr-06-upload-abstraction-and-filecoin-runner-support-ready.json

What this cut keeps:

• upload manifest emission and preview/runtime handling in packages/cli/src/index.ts
• manifest upload helpers in packages/templates/next-export-ui/src/lib/manifest.ts
• the generated-UI upload client contract in packages/templates/next-export-ui/src/lib/upload.ts
• the standalone adapter example and upload-specific tests

What it intentionally excludes:

• docs/examples-microblog-remote-upload.md
• image-field UI work that belongs to PR 7

Validation:

• git apply --check artifacts/issue-62-stack/curated/pr-06-upload-abstraction-and-filecoin-runner-support-ready.patch

I also updated the stack plan + hot-file split map so PR 7 now treats src/lib/upload.ts as an upstream dependency from PR 6 instead of re-landing it.

[snissn]

Curated PR 7 as the next stack-ready slice.

Path-ready candidate:

• artifacts/issue-62-stack/curated/pr-07-native-generated-ui-image-field-ux-ready.patch
• artifacts/issue-62-stack/curated/pr-07-native-generated-ui-image-field-ux-ready.json

What this cut keeps:

• native image field handling in generated create/edit flows
• image rendering in generic record cards
• runtime helpers for index-backed reads and hashtag queries
• ImageFieldInput and indexing helper modules

What it intentionally excludes:

• packages/templates/next-export-ui/src/lib/upload.ts (owned by curated PR 6)
• the current test/testCliGenerateUi.js microblog addition
• test/integration/testCliLocalIntegration.js because its current diff belongs to dedicated-Anvil hardening, not image-field UX

Validation:

• git apply --check artifacts/issue-62-stack/curated/pr-07-native-generated-ui-image-field-ux-ready.patch

I also updated the stack plan, work log, and hot-file split map so the repo now records PR 2, PR 6, and PR 7 as materialized curated cuts.

[snissn]

Curated PR 3 as the next stack-ready slice.

Path-ready candidate:

• artifacts/issue-62-stack/curated/pr-03-core-generator-indexes-ready.patch
• artifacts/issue-62-stack/curated/pr-03-core-generator-indexes-ready.json

What this cut keeps:

• native equality-index generation in packages/generator/src/solidity/generateAppSolidity.ts
• native hashtag token index generation in packages/generator/src/solidity/generateAppSolidity.ts
• a minimal test/testQueryIndexes.js focused on index generation only

What it intentionally excludes:

• generator limit option exports / explicit limit overrides, which belong to PR 4
• event-hash changes and the current test/testCrudGenerator.js diff, which belong to PR 5

Validation:

• git apply --check artifacts/issue-62-stack/curated/pr-03-core-generator-indexes-ready.patch
• pnpm mocha test/testQueryIndexes.js

I also updated the stack plan, hot-file split map, and work log so the repo now records curated cuts for PR 2, PR 3, PR 6, and PR 7.

[snissn]

PR 4 now has a curated path-ready cut.

Artifacts:

• artifacts/issue-62-stack/curated/pr-04-bounded-cost-controls-and-chain-aware-limits-ready.patch
• artifacts/issue-62-stack/curated/pr-04-bounded-cost-controls-and-chain-aware-limits-ready.json

What this cut keeps:

• generator limit types and explicit limit plumbing
• chain-aware CLI generation limits and chain-config emission
• the th build temp-workspace relocation out of /tmp
• bounded-cost test coverage in test/testQueryIndexes.js, test/testCliBuildArtifacts.js, test/testCliDev.js, and test/testCliVerify.js

What it intentionally excludes:

• upload runtime handling / preview upload wiring in packages/cli/src/index.ts
• dedicated-Anvil fallback changes
• event-hash changes (left for PR 5)

Validation:

• pnpm mocha test/testQueryIndexes.js test/testCliBuildArtifacts.js test/testCliDev.js test/testCliVerify.js
• validated as a stacked patch on top of the curated PR 3 baseline via git apply pr-03 && git apply --check pr-04

I also updated the local stack docs/work log so PR 4 is now listed alongside the already materialized PR 2, PR 3, PR 6, and PR 7 cuts.