Skip to content

Add support for Durable Objects storage#570

Open
cdunster wants to merge 50 commits intomainfrom
547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret
Open

Add support for Durable Objects storage#570
cdunster wants to merge 50 commits intomainfrom
547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret

Conversation

@cdunster
Copy link
Contributor

@cdunster cdunster commented Mar 9, 2026

Summary

This adds the necessary wrangler project that can be deployed to Cloudflare to create a Durable Object store. This store is currently just used by the Unyt Chain Transaction scenario but has been kept someone general as it just stores JSON objects so can be used by other scenarios in the future.

FYI, if all of wrangler stuff gets accepted here then we no longer need holochain/hc-github-config#72 as that was going to be a fresh repo just for these files but I thought that they seemed small enough to keep in this repo and this means that the devs have everything they need to run the scenario locally in this repo. I'm open to suggestions though.

I also renamed the NUMBER_OF_LINKS_TO_PROCESS scenario environment variable to UNYT_NUMBER_OF_LINKS_TO_PROCESS to make it clear that it is only used in the Unyt Chain Transaction scenario and added the missing add_capture_env("UNYT_NUMBER_OF_LINKS_TO_PROCESS") call.

Closes #547.

TODO:

  • All code changes are reflected in docs, including module-level docs
  • All new/edited/removed scenarios are reflected in summary visualiser tool (see checklist)
  • I ran the Nomad CI workflow successfully on my branch

Note that all commits in a PR must follow Conventional Commits before it can be merged, as these are used to generate the changelog

Summary by CodeRabbit

  • New Features

    • Durable per-run data store with HTTP set/get access, secret-protected access, and automatic 12-hour expiry
    • Local dev runnable durable-object service and glue to scenarios via configurable endpoint and secret
  • Documentation

    • Instructions for running the local durable-object store and updated scenario run commands/environment variables
  • Chores

    • Project config, build and dev tooling additions; environment wiring and gitignore expanded for local tooling artifacts

@cdunster cdunster self-assigned this Mar 9, 2026
@coderabbitai
Copy link

coderabbitai bot commented Mar 9, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a Cloudflare Durable Object worker and local dev tooling, replaces hardcoded Durable Object URL/secret with environment/Nomad secrets for the Unyt scenario, and provides Nix and shell helpers to run the Durable Object store locally.

Changes

Cohort / File(s) Summary
Durable Object worker
durable_object_store/src/index.ts, durable_object_store/package.json, durable_object_store/tsconfig.json, durable_object_store/wrangler.jsonc
New TypeScript Durable Object service (RunStore) with POST /set and GET /get, project config, TypeScript settings, and Wrangler config including migrations and durable object binding.
Local dev tooling & script
flake.nix, scripts/local-durable-objects.sh
Adds devShell env vars and packages (nodejs, wrangler), a local-durable-objects shell app, updates shellHook to source the script, and a script to run wrangler dev using UNYT_DURABLE_OBJECTS_URL/UNYT_DURABLE_OBJECTS_SECRET.
Scenario code & docs
scenarios/unyt_chain_transaction/README.md, scenarios/unyt_chain_transaction/src/behaviour/smart_agreements.rs, scenarios/unyt_chain_transaction/src/durable_object.rs, scenarios/unyt_chain_transaction/src/main.rs
Replaces hardcoded Durable Object URL/secret with UNYT_DURABLE_OBJECTS_URL/UNYT_DURABLE_OBJECTS_SECRET, switches to UNYT_NUMBER_OF_LINKS_TO_PROCESS, adds client timeout, and documents local run steps.
Orchestration (Nomad)
nomad/run_scenario.tpl.hcl
Adds a conditional job_secrets block and injects UNYT_DURABLE_OBJECTS_URL and UNYT_DURABLE_OBJECTS_SECRET into the unyt_chain_transaction task env when that scenario is selected.
Repo tooling
.gitignore
Adds ignore rules: node_modules/, .wrangler/, and worker-configuration.d.ts.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Suggested reviewers

  • ThetaSinner
  • veeso
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Add support for Durable Objects storage' clearly and concisely summarizes the main change: introducing a Durable Object store implementation for the Wrangler project.
Linked Issues check ✅ Passed All key requirements from #547 are addressed: Durable Object store implementation, environment variables for URL/secret, local development support via scripts, and Nomad secret block integration.
Out of Scope Changes check ✅ Passed All changes directly support the objectives of #547: new Durable Object store code, environment variable management, local development scripts, and related configuration updates are all in scope.
Description check ✅ Passed The pull request description follows the template structure with a comprehensive summary, completed TODO checklist, and conventional commits note.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 9, 2026

The following will be added to the changelog


[0.6.1] - 2026-03-09

Features

  • Get Unyt durable object store URL and secret from env vars
  • Use UNYT_ prefix on Unyt-only env var
  • Add subdirectory for the Cloudflare durable object store

Bug Fixes

  • Capture the missing Unyt number of links env var

Miscellaneous Tasks

  • Add Unyt Durable Object store URL and secret to the Nomad jobs
  • Add bash function to run a local Durable Objects store
  • Add Unyt durable object store URL and secret to devShell
  • Add generated nodeJS and wrangler dirs and files to gitignore
  • Add nodeJS and wrangler for the durable object store

Documentation

  • Update the readme for Unyt scenario to include durable objects

@cdunster cdunster force-pushed the 547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret branch from d73e89e to 814c729 Compare March 9, 2026 14:49
@cdunster cdunster marked this pull request as ready for review March 9, 2026 14:55
@github-actions

This comment was marked as outdated.

@cdunster cdunster requested review from a team and ThetaSinner March 9, 2026 14:55
@cdunster cdunster force-pushed the 547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret branch from 311b595 to 60ce170 Compare March 9, 2026 14:59
@github-actions

This comment was marked as outdated.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (1)
scripts/local-durable-objects.sh (1)

1-7: Add defensive shell options and environment variable validation.

The script assumes UNYT_DURABLE_OBJECTS_URL and UNYT_DURABLE_OBJECTS_SECRET are set and that the URL contains a port. Consider adding validation to provide clearer error messages when prerequisites are missing.

🛡️ Proposed defensive improvements
 #!/usr/bin/env bash
+set -euo pipefail
 
 run_local_durable_object_store() {
+    if [[ -z "${UNYT_DURABLE_OBJECTS_URL:-}" ]]; then
+        echo "Error: UNYT_DURABLE_OBJECTS_URL is not set" >&2
+        return 1
+    fi
+    if [[ -z "${UNYT_DURABLE_OBJECTS_SECRET:-}" ]]; then
+        echo "Error: UNYT_DURABLE_OBJECTS_SECRET is not set" >&2
+        return 1
+    fi
+
     local port=${UNYT_DURABLE_OBJECTS_URL##*:}
+    if [[ ! "$port" =~ ^[0-9]+$ ]]; then
+        echo "Error: Could not extract valid port from UNYT_DURABLE_OBJECTS_URL" >&2
+        return 1
+    fi
 
     wrangler dev --types --config="$(pwd)/durable_object_store/wrangler.jsonc" --local --port "$port" --var "SECRET_KEY:$UNYT_DURABLE_OBJECTS_SECRET"
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/local-durable-objects.sh` around lines 1 - 7, Add defensive shell
options and validate required env vars inside run_local_durable_object_store:
enable strict mode (set -euo pipefail) at the top of the script, check that
UNYT_DURABLE_OBJECTS_URL and UNYT_DURABLE_OBJECTS_SECRET are defined and
non-empty, verify that UNYT_DURABLE_OBJECTS_URL contains a colon and a valid
port after extracting it (using the current extraction logic with
${UNYT_DURABLE_OBJECTS_URL##*:}), and on any failure print a clear error to
stderr and exit non-zero; keep the call to wrangler dev in
run_local_durable_object_store but ensure variables are quoted.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@durable_object_store/src/index.ts`:
- Around line 51-57: The handler unconditionally returns 200 after calling
env.RUN_STORE.idFromName/run_store.get and executing
stub.fetch("https://internal/set"), so write failures aren't propagated; update
the code that calls stub.fetch (the fetch call on the stub) to await and inspect
the returned Response, and if the response is not ok (status >= 400) return or
throw a Response that propagates the status and body (or an error message)
instead of always returning 200; ensure you reference the stub.fetch call and
use idFromName/get to locate the logic to implement this check and return the
stub's error status/body when it fails.
- Around line 42-60: In handlePost, stop using truthiness checks for
run_id/value/secret and instead validate types and presence: parse the JSON and
if parsing fails return 400, then verify run_id is a non-empty string (typeof
run_id === "string" && run_id.length > 0), verify secret is a string and matches
env.SECRET_KEY, and verify the request actually includes a value key (use
Object.prototype.hasOwnProperty.call(body, "value") or "value" in body) so falsy
values like 0/false/"" are accepted; keep the env.RUN_STORE.idFromName/get/fetch
logic but return 400 for bad input and 403 for secret mismatch, preserving the
catch block for unexpected errors.
- Around line 64-86: handleGet currently returns records to anyone with run_id;
add the same shared-secret validation used in handlePost before fetching the
stored record. In handleGet, read the incoming secret (e.g., from Authorization
header or a "secret" param) and compare it to env.SHARED_SECRET (or the same
validation helper used by handlePost); if missing or mismatched return 401/403
immediately, otherwise proceed to env.RUN_STORE.idFromName/run stub.fetch.
Ensure you reuse the same validation logic/function used by handlePost to keep
behavior consistent.

In `@nomad/run_scenario.tpl.hcl`:
- Around line 135-148: The Durable Objects URL is hard-coded into the env
(UNYT_DURABLE_OBJECTS_URL); replace the literal with a configuration-driven
value fetched from Nomad-managed config/secrets (e.g., use
secret.job_secrets.UNYT_DURABLE_OBJECTS_URL or a template variable from ds
"vars" such as index (ds "vars") "unyt_durable_objects_url" with a sensible
default) so the connection endpoint is sourced from the existing secret
"job_secrets" or template data rather than baked into UNYT_DURABLE_OBJECTS_URL.
- Line 147: The HCL2 attribute assignment for UNYT_DURABLE_OBJECTS_URL currently
ends with an invalid trailing semicolon; remove the semicolon after the URL
value so the line reads as a plain HCL attribute assignment (i.e.,
UNYT_DURABLE_OBJECTS_URL =
"https://wind-tunnel-durable-objects.holochain.workers.dev") to fix the template
validation error.

---

Nitpick comments:
In `@scripts/local-durable-objects.sh`:
- Around line 1-7: Add defensive shell options and validate required env vars
inside run_local_durable_object_store: enable strict mode (set -euo pipefail) at
the top of the script, check that UNYT_DURABLE_OBJECTS_URL and
UNYT_DURABLE_OBJECTS_SECRET are defined and non-empty, verify that
UNYT_DURABLE_OBJECTS_URL contains a colon and a valid port after extracting it
(using the current extraction logic with ${UNYT_DURABLE_OBJECTS_URL##*:}), and
on any failure print a clear error to stderr and exit non-zero; keep the call to
wrangler dev in run_local_durable_object_store but ensure variables are quoted.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5c17c813-3229-4f58-a6a9-36dfba79a738

📥 Commits

Reviewing files that changed from the base of the PR and between 6d4fc48 and 311b595.

📒 Files selected for processing (12)
  • .gitignore
  • durable_object_store/package.json
  • durable_object_store/src/index.ts
  • durable_object_store/tsconfig.json
  • durable_object_store/wrangler.jsonc
  • flake.nix
  • nomad/run_scenario.tpl.hcl
  • scenarios/unyt_chain_transaction/README.md
  • scenarios/unyt_chain_transaction/src/behaviour/smart_agreements.rs
  • scenarios/unyt_chain_transaction/src/durable_object.rs
  • scenarios/unyt_chain_transaction/src/main.rs
  • scripts/local-durable-objects.sh

Comment on lines +64 to +86
async function handleGet(url: URL, env: Env): Promise<Response> {
const run_id = url.searchParams.get("run_id");
if (!run_id) {
return new Response(JSON.stringify({ error: "Missing run_id" }), { status: 400 });
}
const id = env.RUN_STORE.idFromName(run_id);
const stub = env.RUN_STORE.get(id);
const resp = await stub.fetch("https://internal/get");
const data = await resp.json<{ error?: string; value?: unknown }>();
if (data.error) {
return new Response(JSON.stringify(data), { status: 404 });
}
return new Response(JSON.stringify(data), { status: 200 });
}

export default {
async fetch(request: Request, env: Env): Promise<Response> {
const url = new URL(request.url);
const method = request.method;
if (method === "POST") {
return await handlePost(request, env);
} else if (method === "GET") {
return await handleGet(url, env);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Apply the shared secret to reads as well.

handleGet currently makes every stored record readable to any caller that knows run_id. If these payloads are not intentionally public, this endpoint needs the same secret check as handlePost.

🔐 Proposed fix
-async function handleGet(url: URL, env: Env): Promise<Response> {
+async function handleGet(request: Request, env: Env): Promise<Response> {
+	const url = new URL(request.url);
+	const secret = request.headers.get("x-secret");
+	if (secret !== env.SECRET_KEY) {
+		return new Response(JSON.stringify({ error: "Unauthorized" }), { status: 403 });
+	}
 	const run_id = url.searchParams.get("run_id");
 	if (!run_id) {
 		return new Response(JSON.stringify({ error: "Missing run_id" }), { status: 400 });
 	}
@@
 		const url = new URL(request.url);
 		const method = request.method;
 		if (method === "POST") {
 			return await handlePost(request, env);
 		} else if (method === "GET") {
-			return await handleGet(url, env);
+			return await handleGet(request, env);
 		} else {
 			return new Response("Method Not Allowed", { status: 405 });
 		}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@durable_object_store/src/index.ts` around lines 64 - 86, handleGet currently
returns records to anyone with run_id; add the same shared-secret validation
used in handlePost before fetching the stored record. In handleGet, read the
incoming secret (e.g., from Authorization header or a "secret" param) and
compare it to env.SHARED_SECRET (or the same validation helper used by
handlePost); if missing or mismatched return 401/403 immediately, otherwise
proceed to env.RUN_STORE.idFromName/run stub.fetch. Ensure you reuse the same
validation logic/function used by handlePost to keep behavior consistent.

Comment on lines +37 to +40
let base_url = std::env::var("UNYT_DURABLE_OBJECTS_URL")
.expect("UNYT_DURABLE_OBJECTS_URL needs to be set for this scenario to run");
let secret = std::env::var("UNYT_DURABLE_OBJECTS_SECRET")
.expect("UNYT_DURABLE_OBJECTS_SECRET needs to be set for this scenario to run");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm tempted to make these WT_DURABLE_OBJECTS_URL and WT_DURABLE_OBJECTS_SECRET and make them generic to Wind Tunnel itself so that other scenarios could use them in the future but then we should probably move all of the Durable Object stuff into common code so maybe that should come later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine either way, no objection to these being scenario specific

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave them as they are for now then. If more scenarios use them then we can make the Durable Object stuff common

mattyg
mattyg previously approved these changes Mar 10, 2026
@holochain holochain deleted a comment from cocogitto-bot bot Mar 11, 2026
@holochain holochain deleted a comment from cocogitto-bot bot Mar 11, 2026
@cdunster cdunster marked this pull request as draft March 11, 2026 14:25
@cdunster cdunster force-pushed the 547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret branch from 6758f8e to a3fae8d Compare March 11, 2026 14:25
@cdunster cdunster force-pushed the 547-hcf-to-manage-the-durable-object-storage-and-dont-hard-code-the-url-and-secret branch from af3285b to b9ab6b0 Compare March 12, 2026 15:59
wrangler secret put SECRET_KEY
```

After updating the `SECRET_KEY`, the value of `UNYT_DURABLE_OBJECTS_SECRET` in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may need to add a bit about being careful of special characters but I'm really not sure what/where the problems were with the special characters not working in the secret.

@cdunster cdunster marked this pull request as ready for review March 12, 2026 17:06
@holochain holochain deleted a comment from cocogitto-bot bot Mar 12, 2026
@holochain holochain deleted a comment from cocogitto-bot bot Mar 12, 2026
@cdunster cdunster requested review from ThetaSinner and mattyg March 12, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HCF to manage the durable object storage and don't hard-code the URL and secret

3 participants