Added Westend para workflow to CI and justfile #152

x3c41a · 2025-12-18T09:40:54Z

Added all integration tests for Polkadot solochain and Westend parachain

Also:

Made tests work on darwin/MacOS
Paramterised justfile functions
Added chainspecpath command line argument to examples/authorize_and_store_papi_smoldot.js
Stabilize CI

x3c41a · 2025-12-18T09:41:53Z

examples/justfile

+    # Check out the required branch
+    echo "   Checking out branch bko-bulletin-para-support..."
+    git fetch origin
+    # TODO: later, before merging, switch to the `bko-bulletin-support` https://github.com/paritytech/polkadot-sdk/pull/10662 (not now)


Shall this TODO be fixed in this PR?

Shall this TODO be fixed in this PR?

Let's better do a follow-up, but you need to add this runtime API to the bulletin-westend runtime: https://github.com/paritytech/polkadot-sdk/pull/10656/files#diff-443b1be7b55963d65b470281941835067391449385b417b01a5ab5183951f253 (or bump polkadot sdk master if that PR is merged).

bkontur · 2025-12-18T13:27:34Z

examples/justfile

    echo "   Using zombienet: $ZOMBIENET_BIN"
+
+    # Handle different runtimes
+    if [ "{{ runtime }}" = "bulletin-westend-runtime" ]; then


@x3c41a I would rather keep bulletin-solo-zombienet-start as it is and instead create bulletin-westend-para-zombienet-start and move this IF to the setup-services where we decide what zombienet to use, it will be easier to review and we don't need to change anything for bulletin-solo-zombienet-start

reasonable! this function started to grow substantially.
I'll remember to make this refactor

…ts and kill processes from old runs -- very useful when you run test locally

x3c41a · 2025-12-19T15:27:35Z

State of the art:

Polkadot solochain works ✅,
Westend parachain fails with No sudo error <-- needs to be fixed ‼️,
CI now runs both --> will be moved to GH matrix in a separate PR,

Improvements:

Chainspec is read from local file not from chain,
- parametrised script,
Found the root cause of librocksdb-sys issue on darwin ,
- added symlink and fixed it,
Zombienet just commands now clean-up all past jobs - very useful locally, when debugging and running many processes
- may be an issue if we'd want to parallelise at some point ⚠️

Left to do:

once Westend parachain works, review justfile and possibly refactor: clearly separate solochain and parachain flows

bkontur · 2025-12-21T00:14:53Z

Westend parachain fails with No sudo error <-- needs to be fixed ‼️,

hmm, it worked for me locally for the first run:

│ Name                         │ bulletin-westend-collator-2                                                                        │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Direct Link (pjs)            │ https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:12346#/explorer                                   │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Direct Link (papi)           │ https://dev.papi.how/explorer#networkId=custom&endpoint=ws://127.0.0.1:12346                       │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prometheus Link              │ http://127.0.0.1:42989/metrics                                                                     │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Log Cmd                      │ tail -f  /tmp/bulletin-tests-run/bulletin-westend-collator-2.log                                   │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Parachain ID                 │ 1006                                                                                               │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ChainSpec Path               │ /tmp/bulletin-tests-run/1006-westend-local.json                                                    │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
🔗 Connecting IPFS nodes (Docker, runtime: bulletin-westend-runtime)...
   Using Docker host: /ip4/127.0.0.1
connect 12D3KooWJKVVNYByvML4Pgx1GWAYryYo6exA68jQX9Mw3AJ6G5gQ success
connect 12D3KooWJ8sqAYtMBX3z3jy2iM98XGLFVzVfUPtmgDzxXSPkVpZZ success
🔄 Starting IPFS reconnect script (Docker mode, runtime: bulletin-westend-runtime)...
   Reconnect PID: 4126404
   Log: /tmp/ipfs-reconnect-docker.log
🔧 Generating PAPI descriptors...

> papi:generate
> papi add -w ws://localhost:10000 bulletin

Replacing existing bulletin config
✔ Metadata saved as .papi/metadata/bulletin.scale
Saved new spec "bulletin"
Reading metadata
Generating descriptors
CLI Building entry: .papi/descriptors/src/index.ts
CLI tsup v8.5.0
CLI Target: es2022
CJS Build start
ESM Build start
CJS .papi/descriptors/dist/index.js 259.08 KB
CJS ⚡️ Build success in 29ms
ESM .papi/descriptors/dist/index.mjs                      4.21 KB
ESM .papi/descriptors/dist/descriptors-UFYZNTV6.mjs       10.08 KB
ESM .papi/descriptors/dist/metadataTypes-LB7R5ILF.mjs     35.22 KB
ESM .papi/descriptors/dist/bulletin_metadata-WW6GZQWG.mjs 205.66 KB
ESM ⚡️ Build success in 29ms
Compilation started
Compilation successful
npm install

up to date, audited 443 packages in 2s

76 packages are looking for funding
  run `npm fund` for details

5 vulnerabilities (2 moderate, 3 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
✅ Services setup complete (Docker mode)
[CID]: Using cidCodec: 85 and mhCode: 45600
Authorizing account...
✅ Authorize event: signed
✅ Authorize event: broadcasted
✅ Authorize event: txBestBlocksState
📦 Authorize included in block: 0x9f70013224b7def9cf508b26727604518e07327e3bba54009f5383dc1656d927
⬆️ Storing data with length= 90
[CID]: Using cidCodec: 85 and mhCode: 45600
✅ Store event: signed
✅ Store event: broadcasted
✅ Store event: txBestBlocksState
📦 Store included in block: 0x0e2b6032d10c69ecb8e8a7f19eee667e9dc79748b370c07203f66cff917dfb32
✅ Data stored successfully with CID: CID(bafk2bzacebxyvykdvvjet4uui5saahltm52k33zvdmxr6p762unstsdob3v64)
⬇️ Downloading the full content (no chunking) by cid from url:  http://127.0.0.1:8080/ipfs/bafk2bzacebxyvykdvvjet4uui5saahltm52k33zvdmxr6p762unstsdob3v64
✅ Content fetched successfully on attempt 1
✅ Downloaded content: Hello, Bulletin with PAPI - Sun Dec 21 2025 00:12:19 GMT+0000 (Coordinated Universal Time)
✅ Verified content!



✅✅✅ Test passed! ✅✅✅

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Example completed successfully!
🧹 Stopping all Docker services...
   ⚠ ipfs-reconnect-docker (PID: 4126404) not running or already stopped
   ✓ Stopped zombienet (PID: 4121971)
🛑 Shutting down IPFS Docker container...
ipfs-node
ipfs-node
   ✓ Cleaned up container file
✅ Docker services stopped

x3c41a · 2026-01-05T08:08:59Z

hmm, it worked for me locally for the first run:

Ehh, we do need a comprehensive docker setup to avoid things working on one OS and not working on another one :(
I'll try to take care of it!

# Conflicts: # .github/workflows/integration-test.yml # examples/justfile

…eanup - Add fallback to commit hash when bko-bulletin-para-support branch doesn't exist - Clean up zombienet directory before starting to prevent interactive prompts - Fixes 'Directory already exists' prompt that was blocking CI runs

When bko-bulletin-para-support branch doesn't exist, now falls back to origin/master to get the latest stable polkadot-sdk instead of an old pinned commit.

x3c41a · 2026-01-06T08:20:47Z

examples/justfile

+# Setup prerequisites for Westend runtime (polkadot and polkadot-parachain binaries)
+# This recipe clones polkadot-sdk, builds required binaries, and copies them to ~/local_bulletin_testing/bin
+setup-westend-prerequisites:


I know the function is long but splitting it into sub-functions will make the file even longer...
Not sure which one of two devils to choose?

raymondkfcheung · 2026-01-06T09:36:38Z

examples/justfile

-    echo "   Waiting for chain to start (15 seconds)..."
-    sleep 15
+    echo "   Waiting for chain to start (120 seconds)..."
+    sleep 120


It would be nice to replace the fixed sleep 120 with a bounded readiness probe (e.g. polling RPC or logs) to make startup more reliable and avoid unnecessary CI delays.

Yeah, we already have it in our backlog.
The only purpose of this PR was to make those two flows work (and they do work!)

All the refactors and improvements will be handled separately

@x3c41a add here please TODO or better let's fix it here now :)

examples/justfile

karolk91 · 2026-01-06T09:51:51Z

examples/justfile

+# Setup prerequisites for Westend runtime (polkadot and polkadot-parachain binaries)
+# This recipe clones polkadot-sdk, builds required binaries, and copies them to ~/local_bulletin_testing/bin
+setup-westend-prerequisites:
+    #!/usr/bin/env bash


please extract this whole logic into separate *.sh file and execute it here, right now its a mix of just-style variables and bash variables and its hard to read

extracted into scripts/setup_westend_prerequisites.sh

karolk91 · 2026-01-06T09:53:06Z

examples/justfile

+    #!/usr/bin/env bash
+    set -e
+
+    BIN_DIR=~/local_bulletin_testing/bin


please avoid using ~/$HOME in general its bad practice to permanently modify people system files outside of scope of this project files - I would use /tmp instead if needed, but in this case maybe polkadot-sdk git submodule would be good (as mentioned in other comment)

neat! I'll remember about ~($HOME) vs /tmp!

examples/justfile

x3c41a added 2 commits December 18, 2025 10:39

Added Westend solochain workflow to CI and justfile

e4c2841

added newline

f0d5e76

x3c41a marked this pull request as draft December 18, 2025 09:41

x3c41a commented Dec 18, 2025

View reviewed changes

x3c41a added 3 commits December 18, 2025 10:44

removed dyld references

3b2980f

added retries to fetchCID fun

3c1b9db

updated default in just

4203d1e

bkontur reviewed Dec 18, 2025

View reviewed changes

x3c41a added 3 commits December 18, 2025 14:34

fixed Mac OS libclang error by adding symlink

e49b719

now chainspec is being read from zombienet folder, parametrized scrip…

acf37a6

…ts and kill processes from old runs -- very useful when you run test locally

added safeguards

4f8ba15

Merge branch 'main' into fix_westend

54a69de

# Conflicts: # .github/workflows/integration-test.yml # examples/justfile

x3c41a force-pushed the fix_westend branch from a38a8d5 to 54a69de Compare January 5, 2026 09:04

x3c41a added 6 commits January 5, 2026 11:29

tmp fix with fallback

70c09bb

Use latest master as fallback instead of pinned commit

e139c60

When bko-bulletin-para-support branch doesn't exist, now falls back to origin/master to get the latest stable polkadot-sdk instead of an old pinned commit.

always use master

f90da7b

reverterd api.js

a988338

reverted again

8d5e2c3

x3c41a commented Jan 6, 2026

View reviewed changes

x3c41a marked this pull request as ready for review January 6, 2026 08:22

raymondkfcheung added the github_actions Pull requests that update GitHub Actions code label Jan 6, 2026

raymondkfcheung reviewed Jan 6, 2026

View reviewed changes

x3c41a requested review from bkontur and raymondkfcheung January 6, 2026 09:55

karolk91 reviewed Jan 6, 2026

View reviewed changes

examples/justfile Outdated Show resolved Hide resolved

bkontur added 16 commits January 8, 2026 16:17

Download binaries as separate step

2cdfddb

Refactor kill-pids

a92b8bd

Separate bulletin-westend-parachain-zombienet-start for setup-services

4162488

Change order

e074f10

check command instead of -f just file

0472ee1

conditionaly skip setup_parachain_prerequisites.sh

8ad4fe3

nit

9b8e652

One more fix for local setup

3f2a6c4

Add zombienet also to the cache

c3b6cdc

let's check command instead of -f

e0aca4f

temporary hit the cache to speed up another runs

b0ba1cc

Revert back save-if

fa74eac

Just file nits

44f92a3

Hmm?

c0e80f3

make runtime required

9a71e88

Add all

0e88660

bkontur changed the title ~~Added Westend solochain workflow to CI and justfile~~ Added Westend para workflow to CI and justfile Jan 10, 2026

bkontur enabled auto-merge (squash) January 10, 2026 01:32

bkontur approved these changes Jan 10, 2026

View reviewed changes

bkontur added 3 commits January 10, 2026 03:03

Wrong cfg for smoldot?

1a379b9

skip smoldot

6284000

Maybe some cleanup problem?

1482927

bkontur disabled auto-merge January 10, 2026 10:59

bkontur added 4 commits January 10, 2026 14:38

correctly set +e before node scripts to handle EXAMPLE_EXIT

289f145

Revert back

7e201d3

Adjust timeouts

3ebc44e

nit

203c7b5

bkontur enabled auto-merge (squash) January 10, 2026 14:22

bkontur merged commit 8ec1075 into main Jan 10, 2026
8 checks passed

bkontur deleted the fix_westend branch January 10, 2026 15:01

Added Westend para workflow to CI and justfile #152

Added Westend para workflow to CI and justfile #152

Uh oh!

Conversation

x3c41a commented Dec 18, 2025 • edited by bkontur Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

x3c41a commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bkontur commented Dec 21, 2025

Uh oh!

x3c41a commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

x3c41a Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

x3c41a commented Dec 18, 2025 •

edited by bkontur

Loading

x3c41a commented Dec 19, 2025 •

edited

Loading

x3c41a Jan 6, 2026 •

edited

Loading