Skip to content

Commit 8ec1075

Browse files
x3c41akarolk91bkontur
authored
Added Westend para workflow to CI and justfile (#152)
* Added Westend solochain workflow to CI and justfile * added newline * removed dyld references * added retries to fetchCID fun * updated default in just * fixed Mac OS libclang error by adding symlink * now chainspec is being read from zombienet folder, parametrized scripts and kill processes from old runs -- very useful when you run test locally * added safeguards * tmp fix with fallback * Fix CI: handle missing polkadot-sdk branch and zombienet directory cleanup - Add fallback to commit hash when bko-bulletin-para-support branch doesn't exist - Clean up zombienet directory before starting to prevent interactive prompts - Fixes 'Directory already exists' prompt that was blocking CI runs * Use latest master as fallback instead of pinned commit When bko-bulletin-para-support branch doesn't exist, now falls back to origin/master to get the latest stable polkadot-sdk instead of an old pinned commit. * always use master * reverterd api.js * reverted again * export DYLD_FALLBACK_LIBRARY_PATH Co-authored-by: Karol Kokoszka <[email protected]> * extracted setup into a separate script * store PID in /tmp * unified process turndown * use inner/outer scope * do not use top level dir /tmp * merged fns * experiment with justfile() * fixed disconnet * Fix zombienet prompts * Apply suggestions from code review * Rename setup_westend_prerequisites.sh to setup_parachain_prerequisites.sh * Update examples/justfile * Use polkadot-sdk cache and fixed commit * Nit * reverted TEST_DIR * Use polkadot-omni-node (build much faster than polkadot-parachain) * Download and cache binaries * Attemp to build omni-node once and cache * Nit * Download binaries as separate step * Refactor kill-pids * Separate bulletin-westend-parachain-zombienet-start for setup-services * Change order * check command instead of `-f` just file * conditionaly skip setup_parachain_prerequisites.sh * nit * One more fix for local setup * Add zombienet also to the cache * let's check command instead of `-f` * temporary hit the cache to speed up another runs * Revert back `save-if` * Just file nits * Hmm? * make runtime required * Add all * Wrong cfg for smoldot? * skip smoldot * Maybe some cleanup problem? * correctly `set +e` before node scripts to handle EXAMPLE_EXIT * Revert back * Adjust timeouts * nit --------- Co-authored-by: Karol Kokoszka <[email protected]> Co-authored-by: Branislav Kontur <[email protected]>
1 parent d897072 commit 8ec1075

File tree

4 files changed

+485
-142
lines changed

4 files changed

+485
-142
lines changed

‎.github/workflows/integration-test.yml‎

Lines changed: 134 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -18,20 +18,89 @@ concurrency:
1818

1919
env:
2020
NODE_VERSION: 22
21+
POLKADOT_SDK_VERSION: polkadot-stable2512
22+
POLKADOT_SDK_BIN_DIR: ${{ github.workspace }}/.polkadot-sdk-bin
23+
ZOMBIENET_VERSION: v1.3.138
24+
ZOMBIENET_BIN_DIR: ${{ github.workspace }}/.zombienet-bin
2125

2226
jobs:
23-
integration-tests:
24-
name: Integration Tests
27+
setup:
28+
name: Setup
2529
runs-on: ubuntu-latest
26-
timeout-minutes: 60
27-
2830
steps:
2931
- name: Free Disk Space (Ubuntu)
3032
if: ${{ runner.environment == 'github-hosted' }}
3133
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
3234
with:
3335
tool-cache: false
3436

37+
# TODO: remove when released: https://github.com/paritytech/polkadot-sdk/pull/10662
38+
- name: Install Rust toolchain
39+
uses: dtolnay/rust-toolchain@stable
40+
with:
41+
targets: wasm32-unknown-unknown
42+
components: rust-src
43+
- name: Install system dependencies
44+
run: |
45+
sudo apt-get update
46+
sudo apt-get install -y protobuf-compiler libclang-dev
47+
48+
# Cache the Polkadot SDK binaries
49+
- name: Cache Polkadot SDK binaries
50+
uses: actions/cache@v3
51+
id: polkadot-sdk-cache
52+
with:
53+
path: ${{ env.POLKADOT_SDK_BIN_DIR }}
54+
key: polkadot-sdk-${{ env.POLKADOT_SDK_VERSION }}-binaries
55+
# Download and extract binaries if cache missed
56+
- name: Download Polkadot SDK binaries
57+
if: steps.polkadot-sdk-cache.outputs.cache-hit != 'true'
58+
run: |
59+
mkdir -p $POLKADOT_SDK_BIN_DIR
60+
cd $POLKADOT_SDK_BIN_DIR
61+
echo "Downloading Polkadot SDK binaries..."
62+
curl -L -o polkadot https://github.com/paritytech/polkadot-sdk/releases/download/${POLKADOT_SDK_VERSION}/polkadot
63+
curl -L -o polkadot-prepare-worker https://github.com/paritytech/polkadot-sdk/releases/download/${POLKADOT_SDK_VERSION}/polkadot-prepare-worker
64+
curl -L -o polkadot-execute-worker https://github.com/paritytech/polkadot-sdk/releases/download/${POLKADOT_SDK_VERSION}/polkadot-execute-worker
65+
curl -L -o chain-spec-builder https://github.com/paritytech/polkadot-sdk/releases/download/${POLKADOT_SDK_VERSION}/chain-spec-builder
66+
# TODO: remove when released: https://github.com/paritytech/polkadot-sdk/pull/10662
67+
# curl -L -o polkadot-omni-node https://github.com/paritytech/polkadot-sdk/releases/download/${POLKADOT_SDK_VERSION}/polkadot-omni-node
68+
chmod +x *
69+
# TODO: remove when released: https://github.com/paritytech/polkadot-sdk/pull/10662
70+
git clone https://github.com/paritytech/polkadot-sdk.git
71+
cd polkadot-sdk
72+
git reset --hard b2bcb74b13f1a1e082f701e3e05ce1be44d16790
73+
cargo build -p polkadot-omni-node -r
74+
cd ..
75+
cp polkadot-sdk/target/release/polkadot-omni-node .
76+
rm -R polkadot-sdk
77+
78+
# Cache the Zombienet binaries
79+
- name: Cache Zombienet
80+
uses: actions/cache@v3
81+
id: zombienet-cache
82+
with:
83+
path: ${{ env.ZOMBIENET_BIN_DIR }}
84+
key: zombienet-${{ env.ZOMBIENET_VERSION }}-binaries
85+
# Download and extract binaries if cache missed
86+
- name: Download Zombienet binaries
87+
if: steps.zombienet-cache.outputs.cache-hit != 'true'
88+
run: |
89+
mkdir -p $ZOMBIENET_BIN_DIR
90+
cd $ZOMBIENET_BIN_DIR
91+
curl -L \
92+
-H "Authorization: token ${{ github.token }}" \
93+
-o zombienet-linux-x64 \
94+
"https://github.com/paritytech/zombienet/releases/download/${ZOMBIENET_VERSION}/zombienet-linux-x64"
95+
chmod +x zombienet-linux-x64
96+
97+
integration-tests:
98+
needs: [setup]
99+
name: Integration Tests
100+
runs-on: ubuntu-latest
101+
timeout-minutes: 200
102+
103+
steps:
35104
- name: Checkout sources
36105
uses: actions/checkout@v4
37106

@@ -63,35 +132,78 @@ jobs:
63132
- name: Install just
64133
run: cargo install just --locked || true
65134

66-
- name: Download zombienet
135+
- name: Cache Polkadot SDK binaries
136+
uses: actions/cache@v3
137+
id: polkadot-sdk-cache
138+
with:
139+
path: ${{ env.POLKADOT_SDK_BIN_DIR }}
140+
key: polkadot-sdk-${{ env.POLKADOT_SDK_VERSION }}-binaries
141+
- name: Cache Zombienet
142+
uses: actions/cache@v3
143+
id: zombienet-cache
144+
with:
145+
path: ${{ env.ZOMBIENET_BIN_DIR }}
146+
key: zombienet-${{ env.ZOMBIENET_VERSION }}-binaries
147+
- name: Add binaries to PATH
67148
run: |
68-
curl -L \
69-
-H "Authorization: token ${{ github.token }}" \
70-
-o zombienet-linux-x64 \
71-
"https://github.com/paritytech/zombienet/releases/download/v1.3.138/zombienet-linux-x64"
72-
chmod +x zombienet-linux-x64
73-
echo "ZOMBIENET_BINARY=$GITHUB_WORKSPACE/zombienet-linux-x64" >> $GITHUB_ENV
74-
75-
- name: Run authorize and store (PAPI, RPC node)
149+
ls -lrt "${POLKADOT_SDK_BIN_DIR}"
150+
ls -lrt "${ZOMBIENET_BIN_DIR}"
151+
echo "${POLKADOT_SDK_BIN_DIR}" >> "$GITHUB_PATH"
152+
echo "SKIP_PARACHAIN_SETUP=1" >> "$GITHUB_ENV"
153+
echo "${ZOMBIENET_BIN_DIR}" >> "$GITHUB_PATH"
154+
echo "ZOMBIENET_BINARY=zombienet-linux-x64" >> "$GITHUB_ENV"
155+
156+
# Westend parachain
157+
- name: Run authorize and store (PAPI, RPC node, Westend parachain)
76158
working-directory: examples
77159
run: |
78160
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
79-
echo "TEST_DIR=$TEST_DIR" >> $GITHUB_ENV
80-
just run-authorize-and-store "ws"
81-
82-
- name: Run authorize and store (PAPI, smoldot)
161+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
162+
mkdir -p "$TEST_DIR"
163+
just run-authorize-and-store "bulletin-westend-runtime" "ws"
164+
- name: Run authorize and store (PAPI, smoldot, Westend parachain)
165+
working-directory: examples
166+
# TODO: remove when smoldot for para is fixed
167+
continue-on-error: true
168+
run: |
169+
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
170+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
171+
mkdir -p "$TEST_DIR"
172+
just run-authorize-and-store "bulletin-westend-runtime" "smoldot"
173+
- name: Run store chunked data + DAG-PB (PJS-API, RPC node, Westend parachain)
83174
working-directory: examples
175+
# TODO: tmp
176+
continue-on-error: true
84177
run: |
85178
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
86-
echo "TEST_DIR=$TEST_DIR" >> $GITHUB_ENV
87-
just run-authorize-and-store "smoldot"
179+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
180+
mkdir -p "$TEST_DIR"
181+
just run-store-chunked-data "bulletin-westend-runtime"
182+
183+
# TODO: Polkadot parachain
88184

89-
- name: Run store chunked data + DAG-PB (PJS-API, RPC node)
185+
# Polkadot solochain
186+
- name: Run authorize and store (PAPI, RPC node, Polkadot solochain)
187+
working-directory: examples
188+
run: |
189+
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
190+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
191+
mkdir -p "$TEST_DIR"
192+
just run-authorize-and-store "bulletin-polkadot-runtime" "ws"
193+
- name: Run authorize and store (PAPI, smoldot, Polkadot solochain)
194+
working-directory: examples
195+
run: |
196+
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
197+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
198+
mkdir -p "$TEST_DIR"
199+
just run-authorize-and-store "bulletin-polkadot-runtime" "smoldot"
200+
- name: Run store chunked data + DAG-PB (PJS-API, RPC node, Polkadot solochain)
90201
working-directory: examples
91202
run: |
92203
export TEST_DIR="$(mktemp -d $GITHUB_WORKSPACE/bulletin-tests-run-XXXXX)/test"
93-
echo "TEST_DIR=$TEST_DIR" >> $GITHUB_ENV
94-
just run-store-chunked-data
204+
echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV"
205+
mkdir -p "$TEST_DIR"
206+
just run-store-chunked-data "bulletin-polkadot-runtime"
95207
96208
# Collects logs from the last failed zombienet run.
97209
- name: Upload Zombienet logs (on failure)

‎examples/authorize_and_store_papi_smoldot.js‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import assert from "assert";
22
import * as smoldot from 'smoldot';
3-
import { ApiPromise, WsProvider } from "@polkadot/api";
3+
import { readFileSync } from 'fs';
44
import { createClient } from 'polkadot-api';
55
import { getSmProvider } from 'polkadot-api/sm-provider';
66
import { cryptoWaitReady } from '@polkadot/util-crypto';
@@ -10,22 +10,14 @@ import { cidFromBytes } from "./cid_dag_metadata.js";
1010
import { bulletin } from './.papi/descriptors/dist/index.mjs';
1111

1212
// Constants
13-
const BOB_NODE_WS = 'ws://localhost:12346';
1413
const SYNC_WAIT_SEC = 15;
1514
const SMOLDOT_LOG_LEVEL = 3; // 0=off, 1=error, 2=warn, 3=info, 4=debug, 5=trace
1615
const HTTP_IPFS_API = 'http://127.0.0.1:8080' // Local IPFS HTTP gateway
1716

18-
async function fetchChainSpec(nodeWs) {
19-
console.log('Fetching chainspec from node...');
20-
const provider = new WsProvider(nodeWs);
21-
const api = await ApiPromise.create({ provider });
22-
await api.isReady;
23-
24-
const chainSpec = (await api.rpc.syncstate.genSyncSpec(true)).toString();
25-
const chainSpecObj = JSON.parse(chainSpec);
26-
chainSpecObj.protocolId = null; // Allow smoldot to sync with local chain
27-
28-
await api.disconnect();
17+
function readChainSpec(chainspecPath) {
18+
const chainSpecContent = readFileSync(chainspecPath, 'utf8');
19+
const chainSpecObj = JSON.parse(chainSpecContent);
20+
chainSpecObj.protocolId = null;
2921
return JSON.stringify(chainSpecObj);
3022
}
3123

@@ -41,8 +33,8 @@ function initSmoldot() {
4133
return sd;
4234
}
4335

44-
async function createSmoldotClient() {
45-
const chainSpec = await fetchChainSpec(BOB_NODE_WS);
36+
async function createSmoldotClient(chainspecPath) {
37+
const chainSpec = readChainSpec(chainspecPath);
4638
const sd = initSmoldot();
4739
const chain = await sd.addChain({ chainSpec });
4840
const client = createClient(getSmProvider(chain));
@@ -53,10 +45,18 @@ async function createSmoldotClient() {
5345
async function main() {
5446
await cryptoWaitReady();
5547

48+
// Get chainspec path from command line argument
49+
const chainspecPath = process.argv[2];
50+
if (!chainspecPath) {
51+
console.error('❌ Error: Chainspec path is required as first argument');
52+
console.error('Usage: node authorize_and_store_papi_smoldot.js <chainspec-path>');
53+
process.exit(1);
54+
}
55+
5656
let sd, client, resultCode;
5757
try {
5858
// Init Smoldot PAPI client and typed api.
59-
({ client, sd } = await createSmoldotClient());
59+
({ client, sd } = await createSmoldotClient(chainspecPath));
6060
console.log(`⏭️ Waiting ${SYNC_WAIT_SEC} seconds for smoldot to sync...`);
6161
// TODO: check better way, when smoldot is synced, maybe some RPC/runtime api that checks best vs finalized block?
6262
await new Promise(resolve => setTimeout(resolve, SYNC_WAIT_SEC * 1000));

0 commit comments

Comments
 (0)