This repository provides a modular workspace for portable HPC workflow orchestration with Prefect. It is designed so the same workflow code can run across multiple HPC systems by switching profile blocks.
The workspace is organized into core packages, optional integrations, and a PyPI meta-package:
qcsc-prefect/
├── docs
│ └── concept.md
├── examples
│ ├── fugaku_prefect_hello_demo
│ ├── prefect_bitcount_demo
│ └── miyabi_prefect_hello_demo
├── packages
│ ├── qcsc-prefect
│ ├── qcsc-prefect-core
│ ├── qcsc-prefect-blocks
│ ├── qcsc-prefect-adapters
│ ├── qcsc-prefect-executor
│ ├── qcsc-prefect-qiskit
│ ├── qcsc-prefect-dice
│ ├── qcsc-prefect-fulqrum
│ └── qcsc-prefect-sbd
├── pyproject.toml
└── .pre-commit-config.yaml
packages/qcsc-prefect/Meta-package for installing the core QCSC Prefect packages from PyPI.packages/qcsc-prefect-core/Common execution model definitions (for exampleExecutionProfile) shared by all targets.packages/qcsc-prefect-blocks/Prefect Block schemas for command, execution profile, and HPC profile layers.packages/qcsc-prefect-adapters/Local process runtime plus target-specific scheduler script builders and adapters.packages/qcsc-prefect-executor/High-level execution entrypoints that resolve blocks, derive scheduler routing, and dispatch to target runtimes.packages/qcsc-prefect-qiskit/Optional native Qiskit Runtime integration utilities.packages/qcsc-prefect-dice/Optional DICE SHCI solver integration.packages/qcsc-prefect-fulqrum/Optional fulqrum subspace eigensolver integration, a drop-in alternative to DICE for SQD.packages/qcsc-prefect-sbd/Optional SBD Davidson solver integration (SBDSolverJob).algorithms/Reference QCSC workflows. Each one has its ownpyproject.tomland README and is installed on its own.open_loop/sqd: sample once, then diagonalize (fulqrum by default, DICE or SBD).open_loop/skqd: sample once, then diagonalize (DICE or SBD).closed_loop/sqd_de: SQD with differential evolution of the circuit parameters (SBD or DICE).qcsc_workflow_utility: chemistry and solver helpers shared by the workflows.
examples/End-to-end runnable examples for Miyabi and Fugaku.docs/Concept and architecture documents for the block-based execution model.
QCSC Prefect requires Python 3.11 or later.
Install the core QCSC Prefect packages from PyPI with:
pip install qcsc-prefectInstall the core packages plus the native Qiskit integration with:
pip install "qcsc-prefect[qiskit]"Install the core packages plus the Python-side DICE integration with:
pip install "qcsc-prefect[dice]"Install the core packages plus the Python-side SBD integration with:
pip install "qcsc-prefect[sbd]"Install all optional integrations with:
pip install "qcsc-prefect[all]"The DICE integration can also be installed directly:
pip install qcsc-prefect-diceFor source development, install the workspace with uv from the repository root:
git clone https://github.com/qiskit-community/qcsc-prefect.git
cd qcsc-prefect
uv sync --all-packages
uv run pytestThe root pyproject.toml is a workspace coordinator and is not the package published to PyPI.
Each package under packages/ remains independently buildable.
qcsc-prefect-dice is a Python-side integration package. Installing it provides Prefect
tasks/blocks, scheduler templates, DICE input/output handling, and documentation. It does
not build, download, vendor, or install the external DICE/SBD executable.
The DICE/SBD executable must already be compiled on the target HPC system. Build instructions are site-specific because compilers, MPI stacks, BLAS/LAPACK libraries, GPU support, filesystem layout, and scheduler environments differ across HPC systems.
Configure the DICE executable through the command configuration used by the target HPC profile.
In the current block model, the CommandBlock names the executable key and the HPCProfileBlock
maps that key to the executable path visible on the target HPC system:
from qcsc_prefect_blocks.common.blocks import CommandBlock, HPCProfileBlock
CommandBlock(
command_name="dice",
executable_key="dice_solver",
description="DICE SHCI solver executable",
default_args=[],
).save("cmd-dice-solver", overwrite=True)
HPCProfileBlock(
hpc_target="miyabi",
queue_cpu="regular-c",
queue_gpu="regular-g",
project_cpu="gz00",
project_gpu="gz00",
executable_map={
"dice_solver": "/work/gz00/<user>/dice/bin/Dice",
},
).save("hpc-miyabi-dice", overwrite=True)Generated Miyabi, Fugaku, and Slurm job scripts include a preflight check that runs on the
HPC node before launching the job. The script fails early with a clear message if the configured
executable path is empty, missing, not executable, or not found on PATH.
For execution on the Prefect worker itself, set HPCProfileBlock.hpc_target="local".
Local execution invokes the mapped executable directly without a shell, job script, or scheduler.
environments are merged into the process environment; modules and pre_commands are not
supported and cause an error before the process starts.
See DICE Integration for a longer setup example.
qcsc-prefect-fulqrum runs the fulqrum
subspace eigensolver through the same block model. Its FulqrumSolverJob has the same run()
signature and SCIResult return type as DiceSHCISolverJob, so SQD workflows can switch
solvers by changing the solver block.
Like DICE, fulqrum itself is not installed by pip. Build it per site with
packages/qcsc-prefect-fulqrum/native/build_fulqrum.sh, which pins the fulqrum 1.0.0 release candidate (branch v1.0.0-rc), with
FULQRUM_WITH_GPU=1 for the GPU backends. Then install qcsc-prefect-fulqrum into that
environment with --no-deps. The compute node needs only fulqrum and its dependencies, and
runs the qcsc-fulqrum-solve entry point. The solver runs on CPUs, on the GPUs of one node,
or on one MPI rank per GPU across nodes, selected by the solver block's backend. Install the package
from the source checkout on the workflow host:
pip install -e packages/qcsc-prefect-fulqrumIt is not a uv workspace member, because uv would resolve its [solver] extra for
uv sync --all-packages. Its tests still run with uv run pytest.
See Fulqrum Integration for the block setup and
algorithms/open_loop/sqd for the sqd workflow, which uses
fulqrum by default.
- Concept and architecture:
- Example guides:
Code quality checks are configured with pre-commit (.pre-commit-config.yaml):
ruff check --fixruff formatbandit(Python security checks for medium/high-confidence findings)detect-secrets(secret scanning backed by.secrets.baseline)- basic repository hygiene hooks (
check-yaml, trailing whitespace, EOF fix, merge conflict checks)
The CI workflow (.github/workflows/ci.yml) runs on every pull request and on pushes to main:
Lint:ruff checkandruff format --checkTest (Python 3.11–3.13):pytestagainst all workspace packagesCI OK: aggregate status; use this as the required check for branch protection
pytest and ruff are defined in the root dev dependency group, which uv sync installs by default.
The ruff version is pinned so pre-commit and CI produce identical results; to upgrade it, bump the pin
in the root pyproject.toml and run uv run ruff format . in a dedicated pull request.
Miyabi/Fugaku integration tests are skipped in CI and must be run manually on the login nodes.
Each sub-package under packages/ maintains its own version in its own pyproject.toml.
The root project is a workspace coordinator (qcsc-prefect-workspace) and is not intended for distribution.
For a coordinated PyPI release, keep the qcsc-prefect-* package versions and exact internal
dependency pins aligned, for example 0.1.0.
Maintainers publish through GitHub Actions with PyPI Trusted Publishing. Do not add PyPI API tokens or password secrets to this repository.
Normal release flow:
- Run the
Prepare releaseworkflow withold_versionandnew_version. - Open a pull request manually from the generated
release/v<new_version>branch. - Review and merge the release PR.
- Push tag
v<new_version>. - Approve the
pypienvironment deployment in GitHub Actions. - Confirm PyPI installation.
After release, verify installation with:
python -m pip install "qcsc-prefect==<version>"
python -m pip install "qcsc-prefect[all]==<version>"Release notes:
- Use the release workflows for normal releases; do not use PyPI API tokens.
- Trusted Publisher is already configured for the PyPI projects.
- The
pypi-publish.ymlworkflow validates tag/package version consistency before publishing. - The DICE/SBD executable is not installed by
pip. - DICE/SBD must be built separately for each target HPC environment.
- The fulqrum solver is not installed by
pipeither; build it per site withpackages/qcsc-prefect-fulqrum/native/build_fulqrum.sh. - If a broken release is published, prefer yanking over deleting.
For the 0.1.0 packaging validation:
- GitHub Actions build-only dry run completed.
- All packages built into wheel and sdist distributions.
twine checkpassed.- Local install from
dist/passed withpip install --find-links qcsc-prefect-dist "qcsc-prefect[all]==0.1.0". - The Qiskit integration import path is
qcsc_prefect.integrations.qiskit. - The DICE executable is not installed by
pip; it must be built separately on the target HPC system. - TestPyPI Trusted Publisher registration for multiple packages is currently blocked by a suspected PyPI/TestPyPI pending publisher issue, so no actual upload was performed.
- Install pre-commit hooks:
pre-commit install
- Run checks before commit:
pre-commit run --all-files- or let them run automatically on
git commit
- Run the same checks as CI before opening a pull request:
uv run ruff check .uv run ruff format --check .uv run pytest
- (Optional) Hide the bulk formatting commit from local
git blame:git config blame.ignoreRevsFile .git-blame-ignore-revs
When adding a new HPC target, include:
- adapter implementation under
packages/qcsc-prefect-adapters/ - executor integration under
packages/qcsc-prefect-executor/ - at least one runnable example under
examples/