Skip to content

developer.md

Latest commit

 

History

History
370 lines (299 loc) · 16.7 KB

developer.md

File metadata and controls

370 lines (299 loc) · 16.7 KB
Raw

Developer Guide

Onboarding, local setup, quality gates, and contribution conventions for the FlowOS monorepo.

Repository layout

flowos/
├── backend/        # Python service
│   ├── src/flowforge/
│   │   ├── domain/          # pure Python, no framework imports
│   │   ├── application/     # compiler, executor, scheduler, events, services
│   │   ├── infrastructure/  # repositories, queues, plugins, security
│   │   ├── observability/   # logging, metrics, tracing
│   │   └── presentation/    # FastAPI composition root + routers
│   ├── migrations/          # Alembic migrations
│   └── tests/
├── sdk/            # flowforge-sdk — node authoring primitives (zero deps)
│   ├── src/flowforge_sdk/
│   └── tests/
├── contracts/      # flowforge-contracts — plugin protocol wire contracts
├── frontend/       # React + TypeScript management UI
├── docs/           # this documentation set, and the README's screenshot producer under assets/
├── examples/       # sample workflows and a reference plugin
├── site/           # the landing page, one static file in the identity DESIGN.md defines
├── DESIGN.md       # the visual identity every surface reads
└── docker-compose.yml

Prerequisites

  • Python 3.13+ (CI runs 3.14)
  • uv
  • Node.js 20+ (frontend)

First-time setup

# Python workspace (backend + sdk + contracts)
uv sync --all-packages

# Frontend
cd frontend && npm install && cd ..

Running the API

# Backend (dev mode, in-memory adapters, hot reload)
make dev-api          # or: uv run --package flowforge uvicorn flowforge.presentation.main:app --reload --port 8000

# Frontend (separate terminal)
make dev-frontend     # or: cd frontend && npm run dev

On Windows, start the API without --reload when process-isolated plugins are enabled:

uv run --package flowforge python -m uvicorn flowforge.presentation.main:app --port 8000

Uvicorn's Windows reload worker uses a selector event loop, which cannot launch the subprocess transport required by process-isolated plugins. The API will isolate that plugin failure, but running without reload keeps all plugins available.

Seed the first (admin) account, then run a workflow through the UI or API. The API docs are at http://localhost:8000/docs.

Quality gates

Every change must pass all gates. make check runs the Python gates; the frontend gates run separately.

Gate Command
Format uv run --package flowforge ruff format --check .
Lint uv run --package flowforge ruff check .
Types uv run --package flowforge mypy --strict
Tests (coverage) uv run --package flowforge pytest --cov --cov-fail-under=95 -q
SDK tests uv run --package flowforge-sdk pytest -q (from sdk/)
Frontend lint npm run lint (from frontend/)
Frontend build npm run build (from frontend/)
Plugin packaging uv build --package flowos-demo-nodes
Frontend lint cd frontend && npm run lint
Frontend build cd frontend && npm run build (tsc + vite)

CI runs the Python gates on every push and pull request (.github/workflows/ci.yml). The frontend gates are not yet wired into CI — run them locally when you touch the frontend.

Conventions enforced by the gates:

  • Ruff: line length 100, double-quoted strings, isort with first-party imports (flowforge, flowforge_sdk, flowforge_contracts).
  • mypy: strict mode over backend/src, sdk/src, contracts/src.
  • Coverage: ≥ 95% across backend/src/flowforge and contracts/src/flowforge_contracts.
  • pytest: asyncio_mode = auto, markers declared in pyproject.toml (integration requires Docker-backed PostgreSQL and Redis).

Common development tasks

Add a core node

Adding an integration (an external service) means writing a plugin, not a built-in node: see docs/plugins.md and copy examples/reference_plugin. The steps below are for nodes bundled with the engine itself.

  1. Create backend/src/flowforge/infrastructure/nodes/<name>.py as a flowforge_sdk.BaseNode subclass (see the SDK guide and the existing nodes for the pattern).
  2. Register it in backend/src/flowforge/infrastructure/nodes/__init__.py through the flowforge.nodes entry-point group (re-export the class so the plugin is importable).
  3. Test it in backend/tests/infrastructure/nodes/ following the existing node tests (returned failure on bad config, output shape, retry/timeout interaction where relevant).

Change the workflow format or graph rules

The workflow aggregate lives in domain/workflow.py and validation is collected by Workflow.create → validate(). Add a ValidationIssue for new rules (codes/severities in domain/errors.py) and extend the compile step in application/compiler.py if the plan needs new metadata.

Add a repository column or table

  1. Change infrastructure/models.py (SQLAlchemy 2 mapped classes on Base).
  2. Generate a migration: uv run --package flowforge alembic revision --autogenerate -m "describe change".
  3. Update both the SQL adapter (infrastructure/persistence/) and the in-memory adapter (infrastructure/repositories/) so behavior stays consistent in development mode.
  4. Update docs/database.md if the schema is user-visible.

Add an API endpoint

  1. Add Pydantic models to presentation/api/schemas.py.
  2. Add the route to the relevant router under presentation/api/.
  3. Gate it with require_permission(...) (see domain/identity.py for the permission enums) and add a rate_limited(...) dependency for public or login endpoints.
  4. Register the router in presentation/main.py and add tests in backend/tests/presentation/.

Writing tests

  • Follow the existing suites: domain tests (pure, fast), application tests (compile/execute/events), infrastructure tests, presentation tests (HTTP layer end-to-end with the test client), and the SDK suite.
  • Aim to keep coverage ≥ 95%; new branches need tests.
  • Async tests are collected automatically (pytest-asyncio in auto mode). Do not construct async primitives (e.g. asyncio.StreamReader) in sync tests — run them in an event loop instead.
  • Do not sleep to create a race. Drive the interleave: gate a node on an asyncio.Event, move a claim's visibility deadline into the past, or hold a database session open on its first write. Every concurrency suite here does one of those, which is why they fail for a reason rather than sometimes.

What the default run does not cover

The suite runs entirely on substitutes: fakeredis for the Redis adapter and SQLite for the SQL ones. Both are faithful enough for semantics and neither is the thing that runs in production, so a handful of guarantees are executable but unexecuted by default. They are marked integration and skip unless pointed at a real service:

docker run --rm -p 6379:6379 redis:7-alpine
FLOWOS_TEST_REDIS_URL=redis://localhost:6379/15 uv run --package flowforge     pytest -m integration -q

What that closes, and nothing else does: the in-progress key's TTL being the clock that ends a visibility window, and WATCH aborting an acknowledgment that a competing claim overtook. What remains open either way: PostgreSQL — the SQL adapters' concurrent writes are exercised against file-backed SQLite in WAL mode with separate connections, which is a real race but not Postgres's MVCC.

Running the PostgreSQL and jepsen suites locally

Three more things become executable once a disposable PostgreSQL is reachable, none of which the suite above touches:

  • backend/tests/jepsen — checkpoint monotonicity and execution visibility proved against a real server, including killing it mid-write (FLOWOS_TEST_DESTRUCTIVE=1 because these suites SIGKILL the process they are talking to; see backend/tests/jepsen/conftest.py for the full precondition list). The queue-durability half of that package instead needs Redis and its own systemd unit to kill.
  • backend/tests/infrastructure/test_arq_job_queue_integration.py — the queue's ownership fence against a real Redis, not fakeredis.
  • The SQL repository suites under backend/tests/infrastructure (test_sql_*.py, test_db.py) — parametrised to also run every case against PostgreSQL when a DSN is configured, in addition to the SQLite path every developer runs by default. Point FLOWOS_TEST_POSTGRES_DSN at the server and the extra postgres-parametrised cases collect alongside the sqlite ones; leave it unset and only sqlite collects, unchanged.

Standing up disposable containers to point those variables at (a Docker Desktop or Linux Docker install, not WSL/systemd):

docker run -d --name flowos-ci-postgres \
  -e POSTGRES_USER=flowos -e POSTGRES_PASSWORD=flowos -e POSTGRES_DB=flowos \
  -p 5432:5432 postgres:17-alpine

docker run -d --name flowos-ci-redis \
  -p 6379:6379 redis:7-alpine redis-server --appendonly no
export FLOWOS_TEST_POSTGRES_DSN=postgresql+asyncpg://flowos:flowos@localhost:5432/flowos
export FLOWOS_TEST_REDIS_URL=redis://localhost:6379/0
export FLOWOS_TEST_DESTRUCTIVE=1

# Migrate the target database first — this is the only place the migration
# chain runs against real PostgreSQL rather than SQLite.
FLOWOS_DATABASE_URL="$FLOWOS_TEST_POSTGRES_DSN" uv run --package flowforge \
    alembic -c backend/alembic.ini upgrade head

uv run --package flowforge pytest -q backend/tests/jepsen \
    backend/tests/infrastructure/test_arq_job_queue_integration.py \
    backend/tests/infrastructure

requires_postgres()/requires_redis() (backend/tests/jepsen/conftest.py) skip a whole module unless the configured DSN or URL is both set and actually reachable (a short socket probe at collection time) — a bad target skips instead of failing every dependent fixture with a raw connection error.

A second, narrower precondition applies only to the tests in test_queue_durability.py that use the redis_nemesis fixture: killing and reviving a server needs a controller, not just a reachable one. That controller is one of two things now:

  • a systemd unit, reached through a prefix (FLOWOS_TEST_SYSTEMCTL, default wsl -u root --, since these servers run in WSL on this workstation);
  • the Docker containers started above, addressed as docker:<container-name> — FLOWOS_TEST_REDIS_CONTROLLER=docker:flowos-ci-redis and FLOWOS_TEST_POSTGRES_CONTROLLER=docker:flowos-ci-postgres.

Each service resolves its own controller variable first (resolve_controller() in backend/tests/jepsen/nemesis.py), falling back to FLOWOS_TEST_SYSTEMCTL when the service-specific one is unset — this is what lets CI point Postgres and Redis at their own containers without a systemd unit anywhere in the picture. The literal none still means what it always has: no controller is available, so those two tests skip with "fault injection needs a controllable server", whichever form is in play. test_checkpoint_monotonicity.py and test_execution_visibility.py never touch a nemesis, so a missing controller does not affect them either way; they run whenever PostgreSQL is reachable.

What CI runs

The integration job in .github/workflows/ci.yml runs after quality-gates against postgres:17-alpine and redis:7-alpine, started as plain docker run containers rather than GitHub Actions services: containers:

docker run -d --name flowos-ci-postgres \
  -e POSTGRES_USER=flowos -e POSTGRES_PASSWORD=flowos -e POSTGRES_DB=flowos \
  -p 5432:5432 postgres:17-alpine

docker run -d --name flowos-ci-redis \
  -p 6379:6379 redis:7-alpine redis-server --appendonly no

A services: container is a sidecar the job cannot reach with docker kill, which is exactly what fault injection needs to do to it — a plain container the job started itself, it can. The job waits for each with a pg_isready and redis-cli ping retry loop before doing anything else, runs alembic upgrade head against the container Postgres, then the command above (jepsen, the arq Redis integration suite, and the full backend/tests/infrastructure directory with its PostgreSQL-parametrised cases included), and finally an if: always() step that removes both containers whether the run passed or failed.

FLOWOS_TEST_POSTGRES_CONTROLLER=docker:flowos-ci-postgres and FLOWOS_TEST_REDIS_CONTROLLER=docker:flowos-ci-redis point each suite's nemesis at the container carrying its name — resolve_controller() in backend/tests/jepsen/nemesis.py reads the service-specific variable first and falls back to FLOWOS_TEST_SYSTEMCTL (kept at none here, since there is still no systemd unit for a wsl -u root ---style controller to address). That means CI now runs real fault injection instead of skipping it: kill issues docker kill --signal=KILL <container>, the real crude SIGKILL this whole package is built around, not a graceful docker stop. pause/resume use docker pause/docker unpause (the freezer cgroup, the closest Docker analogue to SIGSTOP/SIGCONT), and revive polls the port until it actually answers, not just until the container is running again — a restarted container's postgres or redis process is not necessarily accepting connections yet.

The two fault-injection tests in test_queue_durability.py that use the redis_nemesis fixture now execute against flowos-ci-redis instead of skipping (test_checkpoint_monotonicity.py and test_execution_visibility.py never take a nemesis fixture, so they were never affected by this either way — they already ran whenever PostgreSQL was reachable, and still do). The rest of CI (quality-gates, plugin-packaging, frontend, live-integrations) is unchanged and still runs on substitutes.

Running the end-to-end (Playwright) suite

frontend/e2e drives the whole editor in a real Chromium browser: bootstrap the first admin, create a workflow and configure it, run it and watch it succeed, pause it on a core.wait node and resume it, and create a credential. Nothing in npm test (vitest) touches a browser or a running API — this suite is the only thing here that does.

It needs a live API and a live frontend dev server, started by hand first (Playwright's webServer option cannot portably shell out to uv run across this repo's machines and CI runners, so it is not used here — see frontend/e2e/playwright.config.ts's comment):

uv run --package flowforge uvicorn flowforge.presentation.main:app --host 127.0.0.1 --port 8000
cd frontend
npm run dev -- --host 127.0.0.1 --port 5173

Then, from frontend/:

npm run e2e

If 8000/5173 are already taken by another dev stack, start both on 8001/5174 instead (VITE_PROXY_TARGET=http://127.0.0.1:8001 for the dev server) and point the suite at it: E2E_BASE_URL=http://127.0.0.1:5174 npm run e2e.

The dev-mode API allows exactly one /auth/bootstrap call for its whole process lifetime, so the suite bootstraps its one admin account itself in a Playwright globalSetup (frontend/e2e/global-setup.ts) and saves its signed-in storageState to frontend/e2e/.auth/; every spec file reuses that session instead of bootstrapping its own. Restart the API between local runs of the suite (or of the same spec twice) so that single bootstrap slot is free again.

A failed run leaves an HTML report at frontend/playwright-report/index.html — open it directly in a browser, or serve it with npx playwright show-report from frontend/, to see each failing step's trace, screenshots, and network log. CI uploads the same report (plus frontend/test-results) as a build artifact whenever the e2e job fails.

Contribution workflow

  1. Read CONTRIBUTING.md and follow the Code of Conduct.
  2. Branch from main, implement, and run all quality gates above.
  3. Open a pull request; CI re-runs the Python gates.
  4. Update CHANGELOG.md under [Unreleased] and the ROADMAP in the same commit as the feature they describe so the documents never drift.
  5. Keep commits conventional-style (fix(web): …, feat(engine): …, docs: …).

Where to look for help