Name	Name	Last commit message	Last commit date
parent directory ..
img	img
openapi	openapi
publications	publications
README.md	README.md
agent.rst	agent.rst
api-reference.rst	api-reference.rst
architecture.rst	architecture.rst
blob_data.rst	blob_data.rst
blob_schema.rst	blob_schema.rst
blob_versioned_model_example.rst	blob_versioned_model_example.rst
blob_versioned_single_layer_perceptron_example.rst	blob_versioned_single_layer_perceptron_example.rst
cli-reference.rst	cli-reference.rst
conf.py	conf.py
contributing.rst	contributing.rst
default_user_guide.rst	default_user_guide.rst
index.rst	index.rst
prov_capture.rst	prov_capture.rst
prov_query.rst	prov_query.rst
prov_storage.rst	prov_storage.rst
publications.rst	publications.rst
quick_start.rst	quick_start.rst
reporting.rst	reporting.rst
rest_api.rst	rest_api.rst
schemas.rst	schemas.rst
setup.rst	setup.rst
task_schema.rst	task_schema.rst
telemetry_capture.rst	telemetry_capture.rst
web_ui.rst	web_ui.rst
workflow_schema.rst	workflow_schema.rst

Flowcept Docs: Workflow Developer Start Here

This is the easiest end-to-end guide for workflow developers.

Docs site: https://flowcept.readthedocs.io/
Swagger (when webservice is running): http://127.0.0.1:8008/docs
OpenAPI (ReadTheDocs):
- https://flowcept.readthedocs.io/en/latest/flowcept-openapi.json
- https://flowcept.readthedocs.io/en/latest/flowcept-openapi.yaml

TL;DR (copy/paste)

If you only want to see Flowcept working in 2 minutes:

pip install flowcept
flowcept --init-settings
python examples/start_here.py

Then inspect flowcept_buffer.jsonl.

Choose Your Path

I want local/offline only (no services): go to Offline path
I want DB/API querying: go to Online DB path
I want natural-language querying with an agent: go to Agentic path
I want polished reports: go to Provenance reports

1) Install and setup
2) Capture provenance
3) Access captured provenance
4) Provenance reports
- Workflow cards (markdown)
- Full reports (pdf)
5) Architecture

1) Install and setup

Install base package:

pip install flowcept

Generate settings:

flowcept --init-settings

Full file vs runtime mode:

flowcept --init-settings --full -y
flowcept --config-profile full-online -y

Important: settings.yaml is the single source of truth for Flowcept runtime behavior.

default path: ~/.flowcept/settings.yaml
override path env var: FLOWCEPT_SETTINGS_PATH

Settings reference:

Quick config profiles (recommended):

flowcept --config-profile full-online
flowcept --config-profile full-telemetry
flowcept --config-profile mq-only
flowcept --config-profile full-offline
flowcept --config-profile mq-only-no-flush

What this does:

Shows exactly which settings.yaml keys will be changed and the new values.
Asks for confirmation before writing.
Writes to FLOWCEPT_SETTINGS_PATH when set, otherwise ~/.flowcept/settings.yaml.

Skip confirmation:

flowcept --config-profile full-online -y

Current profile behavior:

full-online:
- project.db_flush_mode: online
- mq.enabled: true
- kv_db.enabled: true
- databases.mongodb.enabled: true
- databases.lmdb.enabled: false
mq-only:
- project.db_flush_mode: online
- mq.enabled: true
- kv_db.enabled: false
- databases.mongodb.enabled: false
- databases.lmdb.enabled: false
- Use Flowcept(check_safe_stops=False) with this profile.
full-offline:
- project.db_flush_mode: offline
- project.dump_buffer.enabled: true
- mq.enabled: false
- kv_db.enabled: false
- databases.mongodb.enabled: false
- databases.lmdb.enabled: false
mq-only-no-flush:
- project.db_flush_mode: offline
- project.dump_buffer.enabled: true
- mq.enabled: true
- kv_db.enabled: false
- databases.mongodb.enabled: false
- databases.lmdb.enabled: false
- Tasks accumulate locally and are bulk-published to MQ in a single end-of-run flush. Also dumps to local JSONL. Use Flowcept(check_safe_stops=False).
full-telemetry:
- enables CPU, per-CPU, process, memory, disk, network, machine telemetry
- keeps telemetry_capture.gpu: null

Adapter setup is additive:

flowcept --init-settings --dask -y
flowcept --init-settings --mlflow -y
flowcept --init-settings --tensorboard -y

These commands add adapters.<name> to the current settings file.

2) Capture provenance

2.1 Simplest example

Use:

examples/start_here.py

Run:

python examples/start_here.py

Sample code (decorator-based, simplest path):

import json

from flowcept import Flowcept, flowcept_task
from flowcept.instrumentation.flowcept_decorator import flowcept


@flowcept_task(output_names="o1")
def sum_one(i1):
    return i1 + 1


@flowcept_task(output_names="o2")
def mult_two(o1):
    return o1 * 2


@flowcept
def main():
    n = 3
    o1 = sum_one(n)
    o2 = mult_two(o1)
    print("Final output", o2)


if __name__ == "__main__":
    main()
    prov_buffer = Flowcept.read_buffer_file()
    print(json.dumps(prov_buffer, indent=2))

2.2 Custom capture examples

Use:

examples/unmanaged/simple_task2.py
examples/unmanaged/simple_task.py
examples/unmanaged/main.py

These show explicit FlowceptTask, .send(), context manager, tags, and metadata.

Sample code (explicit task objects, no decorators):

import uuid
from time import sleep

from flowcept import Flowcept, FlowceptTask

if __name__ == "__main__":
    agent1 = str(uuid.uuid4())

    flowcept = Flowcept(
        start_persistence=False,
        save_workflow=True,
        workflow_name="My First Workflow",
        campaign_id="my_super_campaign",
    ).start()

    # 1) direct event emission
    FlowceptTask(activity_id="super_func1", used={"x": 1}, agent_id=agent1, tags=["tag1"]).send()

    # 2) context-managed start/end
    with FlowceptTask(activity_id="super_func2", used={"y": 1}, agent_id=agent1, tags=["tag2"]) as t2:
        sleep(0.5)
        t2.end(generated={"o": 3})

    # 3) explicit start + explicit end
    t3 = FlowceptTask(activity_id="super_func3", used={"z": 1}, agent_id=agent1, tags=["tag3"])
    sleep(0.1)
    t3.end(generated={"w": 1})

    flowcept.stop()

2.3 Advanced capture

PyTorch + loops:

docs: https://flowcept.readthedocs.io/en/latest/prov_capture.html
example: examples/single_layer_perceptron_example.py
loop example: examples/instrumented_loop_example.py
tests:
- tests/instrumentation_tests/ml_tests/single_layer_perceptron_test.py
- tests/instrumentation_tests/flowcept_loop_test.py

Adapters:

MLflow:
- examples/mlflow_example.py
- notebooks/mlflow.ipynb
- tests/adapters/test_mlflow.py
Dask:
- examples/dask_example.py
- notebooks/dask.ipynb
- tests/adapters/test_dask.py
TensorBoard:
- examples/tensorboard_example.py
- notebooks/tensorboard.ipynb
- tests/adapters/test_tensorboard.py

Agentic provenance / MCP:

docs: https://flowcept.readthedocs.io/en/latest/agent.html
agent readme: src/flowcept/agents/README.md
code-assistant routing: AGENTS.md
agent tests: tests/agent/agent_tests.py
PROV-AGENT paper: https://arxiv.org/abs/2508.02866

3) Access captured provenance

Offline path (no services)

Best for simple local runs and CI-lite usage.

Install:

pip install flowcept

Use this config:

project:
  db_flush_mode: offline
  dump_buffer:
    enabled: true
    path: flowcept_buffer.jsonl
mq:
  enabled: false
kv_db:
  enabled: false
databases:
  mongodb:
    enabled: false
  lmdb:
    enabled: false

Read file via Flowcept:

from flowcept import Flowcept

docs = Flowcept.read_buffer_file("flowcept_buffer.jsonl")
df = Flowcept.read_buffer_file("flowcept_buffer.jsonl", return_df=True, normalize_df=True)

Read file via pandas:

import pandas as pd
df = pd.read_json("flowcept_buffer.jsonl", lines=True)

Online DB path

Use this when you need workflow/task/object queries after execution.

Typical config requirements:

project.db_flush_mode: online
mq.enabled: true
kv_db.enabled: true
databases.mongodb.enabled: true (or LMDB mode)

Python Query API

Install:

pip install flowcept[mongo]

Examples:

from flowcept import Flowcept

tasks = Flowcept.db.get_tasks_from_current_workflow()

failed = Flowcept.db.query(
    collection="tasks",
    filter={"status": "ERROR"},
    projection={"_id": 0, "task_id": 1, "activity_id": 1, "stderr": 1},
    limit=20,
)

wfs = Flowcept.db.workflow_query(filter={"name": "Perceptron GridSearch"})
objs = Flowcept.db.query(collection="objects", filter={"workflow_id": Flowcept.current_workflow_id})

More docs:

REST API

Install:

pip install flowcept[webservice,mongo]

Start webservice:

flowcept --start --webservice --webservice-host=127.0.0.1 --webservice-port=8008

Quick curl examples:

curl -s http://127.0.0.1:8008/api/v1/health/live
curl -s 'http://127.0.0.1:8008/api/v1/tasks?limit=5'
curl -s -X POST http://127.0.0.1:8008/api/v1/tasks/query \
  -H 'Content-Type: application/json' \
  -d '{"filter":{"status":"FINISHED"},"limit":5}'

Docs endpoints:

Swagger: http://127.0.0.1:8008/docs
ReDoc: http://127.0.0.1:8008/redoc
OpenAPI JSON: http://127.0.0.1:8008/openapi.json
ReadTheDocs REST docs: https://flowcept.readthedocs.io/en/latest/rest_api.html

MQ subscribe

Install:

pip install flowcept[redis]

CLI stream helper:

flowcept --stream-messages
flowcept --stream-messages --keys-to-show activity_id,workflow_id,status

Examples:

examples/consumers/simple_consumer.py
examples/consumers/simple_publisher.py
examples/consumers/ping_pong_example.py

Agentic path

Paper: https://arxiv.org/abs/2509.13978

Install:

pip install flowcept[llm_agent]

Use your own code assistant

Recommended for Codex/Claude/Gemini users.

Start MCP server in a separate terminal:

flowcept --start --agent

Configure external-LLM mode:

agent:
  external_llm: true
  mcp_host: 127.0.0.1
  mcp_port: 8000

In your assistant session, read AGENTS.md, then follow docs/agent.rst and src/flowcept/agents/README.md.

Internal-LLM mode

Flowcept builds the model using build_llm_model() (src/flowcept/agents/llm/builders.py).

Providers in code:

openai
azure
google
sambanova

Common settings under agent:

service_provider
model
model_kwargs
api_key
llm_server_url
mcp_host, mcp_port

Start agent UI:

flowcept --start --agent-gui

Grafana monitoring

Deployment file:

deployment/compose-grafana.yml

Use this for online telemetry dashboards and KPI correlations (for example loss vs CPU/GPU behavior).

Telemetry docs:

https://flowcept.readthedocs.io/en/latest/telemetry_capture.html

4) Provenance reports

Workflow cards (markdown)

Default report mode:

report_type="workflow_card"
format="markdown"

The rendered workflow card follows the upstream Workflow Card template: https://github.com/data-cards/workflow-provenance-card.

Python API:

from flowcept import Flowcept

Flowcept.generate_report(
    report_type="workflow_card",
    format="markdown",
    workflow_id="<workflow_id>",
    output_path="WORKFLOW_CARD.md",
)

REST download:

curl -s -X POST \
  http://127.0.0.1:8008/api/v1/workflows/<workflow_id>/reports/workflow-card/download

Docs:

https://flowcept.readthedocs.io/en/latest/reporting.html

Full reports (pdf)

Install:

pip install flowcept[report_pdf]

Generate:

from flowcept import Flowcept

Flowcept.generate_report(
    report_type="provenance_report",
    format="pdf",
    workflow_id="<workflow_id>",
    output_path="PROVENANCE_REPORT.pdf",
)

PDF report supports ML-specialized KPI plotting when signals are available.

5) Architecture

One-line architecture:

capture (instrumentation/adapters) -> MQ -> DB/file -> query via Python/REST/agent -> report.

Symptom: flowcept: command not found
- Fix: activate your Python env first, then reinstall: pip install flowcept
Symptom: no flowcept_buffer.jsonl after running offline example
- Fix: ensure project.db_flush_mode: offline and project.dump_buffer.enabled: true in settings
Symptom: ValueError about db_flush_mode vs MQ/DB settings
- Fix: keep config consistent:
  - Offline mode (no MQ/KV/DBs): flowcept --config-profile full-offline -y
  - Offline mode with end-of-run MQ flush: flowcept --config-profile mq-only-no-flush -y
  - Online mode: flowcept --config-profile full-online -y or flowcept --config-profile mq-only -y
Symptom: ValueError about check_safe_stops=True requiring KV while MQ is enabled
- Fix: either use flowcept --config-profile full-online -y, or use mq-only / mq-only-no-flush and instantiate Flowcept(check_safe_stops=False)
Symptom: REST API import/start failures (fastapi/uvicorn missing)
- Fix: pip install flowcept[webservice,mongo]
Symptom: Flowcept.db queries fail due to missing Mongo deps
- Fix: pip install flowcept[mongo]
Symptom: agent won’t respond / cannot connect
- Fix:
  - start server: flowcept --start --agent
  - confirm agent.mcp_host/agent.mcp_port in settings
  - in external assistant mode, follow AGENTS.md and docs/agent.rst
Symptom: PDF report generation fails
- Fix: install report deps: pip install flowcept[report_pdf]

Command cheat sheet

# Install
pip install flowcept
pip install flowcept[mongo]
pip install flowcept[webservice,mongo]
pip install flowcept[llm_agent]
pip install flowcept[report_pdf]

# Init settings
flowcept --init-settings
flowcept --config-profile full-online
flowcept --config-profile mq-only
flowcept --config-profile full-offline
flowcept --show-settings

# Start services
flowcept --start --webservice --webservice-host=127.0.0.1 --webservice-port=8008
flowcept --start --agent
flowcept --start --agent-gui

# Stream MQ messages
flowcept --stream-messages
flowcept --stream-messages --keys-to-show activity_id,workflow_id,status

# Run simple example
python examples/start_here.py

# Python quick calls
from flowcept import Flowcept

# Read offline buffer
docs = Flowcept.read_buffer_file("flowcept_buffer.jsonl")
df = Flowcept.read_buffer_file("flowcept_buffer.jsonl", return_df=True, normalize_df=True)

# Generate markdown workflow card
Flowcept.generate_report(
    report_type="workflow_card",
    format="markdown",
    workflow_id="<workflow_id>",
    output_path="WORKFLOW_CARD.md",
)

# Generate PDF provenance report
Flowcept.generate_report(
    report_type="provenance_report",
    format="pdf",
    workflow_id="<workflow_id>",
    output_path="PROVENANCE_REPORT.pdf",
)

Navigation Menu

Uh oh!

README.md

Flowcept Docs: Workflow Developer Start Here

TL;DR (copy/paste)

Choose Your Path

Table of Contents

1) Install and setup

2) Capture provenance

2.1 Simplest example

2.2 Custom capture examples

2.3 Advanced capture

3) Access captured provenance

Offline path (no services)

Online DB path

Python Query API

REST API

MQ subscribe

Agentic path

Use your own code assistant

Internal-LLM mode

Grafana monitoring

4) Provenance reports

Workflow cards (markdown)

Full reports (pdf)

5) Architecture

Common failures and quick fixes

Command cheat sheet

Uh oh!

FilesExpand file tree

docs

Directory actions

More options

Directory actions

More options

Latest commit

History

docs

Folders and files

parent directory

README.md

Flowcept Docs: Workflow Developer Start Here

TL;DR (copy/paste)

Choose Your Path

Table of Contents

1) Install and setup

2) Capture provenance

2.1 Simplest example

2.2 Custom capture examples

2.3 Advanced capture

3) Access captured provenance

Offline path (no services)

Online DB path

Python Query API

REST API

MQ subscribe

Agentic path

Use your own code assistant

Internal-LLM mode

Grafana monitoring

4) Provenance reports

Workflow cards (markdown)

Full reports (pdf)

5) Architecture

Common failures and quick fixes

Command cheat sheet