forked from usnistgov/agentic-research-evaluation-probes
-
Notifications
You must be signed in to change notification settings - Fork 0
Benchmarks
Paul Szabo edited this page Jun 29, 2026
·
1 revision
Stub for various benchmarks.
Roadmap:
- Compare execution time of the example.py from mmajurski's repo with the DAG running as a task on Apache Airflow, for the same Ollama model
Note: This is effectively a comparison of
from openai import AsyncOpenAIvs.from airflow.providers.common.ai.decorators.agent import agent_task. I expect Airflow to perform better, but I don't actually know why.. yet
- Compare execution time of the "deep research" DAG running on a single host/worker vs. distributed architecture (Apache Airflow workers running on different hosts, subject to network delay, etc.)
- Compare execution time of the same running on distributed Airflow Workers vs. in Kubernetes Pods with the
apache-airflow-providers-cncf-kubernetesprovider (subject to network delay + container instantiation and teardown delay, etc.)