Goal

The goal of this project is to stand up a self-hosted pipeline for "deep research".

Roadmap

• Stand up the infrastructure required to self-host a lab environment (security will be ok but not great, no expectation of consistent load of any kind, this will not be exposed to the public Internet):

1. Self-hosted OpenAI compliant model with "thinking" capability running with Ollama ✅
2. Apache Airflow to host the workflow, captured in a directed acyclic graph (DAG) ✅
   • Fully replicate all functionality of mmajurski's example.py:

     1. Manager Agent
     2. Exhaustive Scanner
     3. Synthesis Manager
     4. Section Writers
     5. Evaluation Probes

3. Test under various conditions, attempt to identify areas that could be optimized
4. Distribute the optimized DAG to run on multiple workers with different specs, hosted on different networks
5. Tear it all down, write Infrastructure as Code (IaC) to spin up the project using Ansible or similar
6. Re-evaluate the architecture with security top of mind (hardening for in-production environments)
7. Airflow has a Kubernetes provider (apache-airflow-providers-cncf-kubernetes), re-architect the solution to run in Pods
8. Deploy to a cutting edge public cloud environment (Google Cloud Platform)
9. Document cost to run, apply FinOps to optimize and forecast for a likely enterprise (rightsized infra) scenario
10. Deploy to a budget public cloud platform (Digital Ocean)
11. Document cost to run, apply FinOps to forecast for a likely academic (spot instances/best price) scenario

Some assembly required

The parts to be assembled are:

• Self-hosted "AI" (LLM) (Ollama in this case, which can expose OpenAI spec compliant models, provided you have the required hardware resources)
• Workflow management platform (Apache Airflow), which contains modules to connect to your chosen model, any external tools, "skills", etc.
• Directed acyclic graphs (pipelines), or workflows which are a representation in code (Python)

Why bother?

The pattern shared by mmajurski/(U.S.) NIST in the repo this project was forked from contains an example.py. You run this from a terminal. It generates a report.md containing the (annotated) resulted "deep research", along with citations in a separate file. Everything is designed to be machine readable.

The value of the shared pattern is that it contains everything required to do a whole lot more ..what that means to you is an exercise left to the reader.