Add Workflow Requirements and detailed codebase index to AGENTS.md

[jiaenren]

• Add explore-first/plan/verify workflow requirements
• Add comprehensive codebase structure index covering all services, libraries, runtime containers, CLI, frontend, tests, and architecture patterns.
  • Include a "Keeping This Document Up to Date" instruction
  • Add CodeRabbit automated review for keeping codebase indexing up to date
• Add CLAUDE.md pointing to AGENTS.md for Claude Code compatibility

Description

Issue #674

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

Summary by CodeRabbit

• Documentation

  • New workflow requirements for planning, impact assessment, verification, simplification, and doc updates
  • Reworked coding standards with Python-focused guidance (imports, naming, strict typing, dataclasses, safe defaults, error handling, assertions)
  • Expanded architecture and inter-service communication overview, module purposes, error types, and workflow execution
  • Updated build and test guidance, linters, and generation workflows

• Chores

  • Added automated review guidance to surface the workflow doc when relevant paths change

Tests

Manual A/B test with Claude Code using sub-agents with and without reading AGENTS.md to do the same task:

⏺ Agent(Test WITH AGENTS.md - simple prompt)
  ⎿  Backgrounded agent
  ⎿  Prompt:
       You are working in the codebase at /Users/jiaenr/osmo/external/.

       First, read /Users/jiaenr/osmo/external/AGENTS.md.test for codebase guidance and follow its instructions.

       Task: Add a new CLI command osmo pool describe <pool_name> that shows detailed pool info including node count, GPU types, and resource utilization. Create a plan for this.

⏺ Agent(Test WITHOUT AGENTS.md - simple prompt)
  ⎿  Backgrounded agent
  ⎿  Prompt:
       You are working in the codebase at /Users/jiaenr/osmo/external/. Do NOT read any AGENTS.md or CLAUDE.md files.

       Task: Add a new CLI command osmo pool describe <pool_name> that shows detailed pool info including node count, GPU types, and resource utilization. Create a plan for this.

                                                                                                                                                                                     05:17 PM claude-opus-4-6
⏺ Both agents launched with the simplified prompt. I'll share results once they complete.

⏺ Agent "Test WITH AGENTS.md - simple prompt" completed

                                                                                                                                                                                     05:19 PM claude-opus-4-6
⏺ The with-agents-md agent is done. Waiting on the without-agents-md agent...

⏺ Agent "Test WITHOUT AGENTS.md - simple prompt" completed

                                                                                                                                                                                     05:20 PM claude-opus-4-6
⏺ Both agents are done. Here's the comparison with the simplified prompt:

  Results Comparison (v2 — Simple Prompt)

  ┌───────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────┐
  │                 Behavior                  │                             Without AGENTS.md                             │                                    With AGENTS.md                                    │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Explored first before planning?           │ Yes — read pool.py, resources.py, workflow_service.py, objects.py,        │ Yes — same files, guided by the codebase index                                       │
  │                                           │ postgres.py                                                               │                                                                                      │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Found service/core/ as backend?           │ Yes — found the right endpoints with line numbers                         │ Yes — same                                                                           │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Created explicit plan?                    │ Yes                                                                       │ Yes                                                                                  │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Checked downstream impact?                │ Partially — noted authz and large response risk                           │ Yes — noted authz, shared code reuse, GPU label provenance (K8_GPU_PRODUCT in        │
  │                                           │                                                                           │ workflow.py)                                                                         │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Proposed verification plan?               │ Implicitly — listed test cases but no specific commands                   │ Yes — listed specific bazel test and bazel run //:ruff commands                      │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ New API endpoint vs client-side           │ No new endpoint — calls 2 existing APIs from CLI                          │ New endpoint — proposed GET /api/pool/{pool_name} with PoolDescribeResponse model    │
  │ aggregation?                              │                                                                           │                                                                                      │
  ├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
  │ Tool calls / tokens used                  │ 30 tool calls / 44.8k tokens / 120s                                       │ 15 tool calls / 40.3k tokens / 91s                                                   │
  └───────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘

  Key Observations

  The pattern from round 1 held up with the simpler prompt:

  1. Architecture: The "with" agent again proposed a dedicated backend endpoint rather than making multiple API calls from the CLI. This is the more scalable design — it keeps the CLI thin and avoids client-side
  data aggregation.
  2. Efficiency: The "with" agent was faster and used fewer tool calls (15 vs 30, 91s vs 120s). The codebase index let it navigate directly to the right files instead of exploring broadly.
  3. Specificity: The "with" agent referenced K8_GPU_PRODUCT from utils/job/workflow.py for GPU label handling — a detail it found via the index. The "without" agent guessed at resource_type as the GPU field
  (which is actually the RESERVED/SHARED/UNUSED enum, not the GPU model).
  4. Verification: The "with" agent gave concrete commands (bazel test //src/service/core/workflow/tests:..., bazel run //:ruff). The "without" agent listed test cases but no runnable commands.
  5. Both agents explored first — even without the AGENTS.md instruction to do so. This is likely because the prompt said "create a plan" which naturally implies exploration.

  Verdict

  Same conclusion as round 1: AGENTS.md primarily helps with backend/architectural decisions, navigation efficiency, and concrete verification steps. The "without" agent does respectably on CLI-layer planning
  since that's discoverable from the file structure alone. The biggest gap remains the wrong field identification (resource_type enum vs nvidia.com/gpu.product label) — the kind of mistake that would waste real
  implementation time.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds AGENTS.md, a comprehensive governance document introducing mandatory pre-change workflows, expanded Python-focused coding standards, detailed src/ codebase structure and inter-service guidance, and refreshed build/test instructions; adds CLAUDE.md referencing AGENTS.md; updates .coderabbit.yaml with a path-specific review instruction. (38 words)

Changes

Cohort / File(s)	Summary
Governance & Standards AGENTS.md	New, extensive governance doc: adds Workflow Requirements (explore, plan, assess downstream impact, verify, simplify, update docs); removes prior Tool Usage Preferences; renames & expands Coding Standards into a Python-focused section (imports, naming, typing, dataclasses, error handling, assertions).
Type & Structure Guidance AGENTS.md	Expands Type Annotations and Data Structures guidance (strict typing, prefer dataclasses over dicts, avoid unnecessary Optional, safe default args); strengthens assertions guidance (discourage production asserts, prefer exceptions).
Codebase Layout & Architecture AGENTS.md	Adds detailed src/ codebase structure and module tables: Core Service, Supporting Services, Libraries, Utilities, CLI, Go runtime, frontend, operator, tests; documents inter-service communication, architecture patterns, and execution/data flows.
Build, Test & Tooling AGENTS.md	Updates Build & Test guidance: Bazel, linters (ruff), language module notes, testing frameworks, container/image guidance, OpenAPI generation, and CI/test workflows.
Reference File CLAUDE.md	New file containing a single-line pointer: @AGENTS.md.
Review Automation Config .coderabbit.yaml	Adds path_instructions under reviews.auto_review with a rule for src/** instructing reviewers to check AGENTS.md when a PR adds, removes, or renames a service/module/major component.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I hopped through pages, tidy and bright,
I nudged the imports, typed them just right,
I sketched the services and pathways galore,
Left carrots of guidance by every door,
Hop on — the map will help us explore.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: adding Workflow Requirements and a detailed codebase index to AGENTS.md, which aligns with the primary objectives of the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch jiaenr/agents-codebase-index

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@AGENTS.md`:
- Around line 243-257: The fenced diagram block in AGENTS.md is missing a
language identifier which triggers markdownlint MD040; update the opening fence
for the block containing "Client → Envoy → authz_sidecar ..." to include a
language tag (for example change ``` to ```text) so the code block is properly
labeled and lint warnings are resolved.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 758eecbe-77c5-44c1-ad49-1feafbf8549d

📥 Commits

Reviewing files that changed from the base of the PR and between 5501daa and 6690bff.

📒 Files selected for processing (1)

• AGENTS.md

[coderabbitai]

♻️ Duplicate comments (1)

AGENTS.md (1)

228-242: ⚠️ Potential issue | 🟡 Minor

Add a language identifier to the fenced diagram block.

The block still opens with a bare fence, so markdownlint MD040 will keep failing here. Please label it as plain text.

Suggested fix

-```
+```text
 Client → API Gateway → authz_sidecar (gRPC Check) → Core Service (FastAPI)
                                                          ├── PostgreSQL (state)
                                                          ├── Redis (cache, job queue, events)
                                                          ├── → Worker (job consumer)
                                                          ├── ↔ Agent (WebSocket backend events)
                                                          ├── ↔ Logger (WebSocket log streaming)
                                                          ├── → Router (HTTP/WS request routing)
                                                          └── → Delayed Job Monitor (scheduled jobs)
 
 Workflow Execution:
   Core Service → K8s Backend → [osmo_ctrl ↔ osmo_user ↔ osmo_rsync]
   osmo_ctrl ↔ Core Service (WebSocket)
   osmo_ctrl → Logger (WebSocket logs/metrics)

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @AGENTS.md around lines 228 - 242, The fenced diagram in AGENTS.md is missing
a language identifier (causing markdownlint MD040); edit the fenced block that
starts with the triple backticks containing the architecture diagram and add
"text" immediately after the opening backticks (i.e., change totext) so
the diagram is treated as plain text; keep the diagram content unchanged and
ensure the closing backticks remain.

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>🤖 Prompt for all review comments with AI agents</summary>

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In @AGENTS.md:

• Around line 228-242: The fenced diagram in AGENTS.md is missing a language
  identifier (causing markdownlint MD040); edit the fenced block that starts with
  the triple backticks containing the architecture diagram and add "text"
  immediately after the opening backticks (i.e., change totext) so the
  diagram is treated as plain text; keep the diagram content unchanged and ensure
  the closing backticks remain.

</details>

---

<details>
<summary>ℹ️ Review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Path: .coderabbit.yaml

**Review profile**: CHILL

**Plan**: Pro

**Run ID**: `778ee33e-b042-46b4-aca4-e4ff09eee9d0`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 6690bffb7d566602faf45eb584d6ae5a1bfccda7 and 39815f52e493428cee458dc91996476767834d2f.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `AGENTS.md`

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

[coderabbitai]

♻️ Duplicate comments (1)

AGENTS.md (1)

228-228: ⚠️ Potential issue | 🟡 Minor

Add a language tag to the diagram fence.

The opening fence for the inter-service communication diagram is still unlabeled, so markdownlint MD040 will keep failing until this is changed to something like ```text.

Suggested fix

-```
+```text
 Client → API Gateway → authz_sidecar (gRPC Check) → Core Service (FastAPI)
                                                          ├── PostgreSQL (state)
                                                          ├── Redis (cache, job queue, events)
                                                          ├── → Worker (job consumer)
                                                          ├── ↔ Agent (WebSocket backend events)
                                                          ├── ↔ Logger (WebSocket log streaming)
                                                          ├── → Router (HTTP/WS request routing)
                                                          └── → Delayed Job Monitor (scheduled jobs)

 Workflow Execution:
   Core Service → K8s Backend → [osmo_ctrl ↔ osmo_user ↔ osmo_rsync]
   osmo_ctrl ↔ Core Service (WebSocket)
   osmo_ctrl → Logger (WebSocket logs/metrics)

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @AGENTS.md at line 228, The markdown code fence for the inter-service
communication diagram in AGENTS.md is missing a language tag (causing
markdownlint MD040); update the opening fence for that diagram (the block
starting with ) to include a language identifier such as text (i.e. change the opening fence to text) so the diagram fence is labeled; ensure the rest
of the diagram content (lines showing Client → API Gateway → authz_sidecar …
Workflow Execution) remains unchanged.

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>🤖 Prompt for all review comments with AI agents</summary>

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In @AGENTS.md:

• Line 228: The markdown code fence for the inter-service communication diagram
  in AGENTS.md is missing a language tag (causing markdownlint MD040); update the
  opening fence for that diagram (the block starting with ) to include a language identifier such as text (i.e. change the opening fence to text) so
  the diagram fence is labeled; ensure the rest of the diagram content (lines
  showing Client → API Gateway → authz_sidecar … Workflow Execution) remains
  unchanged.

</details>

---

<details>
<summary>ℹ️ Review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Path: .coderabbit.yaml

**Review profile**: CHILL

**Plan**: Pro

**Run ID**: `45a71205-d98f-4cb6-9e1e-e383ab9d69b4`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 39815f52e493428cee458dc91996476767834d2f and ced482497cfec3233a31654f3f9356f46c5f5f4e.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `AGENTS.md`
* `CLAUDE.md`

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* CLAUDE.md

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

[ethany-nv]

nit