docs: clarify rust workspace policy and follow-ups

Signed-off-by: lucarlig <luca.carlig@ibm.com>

Author: lucarlig

AGENTS.md

@@ -8,7 +8,7 @@ For domain-specific guidance, see subdirectory AGENTS.md files:
 - `charts/AGENTS.md` - Helm chart operations
 - `docs/AGENTS.md` - Documentation authoring
 - `mcp-servers/AGENTS.md` - MCP server implementation
-- `tools_rust/mcp_runtime/DEVELOPING.md` - Rust MCP runtime development workflows, command matrix, and validation
+- `crates/mcp_runtime/DEVELOPING.md` - Rust MCP runtime development workflows, command matrix, and validation
 
 **Note:** The `llms/` directory contains guidance for LLMs *using* ContextForge solution (end-user runtime guidance), not for code agents working on this codebase.
 
@@ -38,7 +38,7 @@ charts/                     # Helm charts (see charts/AGENTS.md)
 docs/                       # Architecture and usage documentation (see docs/AGENTS.md)
 a2a-agents/                 # A2A agent implementations (used for testing/examples)
 mcp-servers/                # MCP server templates (see mcp-servers/AGENTS.md)
-tools_rust/                 # Rust utilities and MCP runtime (see tools_rust/mcp_runtime/DEVELOPING.md)
+crates/                     # Direct Rust crate folders (runtime and wrapper)
 llms/                       # End-user LLM guidance (not for code agents)
 ```
 
@@ -48,9 +48,8 @@ llms/                       # End-user LLM guidance (not for code agents)
 ```bash
 cp .env.example .env && make install-dev check-env    # Complete setup
 make venv                          # Create virtual environment with uv
-make install-dev                   # Install with dev dependencies (includes build-ui)
+make install-dev                   # Install with dev dependencies
 make check-env                     # Verify .env against .env.example
-make build-ui                      # Rebuild Admin UI JS bundle (requires npm)
 ```
 
 ### Development
@@ -66,11 +65,10 @@ make serve-ssl                    # HTTPS on :4444 (creates certs if needed)
 make autoflake isort black pre-commit
 
 # Before committing, use ty, mypy and pyrefly to check just the new files, then run:
-make ruff bandit interrogate pylint verify
+make flake8 bandit interrogate pylint verify
 
-# Before committing Rust changes (tools_rust/):
-# Run fmt-check, clippy -D warnings, and cargo test for Rust crates
-cd tools_rust/mcp_runtime && cargo fmt --check && cargo clippy -- -D warnings && cargo test
+# Before committing Rust changes (crates/ or tools_rust/):
+make rust-check                   # Runs fmt-check, clippy -D warnings, and cargo test for all Rust crates
 ```
 
 ## Authentication & RBAC Overview
@@ -137,39 +135,6 @@ ContextForge implements a **two-layer security model**:
 - **Multi-tenancy architecture**: `docs/docs/architecture/multitenancy.md`
 - **OAuth token delegation**: `docs/docs/architecture/oauth-design.md`
 
-## Observability Transaction Behavior
-
-**Issue #3883 - Separate Session Pattern**
-
-Observability write operations use **independent database sessions** that commit immediately (best-effort pattern). This means:
-
-- Observability data persists even when the main request fails
-- Traces may show "in progress" or partial states for failed requests
-- **NOT atomic** with main request transaction (intentional trade-off)
-- Provides visibility into partial failures at the cost of atomicity
-
-### Implementation Details
-
-**Write methods** (use independent sessions):
-- `start_trace()`, `end_trace()`
-- `start_span()`, `end_span()`
-- `add_event()`, `record_token_usage()`, `record_metric()`, `delete_old_traces()`
-
-**Query methods** (use request-scoped sessions):
-- `get_trace()`, `get_traces()`, `get_spans()`, etc.
-- These accept a `db: Session` parameter for RBAC/token scoping
-
-**Context managers** (create single independent session for lifecycle):
-- `trace_span()`, `trace_tool_invocation()`, `trace_a2a_request()`
-
-**Pattern**: Follows existing SQL instrumentation approach in `instrumentation/sqlalchemy.py:58-87`
-
-**Middleware**: `ObservabilityMiddleware` no longer creates `request.state.db`. Each observability operation creates its own short-lived session.
-
-**Security**: Query operations use request-scoped sessions for RBAC/token scoping. Write operations are not RBAC-protected (observability visibility is platform-wide).
-
-**Connection Pool Sizing**: The separate session pattern creates 4-6 independent database sessions per traced request (trace start/end, span start/end, metrics, events). Default configuration (`DB_POOL_SIZE=200`, `DB_MAX_OVERFLOW=10`) provides 210 total connections, supporting ~35 concurrent traced requests. This is adequate for typical deployments. High-traffic production systems (>50 req/sec sustained) should increase pool size via environment variables: `DB_POOL_SIZE=500`, `DB_MAX_OVERFLOW=100` to support 80+ concurrent requests. Monitor for "QueuePool limit exceeded" errors and adjust pool sizing accordingly. Note: SQLite connections are capped at 50 due to file-based limitations.
-
 ## Key Environment Variables
 
 Defaults come from `mcpgateway/config.py`. `.env.example` intentionally overrides a few for local/dev convenience.
@@ -368,6 +333,5 @@ When posting PR reviews, issue comments, or any public-facing text on GitHub, us
 
 - `gh` for GitHub operations
 - `make` for build/test automation
-- `uv` for virtual environment management and for `uv tool run` linter invocations
-- Dev-group tools installed in the venv: `pytest`, `mypy`, `bandit`, `pre-commit`, `prospector`, etc. (see `pyproject.toml` `[dependency-groups]`)
-- Formatters and linters (`black`, `isort`, `ruff`, `pylint`, `vulture`, `interrogate`, `radon`, `yamllint`, `tomlcheck`) are pinned in the `Makefile` and invoked on demand via `uv tool run`; always prefer the Makefile targets (`make black`, `make ruff`, `make pylint`, etc.) over calling the underlying tools directly
+- `uv` for virtual environment management
+- Standard tools: pytest, black, isort, ruff, pylint

CHANGELOG.md

@@ -68,6 +68,11 @@ conditions:
 
 **Rollback**: Keep configuration backups. Restore previous `plugins/config.yaml` if issues arise.
 
+### Changed
+
+- Consolidated the repository-owned Rust workspace under `crates/`, kept root-level `cargo build` / `cargo test` / `cargo check` support, and documented that `mcp-servers/rust/` stays outside the shared workspace for now. ([#4087](https://github.com/IBM/mcp-context-forge/pull/4087))
+- Added workspace-level Rust policy notes for contributors, tracked `cargo-vet` exemption reduction in [#4173](https://github.com/IBM/mcp-context-forge/issues/4173), and tracked Rust CI workflow factoring in [#4174](https://github.com/IBM/mcp-context-forge/issues/4174).
+
 
 
 ## [1.0.0] - 2026-03-31 - General Availability

README.md

@@ -659,6 +659,12 @@ make venv install-dev      # create .venv + install deps + build Admin UI
 make serve                 # gunicorn on :4444
 ```
 
+Rust workspace note:
+- Workspace-owned Rust crates live under `crates/` and are picked up by the root `Cargo.toml` via `crates/*`.
+- Run `cargo build`, `cargo test`, and `cargo check` from the repo root to cover the shared workspace.
+- `mcp-servers/rust/` stays outside the shared workspace on purpose and is managed separately.
+- `make venv install-dev` creates the root `.venv`, which is also reused by the workspace's PyO3/maturin builds.
+
 <details>
 <summary><strong>Alternative: UV or pip</strong></summary>
 

SECURITY.md

@@ -93,7 +93,7 @@ Our security pipeline operates at multiple levels:
 
 **Pre-commit Security Gates**: Before any code reaches our repository, it must pass through rigorous pre-commit hooks that include multiple security scanners like Bandit for common security issues, Semgrep for semantic pattern matching, Dodgy for hardcoded secrets detection, and detect-private-key for catching committed private keys, along with type checking and code quality enforcement. Pre-commit hooks also enforce **AI content integrity** (preventing AI-generated artifacts such as hallucinated citations, stock phrases, and malformed code fences) and **Unicode safety** (fixing smart quotes, ligatures, and forbidding BiDi control characters to prevent [trojan-source attacks](https://trojansource.codes/)). Developers can run `make security-all` or `make pre-commit bandit semgrep dodgy lint` locally to execute these same security checks before pushing code.
 
-**Continuous Integration Security**: Our GitHub Actions workflows implement automated security scanning on every pull request and commit, with **40+ security scans** triggering automatically on every PR, including Semgrep for semantic analysis, Gitleaks for secret detection, comprehensive dependency vulnerability scanning with pip-audit, npm audit, and cargo audit, SBOM generation, and Hadolint-style linting where configured, IaC scanning with Checkov and kube-linter, GitHub Actions security linting with Zizmor, and multi-language static analysis across Python, Go, Rust, Shell, and JavaScript.
+**Continuous Integration Security**: Our GitHub Actions workflows implement automated security scanning on pull requests and commits, with **40+ security scans** across the full CI suite, including Semgrep for semantic analysis, Gitleaks for secret detection, comprehensive dependency vulnerability scanning with pip-audit, npm audit, cargo-deny advisories, and cargo-vet, SBOM generation, and Hadolint-style linting where configured, IaC scanning with Checkov and kube-linter, GitHub Actions security linting with Zizmor, and multi-language static analysis across Python, Go, Rust, Shell, and JavaScript. Some language-specific security jobs, including the Rust policy pipeline, are path-filtered to Rust-relevant files rather than running on every repository change.
 
 **Code Review Security**: All code changes undergo mandatory peer review with security-focused review criteria, ensuring that security considerations are evaluated by human experts in addition to automated tooling.
 
@@ -109,12 +109,12 @@ Our security toolchain includes **40+ different security and quality tools**, ea
 
 - **Static Analysis Security Testing (SAST)**: CodeQL, Bandit, Semgrep, DevSkim (Microsoft security anti-patterns), and multiple type checkers
 - **Secret Detection**: Gitleaks for git history scanning, Dodgy for hardcoded secrets in code, detect-private-key for committed private keys, and Snyk custom rules for hardcoded JWT secrets and credentials (CWE-798)
-- **Dependency Vulnerability Scanning**: OSV-Scanner, pip-audit, npm audit, cargo audit (Rust), govulncheck (Go), and GitHub dependency review with license policy enforcement
+- **Dependency Vulnerability Scanning**: OSV-Scanner, pip-audit, npm audit, cargo-deny advisories (Rust), govulncheck (Go), and GitHub dependency review with license policy enforcement
 - **Container Security**: Dockerfile linting, SBOM generation, and Dockle where used
 - **Infrastructure as Code (IaC) Security**: Checkov for IaC security scanning (Dockerfiles, Helm charts, docker-compose), kube-linter for Kubernetes/Helm manifest best practices
 - **CI/CD Pipeline Security**: Zizmor for GitHub Actions workflow security linting, actionlint for workflow syntax validation
 - **Go Security**: gosec for Go static security analysis, golangci-lint with security rules, govulncheck for Go vulnerability database checking
-- **Rust Security**: cargo audit for Rust dependency vulnerability scanning, cargo clippy for Rust linting
+- **Rust Security**: cargo-deny for Rust dependency policy and advisory scanning, cargo-vet for supply-chain trust enforcement, cargo clippy for Rust linting
 - **Shell Security**: shellcheck for shell script security and correctness linting
 - **Web & Frontend Security**: ESLint, HTMLHint, Stylelint, retire.js for known-vulnerable JS library detection, nodejsscan for JavaScript/Node.js security vulnerability scanning, npm audit for package vulnerabilities
 - **Code Quality & Best Practices**: Prospector comprehensive analysis, Interrogate for docstring coverage
@@ -489,8 +489,9 @@ flowchart TD
     ML1 --> ML1B[golangci-lint - Linting]
     ML1 --> ML1C[govulncheck - Vuln DB]
 
-    ML2 --> ML2A[cargo audit - Dependency Vulns]
-    ML2 --> ML2B[cargo clippy - Linting]
+    ML2 --> ML2A[cargo-deny - Advisories & Policy]
+    ML2 --> ML2B[cargo-vet - Supply Chain Trust]
+    ML2 --> ML2C[cargo clippy - Linting]
 
     ML3 --> ML3A[shellcheck - Script Analysis]
 

docs/docs/architecture/adr/041-top-level-rust-workspace.md

@@ -1,6 +1,6 @@
 # ADR-0041: Top-Level Rust Workspace (Cargo.toml at Repository Root)
 
-- *Status:* Partially superseded — `plugins_rust/` was removed when in-tree Rust plugins migrated to standalone PyPI packages (`cpex-*`). The remaining Rust workspace members (`tools_rust/`, etc.) are unaffected.
+- *Status:* Accepted
 - *Date:* 2026-02-26
 - *Deciders:* Core Engineering Team
 
@@ -13,30 +13,34 @@ The repository is primarily Python-based with some Rust usage (e.g. plugins, too
 Adopt **Option 1: workspace at repository root**.
 
 - Add a root `Cargo.toml` defining a Rust workspace.
-- Include Rust crates as workspace members at the repository root. At the time of the decision that meant `mcpgateway_rust/`, `tools_rust/`, and `plugins_rust/`. After the plugin extraction, only the remaining in-repo Rust crates (for example `tools_rust/`) still participate in this workspace.
-- Keep the existing directory layout: Python in `mcpgateway/`, `plugins/`, etc.; Rust crates remain where they are and are referenced from the root workspace.
+- Keep the root workspace member policy simple: workspace-owned crates live under `crates/`, and the root manifest includes them via `crates/*`.
+- Keep the existing directory layout: Python stays in `mcpgateway/`, `plugins/`, and related top-level folders; the Rust workspace-owned crates live in `crates/`.
+- Keep `mcp-servers/rust/` out of the shared workspace. Those sample/test servers remain separately managed and can move out of the repository later if we give them a plugin-like distribution path.
 - PyO3/maturin bindings and CI for Rust builds and tests follow this workspace (see [#3027](https://github.com/IBM/mcp-context-forge/issues/3027) for make targets and acceptance criteria).
 
 ## Consequences
 
 ### Positive
 
 - Single `cargo build` / `cargo test` / `maturin build` at repo root for all Rust code.
+- Clear placement rule for future workspace crates: if it belongs to the shared root workspace, it goes under `crates/`.
 - Centralized dependency management and simpler CI.
 - Easier cross-crate refactors; natural place to add future Rust components.
 - **Maturin** (by default with a top-level workspace) uses the root `.venv` instead of creating venvs at lower levels—one shared Python environment and simpler dev setup.
 
 ### Negative
 
 - Rust and Python directories live side-by-side at root; language boundary is less visually isolated than a dedicated `rust/` folder.
+- Rust sample/test servers outside `crates/` need their own packaging and release handling until they move to a separate distribution model.
 
 ## Alternatives Considered
 
-- **Option 2 (dedicated `mcpgateway_rust/` as workspace root)**: Clearer language boundary but extra `cd`/Make indirection and, at the time, no single root-level workspace for plugins.
+- **Option 2 (dedicated `mcpgateway_rust/` as workspace root)**: Clearer language boundary but extra `cd`/Make indirection and no single root-level workspace.
 - **Option 3 (hybrid `rust/` folder with gateway_core boundary)**: Deferred; can be revisited if we want a stricter FFI boundary.
 - **Option 4+ (Rust as services / split repos / full rewrite)**: Out of scope for this decision.
 
 ## Related
 
 - Issue: [https://github.com/IBM/mcp-context-forge/issues/3027](https://github.com/IBM/mcp-context-forge/issues/3027)
-- **Supersedes** (build layout): [ADR-0039](039-adopt-fully-independent-plugin-crates-architecture.md)—plugin crates remained independent per ADR-0039, and while they were still in this repo they also participated in the top-level workspace.
+- Follow-up: [#4174](https://github.com/IBM/mcp-context-forge/issues/4174) tracks factoring the large Rust CI workflow into reusable building blocks.
+- **Supersedes** (build layout): [ADR-0039](039-adopt-fully-independent-plugin-crates-architecture.md) for the internal workspace-owned Rust layout. Independent Rust packages can still live outside the workspace when they need separate release metadata or lifecycles.

docs/docs/architecture/adr/042-enforce-rust-in-build-process.md

@@ -125,4 +125,4 @@ Making Rust mandatory in the build affects **new and existing contributors** who
 - [Open PRs with `rust` label](https://github.com/IBM/mcp-context-forge/pulls?q=is%3Aopen+is%3Apr+label%3Arust) — examples of main Rust components being implemented
 - ADR-0020: Multi-Format Packaging Strategy (wheels, containers, Helm)
 - ADR-0038: Experimental Rust Transport Backend (Streamable HTTP) — first main Rust component in the gateway
-- Plugin crates (e.g. `plugins_rust/`) are out of scope for this ADR; they have their own packaging and optional/required policy.
+- Plugin crates and Rust MCP servers with their own packaging lifecycle are out of scope for this ADR; they keep their own optional/required policy.

docs/docs/architecture/adr/043-rust-mcp-runtime-sidecar-mode-model.md

@@ -194,5 +194,5 @@ the documented operator model is the high-level mode switch above.
 
 - [Rust MCP Runtime Architecture](../rust-mcp-runtime.md)
 - [Performance Architecture](../performance-architecture.md)
-- `tools_rust/mcp_runtime/TESTING-DESIGN.md` in the repository
-- `tools_rust/mcp_runtime/README.md` in the repository
+- `crates/mcp_runtime/TESTING-DESIGN.md` in the repository
+- `crates/mcp_runtime/README.md` in the repository

docs/docs/architecture/explorer.html

@@ -4821,10 +4821,11 @@ <h3>Testing Matrix</h3>
     { name:"Docker Security Scan", file:"docker-scan.yml", desc:"Build locally, generate SBOM with Syft, upload artifact.", triggers:["push","PR"] },
     { name:"Helm Chart Publish", file:"helm-publish.yml", desc:"Lint mcp-stack chart, publish to OCI registry on release.", triggers:["push","PR","release"] },
     { name:"Python Package Build", file:"python-package.yml", desc:"Build sdist and wheel (make dist) on code changes.", triggers:["push","PR"] },
+    { name:"Pytest With Rust", file:"pytest-rust.yml", desc:"Full pytest suite with Rust build/install enabled for Rust-owned changes.", triggers:["push","PR"] },
     { name:"JavaScript Tests (Vitest)", file:"vitest.yml", desc:"Vitest unit tests for browser JavaScript (jsdom environment).", triggers:["push","PR"] },
     { name:"Web Lint", file:"lint-web.yml", desc:"ESLint, HTMLHint, Stylelint in separate matrix jobs.", triggers:["push","PR"] },
     { name:"Playwright CI Smoke", file:"playwright.yml", desc:"UI automation smoke tests against live gateway.", triggers:["push","PR","manual"] },
-    { name:"Rust Tools CI/CD", file:"rust-tools.yml", desc:"Cargo test and clippy for tools_rust/ and mcp-servers/rust/.", triggers:["push"] },
+    { name:"Rust CI", file:"rust.yml", desc:"Unified Rust workspace CI for build, lint, tests, wheels, audit, coverage, and docs.", triggers:["push","PR","manual"] },
     { name:"Alembic Upgrade Validation", file:"alembic-upgrade-validation.yml", desc:"Validate DB migration paths (upgrade testing).", triggers:["PR","push","manual"] },
     { name:"License Check", file:"license-check.yml", desc:"Scan for license compliance on dependency changes.", triggers:["push","PR"] },
     { name:"Full Linting", file:"linting-full.yml", desc:"Comprehensive linting pass on main branch.", triggers:["push","manual"] },

docs/docs/architecture/rust-mcp-runtime.md

@@ -116,7 +116,7 @@ make test-mcp-session-isolation
 ```
 
 See the detailed threat model and test matrix in
-`tools_rust/mcp_runtime/TESTING-DESIGN.md` in the repository.
+`crates/mcp_runtime/TESTING-DESIGN.md` in the repository.
 
 ## Verification
 
@@ -244,7 +244,7 @@ make testing-rebuild-rust-full
 make test-mcp-cli
 make test-mcp-rbac
 make test-mcp-session-isolation
-cargo test --release --manifest-path tools_rust/mcp_runtime/Cargo.toml
+cargo test --release --manifest-path crates/mcp_runtime/Cargo.toml
 ```
 
 Recommended benchmark wrappers:
@@ -257,4 +257,4 @@ make benchmark-mcp-tools-300
 ```
 
 For Rust-local profiling and crate-level lint/test helpers, see
-`tools_rust/mcp_runtime/README.md` in the repository.
+`crates/mcp_runtime/README.md` in the repository.