docs: clarify rust workspace policy and follow-ups

Signed-off-by: lucarlig <luca.carlig@ibm.com>

Author: lucarlig

AGENTS.md

@@ -8,7 +8,7 @@ For domain-specific guidance, see subdirectory AGENTS.md files:
 - `charts/AGENTS.md` - Helm chart operations
 - `docs/AGENTS.md` - Documentation authoring
 - `mcp-servers/AGENTS.md` - MCP server implementation
-- `tools_rust/mcp_runtime/DEVELOPING.md` - Rust MCP runtime development workflows, command matrix, and validation
+- `crates/mcp_runtime/DEVELOPING.md` - Rust MCP runtime development workflows, command matrix, and validation
 
 **Note:** The `llms/` directory contains guidance for LLMs *using* ContextForge solution (end-user runtime guidance), not for code agents working on this codebase.
 
@@ -38,7 +38,7 @@ charts/                     # Helm charts (see charts/AGENTS.md)
 docs/                       # Architecture and usage documentation (see docs/AGENTS.md)
 a2a-agents/                 # A2A agent implementations (used for testing/examples)
 mcp-servers/                # MCP server templates (see mcp-servers/AGENTS.md)
-tools_rust/                 # Rust utilities and MCP runtime (see tools_rust/mcp_runtime/DEVELOPING.md)
+crates/                     # Direct Rust crate folders (runtime and wrapper)
 llms/                       # End-user LLM guidance (not for code agents)
 ```
 
@@ -48,9 +48,8 @@ llms/                       # End-user LLM guidance (not for code agents)
 ```bash
 cp .env.example .env && make install-dev check-env    # Complete setup
 make venv                          # Create virtual environment with uv
-make install-dev                   # Install with dev dependencies (includes build-ui)
+make install-dev                   # Install with dev dependencies
 make check-env                     # Verify .env against .env.example
-make build-ui                      # Rebuild Admin UI JS bundle (requires npm)
 ```
 
 ### Development
@@ -66,11 +65,10 @@ make serve-ssl                    # HTTPS on :4444 (creates certs if needed)
 make autoflake isort black pre-commit
 
 # Before committing, use ty, mypy and pyrefly to check just the new files, then run:
-make ruff bandit interrogate pylint verify
+make flake8 bandit interrogate pylint verify
 
-# Before committing Rust changes (tools_rust/):
-# Run fmt-check, clippy -D warnings, and cargo test for Rust crates
-cd tools_rust/mcp_runtime && cargo fmt --check && cargo clippy -- -D warnings && cargo test
+# Before committing Rust changes (crates/ or tools_rust/):
+make rust-check                   # Runs fmt-check, clippy -D warnings, and cargo test for all Rust crates
 ```
 
 ## Authentication & RBAC Overview
@@ -137,39 +135,6 @@ ContextForge implements a **two-layer security model**:
 - **Multi-tenancy architecture**: `docs/docs/architecture/multitenancy.md`
 - **OAuth token delegation**: `docs/docs/architecture/oauth-design.md`
 
-## Observability Transaction Behavior
-
-**Issue #3883 - Separate Session Pattern**
-
-Observability write operations use **independent database sessions** that commit immediately (best-effort pattern). This means:
-
-- Observability data persists even when the main request fails
-- Traces may show "in progress" or partial states for failed requests
-- **NOT atomic** with main request transaction (intentional trade-off)
-- Provides visibility into partial failures at the cost of atomicity
-
-### Implementation Details
-
-**Write methods** (use independent sessions):
-- `start_trace()`, `end_trace()`
-- `start_span()`, `end_span()`
-- `add_event()`, `record_token_usage()`, `record_metric()`, `delete_old_traces()`
-
-**Query methods** (use request-scoped sessions):
-- `get_trace()`, `get_traces()`, `get_spans()`, etc.
-- These accept a `db: Session` parameter for RBAC/token scoping
-
-**Context managers** (create single independent session for lifecycle):
-- `trace_span()`, `trace_tool_invocation()`, `trace_a2a_request()`
-
-**Pattern**: Follows existing SQL instrumentation approach in `instrumentation/sqlalchemy.py:58-87`
-
-**Middleware**: `ObservabilityMiddleware` no longer creates `request.state.db`. Each observability operation creates its own short-lived session.
-
-**Security**: Query operations use request-scoped sessions for RBAC/token scoping. Write operations are not RBAC-protected (observability visibility is platform-wide).
-
-**Connection Pool Sizing**: The separate session pattern creates 4-6 independent database sessions per traced request (trace start/end, span start/end, metrics, events). Default configuration (`DB_POOL_SIZE=200`, `DB_MAX_OVERFLOW=10`) provides 210 total connections, supporting ~35 concurrent traced requests. This is adequate for typical deployments. High-traffic production systems (>50 req/sec sustained) should increase pool size via environment variables: `DB_POOL_SIZE=500`, `DB_MAX_OVERFLOW=100` to support 80+ concurrent requests. Monitor for "QueuePool limit exceeded" errors and adjust pool sizing accordingly. Note: SQLite connections are capped at 50 due to file-based limitations.
-
 ## Key Environment Variables
 
 Defaults come from `mcpgateway/config.py`. `.env.example` intentionally overrides a few for local/dev convenience.
@@ -368,6 +333,5 @@ When posting PR reviews, issue comments, or any public-facing text on GitHub, us
 
 - `gh` for GitHub operations
 - `make` for build/test automation
-- `uv` for virtual environment management and for `uv tool run` linter invocations
-- Dev-group tools installed in the venv: `pytest`, `mypy`, `bandit`, `pre-commit`, `prospector`, etc. (see `pyproject.toml` `[dependency-groups]`)
-- Formatters and linters (`black`, `isort`, `ruff`, `pylint`, `vulture`, `interrogate`, `radon`, `yamllint`, `tomlcheck`) are pinned in the `Makefile` and invoked on demand via `uv tool run`; always prefer the Makefile targets (`make black`, `make ruff`, `make pylint`, etc.) over calling the underlying tools directly
+- `uv` for virtual environment management
+- Standard tools: pytest, black, isort, ruff, pylint

CHANGELOG.md

@@ -60,6 +60,11 @@ conditions:
 
 **Rollback**: Keep configuration backups. Restore previous `plugins/config.yaml` if issues arise.
 
+### Changed
+
+- Consolidated the repository-owned Rust workspace under `crates/`, kept root-level `cargo build` / `cargo test` / `cargo check` support, and documented that `mcp-servers/rust/` stays outside the shared workspace for now. ([#4087](https://github.com/IBM/mcp-context-forge/pull/4087))
+- Added workspace-level Rust policy notes for contributors, tracked `cargo-vet` exemption reduction in [#4173](https://github.com/IBM/mcp-context-forge/issues/4173), and tracked Rust CI workflow factoring in [#4174](https://github.com/IBM/mcp-context-forge/issues/4174).
+
 #### **👥 `MAX_MEMBERS_PER_TEAM` No Longer Baked Into Team Rows** ([#3682](https://github.com/IBM/mcp-context-forge/pull/3682), [#3588](https://github.com/IBM/mcp-context-forge/issues/3588))
 
 **Action Required**: New teams now store `NULL` for `max_members` and resolve the limit at check time from the `MAX_MEMBERS_PER_TEAM` environment variable. Existing teams created before this change still have the old default baked into the DB and will **not** automatically pick up env var changes.

README.md

@@ -658,6 +658,12 @@ make venv install-dev      # create .venv + install deps + build Admin UI
 make serve                 # gunicorn on :4444
 ```
 
+Rust workspace note:
+- Workspace-owned Rust crates live under `crates/` and are picked up by the root `Cargo.toml` via `crates/*`.
+- Run `cargo build`, `cargo test`, and `cargo check` from the repo root to cover the shared workspace.
+- `mcp-servers/rust/` stays outside the shared workspace on purpose and is managed separately.
+- `make venv install-dev` creates the root `.venv`, which is also reused by the workspace's PyO3/maturin builds.
+
 <details>
 <summary><strong>Alternative: UV or pip</strong></summary>
 

docs/docs/architecture/adr/041-top-level-rust-workspace.md

@@ -1,6 +1,6 @@
 # ADR-0041: Top-Level Rust Workspace (Cargo.toml at Repository Root)
 
-- *Status:* Partially superseded — `plugins_rust/` was removed when in-tree Rust plugins migrated to standalone PyPI packages (`cpex-*`). The remaining Rust workspace members (`tools_rust/`, etc.) are unaffected.
+- *Status:* Accepted
 - *Date:* 2026-02-26
 - *Deciders:* Core Engineering Team
 
@@ -13,30 +13,34 @@ The repository is primarily Python-based with some Rust usage (e.g. plugins, too
 Adopt **Option 1: workspace at repository root**.
 
 - Add a root `Cargo.toml` defining a Rust workspace.
-- Include Rust crates as workspace members at the repository root. At the time of the decision that meant `mcpgateway_rust/`, `tools_rust/`, and `plugins_rust/`. After the plugin extraction, only the remaining in-repo Rust crates (for example `tools_rust/`) still participate in this workspace.
-- Keep the existing directory layout: Python in `mcpgateway/`, `plugins/`, etc.; Rust crates remain where they are and are referenced from the root workspace.
+- Keep the root workspace member policy simple: workspace-owned crates live under `crates/`, and the root manifest includes them via `crates/*`.
+- Keep the existing directory layout: Python stays in `mcpgateway/`, `plugins/`, and related top-level folders; the Rust workspace-owned crates live in `crates/`.
+- Keep `mcp-servers/rust/` out of the shared workspace. Those sample/test servers remain separately managed and can move out of the repository later if we give them a plugin-like distribution path.
 - PyO3/maturin bindings and CI for Rust builds and tests follow this workspace (see [#3027](https://github.com/IBM/mcp-context-forge/issues/3027) for make targets and acceptance criteria).
 
 ## Consequences
 
 ### Positive
 
 - Single `cargo build` / `cargo test` / `maturin build` at repo root for all Rust code.
+- Clear placement rule for future workspace crates: if it belongs to the shared root workspace, it goes under `crates/`.
 - Centralized dependency management and simpler CI.
 - Easier cross-crate refactors; natural place to add future Rust components.
 - **Maturin** (by default with a top-level workspace) uses the root `.venv` instead of creating venvs at lower levels—one shared Python environment and simpler dev setup.
 
 ### Negative
 
 - Rust and Python directories live side-by-side at root; language boundary is less visually isolated than a dedicated `rust/` folder.
+- Rust sample/test servers outside `crates/` need their own packaging and release handling until they move to a separate distribution model.
 
 ## Alternatives Considered
 
-- **Option 2 (dedicated `mcpgateway_rust/` as workspace root)**: Clearer language boundary but extra `cd`/Make indirection and, at the time, no single root-level workspace for plugins.
+- **Option 2 (dedicated `mcpgateway_rust/` as workspace root)**: Clearer language boundary but extra `cd`/Make indirection and no single root-level workspace.
 - **Option 3 (hybrid `rust/` folder with gateway_core boundary)**: Deferred; can be revisited if we want a stricter FFI boundary.
 - **Option 4+ (Rust as services / split repos / full rewrite)**: Out of scope for this decision.
 
 ## Related
 
 - Issue: [https://github.com/IBM/mcp-context-forge/issues/3027](https://github.com/IBM/mcp-context-forge/issues/3027)
-- **Supersedes** (build layout): [ADR-0039](039-adopt-fully-independent-plugin-crates-architecture.md)—plugin crates remained independent per ADR-0039, and while they were still in this repo they also participated in the top-level workspace.
+- Follow-up: [#4174](https://github.com/IBM/mcp-context-forge/issues/4174) tracks factoring the large Rust CI workflow into reusable building blocks.
+- **Supersedes** (build layout): [ADR-0039](039-adopt-fully-independent-plugin-crates-architecture.md) for the internal workspace-owned Rust layout. Independent Rust packages can still live outside the workspace when they need separate release metadata or lifecycles.

docs/docs/architecture/adr/042-enforce-rust-in-build-process.md

@@ -125,4 +125,4 @@ Making Rust mandatory in the build affects **new and existing contributors** who
 - [Open PRs with `rust` label](https://github.com/IBM/mcp-context-forge/pulls?q=is%3Aopen+is%3Apr+label%3Arust) — examples of main Rust components being implemented
 - ADR-0020: Multi-Format Packaging Strategy (wheels, containers, Helm)
 - ADR-0038: Experimental Rust Transport Backend (Streamable HTTP) — first main Rust component in the gateway
-- Plugin crates (e.g. `plugins_rust/`) are out of scope for this ADR; they have their own packaging and optional/required policy.
+- Plugin crates and Rust MCP servers with their own packaging lifecycle are out of scope for this ADR; they keep their own optional/required policy.

docs/docs/architecture/adr/043-rust-mcp-runtime-sidecar-mode-model.md

@@ -194,5 +194,5 @@ the documented operator model is the high-level mode switch above.
 
 - [Rust MCP Runtime Architecture](../rust-mcp-runtime.md)
 - [Performance Architecture](../performance-architecture.md)
-- `tools_rust/mcp_runtime/TESTING-DESIGN.md` in the repository
-- `tools_rust/mcp_runtime/README.md` in the repository
+- `crates/mcp_runtime/TESTING-DESIGN.md` in the repository
+- `crates/mcp_runtime/README.md` in the repository

docs/docs/architecture/explorer.html

@@ -4821,10 +4821,11 @@ <h3>Testing Matrix</h3>
     { name:"Docker Security Scan", file:"docker-scan.yml", desc:"Build locally, generate SBOM with Syft, upload artifact.", triggers:["push","PR"] },
     { name:"Helm Chart Publish", file:"helm-publish.yml", desc:"Lint mcp-stack chart, publish to OCI registry on release.", triggers:["push","PR","release"] },
     { name:"Python Package Build", file:"python-package.yml", desc:"Build sdist and wheel (make dist) on code changes.", triggers:["push","PR"] },
+    { name:"Pytest With Rust", file:"pytest-rust.yml", desc:"Full pytest suite with Rust build/install enabled for Rust-owned changes.", triggers:["push","PR"] },
     { name:"JavaScript Tests (Vitest)", file:"vitest.yml", desc:"Vitest unit tests for browser JavaScript (jsdom environment).", triggers:["push","PR"] },
     { name:"Web Lint", file:"lint-web.yml", desc:"ESLint, HTMLHint, Stylelint in separate matrix jobs.", triggers:["push","PR"] },
     { name:"Playwright CI Smoke", file:"playwright.yml", desc:"UI automation smoke tests against live gateway.", triggers:["push","PR","manual"] },
-    { name:"Rust Tools CI/CD", file:"rust-tools.yml", desc:"Cargo test and clippy for tools_rust/ and mcp-servers/rust/.", triggers:["push"] },
+    { name:"Rust CI", file:"rust.yml", desc:"Unified Rust workspace CI for build, lint, tests, wheels, audit, coverage, and docs.", triggers:["push","PR","manual"] },
     { name:"Alembic Upgrade Validation", file:"alembic-upgrade-validation.yml", desc:"Validate DB migration paths (upgrade testing).", triggers:["PR","push","manual"] },
     { name:"License Check", file:"license-check.yml", desc:"Scan for license compliance on dependency changes.", triggers:["push","PR"] },
     { name:"Full Linting", file:"linting-full.yml", desc:"Comprehensive linting pass on main branch.", triggers:["push","manual"] },

docs/docs/architecture/rust-mcp-runtime.md

@@ -116,7 +116,7 @@ make test-mcp-session-isolation
 ```
 
 See the detailed threat model and test matrix in
-`tools_rust/mcp_runtime/TESTING-DESIGN.md` in the repository.
+`crates/mcp_runtime/TESTING-DESIGN.md` in the repository.
 
 ## Verification
 
@@ -244,7 +244,7 @@ make testing-rebuild-rust-full
 make test-mcp-cli
 make test-mcp-rbac
 make test-mcp-session-isolation
-cargo test --release --manifest-path tools_rust/mcp_runtime/Cargo.toml
+cargo test --release --manifest-path crates/mcp_runtime/Cargo.toml
 ```
 
 Recommended benchmark wrappers:
@@ -257,4 +257,4 @@ make benchmark-mcp-tools-300
 ```
 
 For Rust-local profiling and crate-level lint/test helpers, see
-`tools_rust/mcp_runtime/README.md` in the repository.
+`crates/mcp_runtime/README.md` in the repository.

docs/docs/best-practices/mcp-best-practices.md

@@ -30,7 +30,7 @@ Make targets are grouped by functionality. Use `make help` to see them all in yo
 | Target               | Description |
 |----------------------|-------------|
 | `make serve`         | Run the MCP server locally (e.g., `mcp-time-server`). |
-| `make test`          | Run all unit and integration tests with `pytest`. |
+| `make test`          | Run all unit and integration tests with `pytest`. Use `pytest -k "<name>" tests/unit/` for targeted runs. |
 | `make test-curl`     | Run public API integration tests using a `curl` script. |
 
 #### 📚 DOCUMENTATION & SBOM

docs/docs/development/profiling.md

@@ -243,15 +243,15 @@ py-spy record -o flamegraph.svg -- python -m mcpgateway
 For Rust-local profiling of the MCP runtime crate:
 
 ```bash
-make -C tools_rust/mcp_runtime setup-profiling
-make -C tools_rust/mcp_runtime flamegraph-test
-make -C tools_rust/mcp_runtime flamegraph-test-rmcp
+make -C crates/mcp_runtime setup-profiling
+make -C crates/mcp_runtime flamegraph-test
+make -C crates/mcp_runtime flamegraph-test-rmcp
 ```
 
 These targets generate flamegraphs under:
 
 ```text
-tools_rust/mcp_runtime/profiles/
+crates/mcp_runtime/profiles/
 ```
 
 Use them to inspect Rust-internal startup and hot-path behavior in the runtime