refactor: consolidate rust workspace under crates

[lucarlig]

✨ Feature / Enhancement PR

🔗 Epic / Issue

Closes #3027

---

🚀 Summary (1-2 sentences)

This PR finishes the Rust workspace move under crates/, adds the new local PyO3 native extension crate, and updates CI, supply-chain policy, docs, and container build paths to match the current layout. It also restores full Rust validation on non-draft PRs and adds regression tests so the repo keeps the intended crate, workflow, and container boundaries.

---

🧪 Checks

• CI fully green
• Local workflow guard tests updated
• CHANGELOG updated (if user-facing)

---

📓 Notes (optional)

• Rust crates now live directly under crates/:
  • crates/mcp_runtime
  • crates/wrapper
  • crates/request_logging_masking_native_extension
• Standalone Rust MCP servers remain outside the root workspace. Their manifests and license handling were adjusted, but they are not workspace members.
• Rust CI now runs the full non-draft PR matrix instead of the reduced pr-* hot path.
• rust fmt and rust clippy now run on Ubuntu only; build/test coverage remains on both Ubuntu and macOS.
• License, deny, and vet policy were updated for the new Rust layout and the new native extension dependency tree.
• Containerfile.lite now:
  • builds local native extensions dynamically from crates/** when a crate has Cargo.toml + pyproject.toml + maturin
  • builds Rust sidecars dynamically from crates/** bin targets
  • uses a UBI 10 Rust builder path instead of the earlier x86-only manylinux builder
  • has a dedicated Rust-enabled CI smoke build so ENABLE_RUST=true is validated on PRs
• Wrapper hardening in this branch includes:
  • redacted URL credentials in debug config output
  • no raw MCP payload logging at debug
  • bounded internal queues instead of unbounded channels
• Guard tests were added/updated for:
  • workspace layout
  • Rust workflow behavior
  • docker scan workflow coverage for Rust-enabled container builds
  • wrapper workflow triggers
  • license-check workflow triggers
• Docs and ADRs were updated to reflect the new crate layout, CI shape, and runtime/container behavior.

[dima-zakharov]

Suggestions (Recommended Improvements)

1. Missing rust-toolchain.toml file

File: Cargo.toml:2 (workspace root)

Issue: The workspace uses resolver = "3" which requires Rust 1.85+, and mcp_runtime/Cargo.toml specifies rust-version = "1.85", but there's no rust-toolchain.toml file in the repository. The CI workflow references it in path triggers but the file doesn't exist.

Impact: Developers and CI may use older Rust versions, causing workspace manifest parsing failures. The MANIFEST.in references it but the file is missing.

Suggested fix: Add a rust-toolchain.toml file:

[toolchain]
channel = "1.85"
components = ["rustfmt", "clippy"]

[lucarlig]

Addressed.

Added a workspace-level rust-toolchain.toml at the repo root so the workspace and CI stop referring to a missing file.

I used:

[toolchain]
channel = "stable"
components = ["rustfmt", "clippy"]

instead of pinning 1.85, since the goal here is to stay on stable while still satisfying the workspace minimum Rust version requirement from rust-version = "1.85" and resolver = "3".

[lucarlig]

Supply-chain update for the Rust workspace CI.

I wired in cargo vet as the strict Rust supply-chain gate and took the dependency graph through three stages:

Stage	What was enabled	Result
1	Public imported audit sets only (mozilla, google, bytecode-alliance, zcash, embark-studios, ariel-os, isrg)	304 unvetted crates
2	Public imports + blanket maintainer trust for high-coverage ecosystems	reduced to 100 unvetted crates
3	Temporary exemptions for the remaining unresolved crates	cargo vet check passes with 299 fully audited, 12 partially audited, 88 exempted

A bit more context on stage 2: I added maintainer trust only where it was clearly high leverage across the graph, for example shared ecosystem maintainers around clap/toml, tokio, tracing, rustls, prost/tonic, opentelemetry, redis, rmcp, Windows crates, and similar families.

That got us a large reduction quickly, but it also has an obvious limitation: blanket maintainer trust is weaker than crate-by-crate audits. It is a pragmatic middle ground for CI hardening, not the end state.

The remaining debt is now explicit in supply-chain/config.toml as temporary exemptions. Those 88 exemptions are the non-audited residue we still need to burn down over time by replacing them with one of:

1. more upstream imported audits
2. narrower trust entries where justified
3. local crate audits/certifications

So the current state is:

• Rust CI is enforceable and green
• public audit imports are in place
• high-value trust entries are in place
• the remaining unaudited set is tracked explicitly instead of being silent

[lucarlig]

Follow-up on the Rust security tooling split:

I also removed cargo-audit from Rust CI.

Reason: cargo-audit and cargo-deny check advisories both consume the RustSec advisory database for Rust dependency vulnerability scanning, so keeping both as required CI gates was duplicate work and duplicate noise.

The current split is now:

• cargo-deny: advisories, bans, sources, licenses
• cargo-vet: supply-chain trust / audit policy
• cargo-clippy: linting

So advisory coverage is still enforced, but it now happens through cargo-deny instead of running a second RustSec gate with cargo-audit.

[jonpspri]

Can we double-check the numbering of the ADRs? Feels like we may have some overlap.

[lucarlig]

Can we double-check the numbering of the ADRs? Feels like we may have some overlap.

don't see any overlap in the ADR folder? this PR is just updating some old ADRs not creating new ones with correct paths.

[dima-zakharov]

I am happy we can run cargo test, cargo check , etc from the root folder to check everything. Must have option.

Some things to consider later:

#	Severity	Description	Recommendation
1	Medium	Root Cargo.toml glob crates/* auto-includes all crates	Consider explicit members = ["crates/mcp_runtime", "crates/wrapper", "crates/request_logging_masking_native_extension"] or document the glob policy
2	Low	88 cargo-vet exemptions	Track as a follow-up issue; aim to reduce over next sprint(s)
3	Low	deny.toml allowlist is long	Audit for unused entries; unused-allowed-license = "allow" makes this non-blocking
4	Low	Removed Makefile venv policy comments	Add a shorter note back for contributor onboarding
5	Low	CHANGELOG not updated	Add entry for Rust workspace consolidation
6	Info	mcp-servers/rust/* excluded from workspace	Document rationale in ADR-041 or root Cargo.toml comment
7	Low	rust.yml at 668 lines	Factor reusable steps into composite actions for maintainability

[lucarlig]

Addressed.

1. Added docs for the workspace policy:

   • root Cargo.toml now has an explicit comment that repo-owned workspace crates live under crates/ and that the crates/* glob is intentional
   • README.md now documents that root-level cargo build / cargo test / cargo check cover the shared workspace and that the root .venv is reused for PyO3/maturin builds

2. Opened a tracking issue for the remaining cargo-vet exemptions and linked it here:

   • [CHORE]: Reduce cargo-vet exemptions in the Rust workspace #4173 [CHORE]: Reduce cargo-vet exemptions in the Rust workspace #4173

3. Added a comment in deny.toml to make the intent explicit:

   • the allowlist is intentionally broad because it represents the IBM-approved set plus ContextForge-specific extensions
   • it should not be shortened opportunistically just because the current dependency graph only uses a subset

4. Added docs in ADR-041 as well:

   • clarified that workspace-owned crates belong under crates/
   • documented that mcp-servers/rust/ is intentionally outside the shared workspace

5. Updated CHANGELOG.md with the Rust workspace consolidation and the follow-up tracking issues

6. Updated ADR-041 to match the current direction:

   • mcp-servers/rust/ is not part of the shared workspace
   • noted that those sample/test servers can move out later if we give them a plugin-like distribution path

7. Opened a tracking issue for the large Rust workflow follow-up:

   • [CHORE]: Refactor Rust CI workflow into reusable building blocks #4174 [CHORE]: Refactor Rust CI workflow into reusable building blocks #4174