Add NumGuardTool: verify a number before an agent asserts it - #6630
Add NumGuardTool: verify a number before an agent asserts it#6630ipezygj wants to merge 2 commits into
Conversation
📝 WalkthroughWalkthroughChangesNumGuardTool is added with a Pydantic input schema, dynamic NumGuard tool integration
Sequence Diagram(s)sequenceDiagram
participant Agent
participant NumGuardTool
participant NumGuard
Agent->>NumGuardTool: run(kind, params)
NumGuardTool->>NumGuard: guard.check(kind, **params)
NumGuard-->>NumGuardTool: verdict payload
NumGuardTool-->>Agent: JSON result
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py (1)
18-27: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the public input model.
NumGuardToolInputis part of the tool contract but has no class docstring. Add a short model-level docstring; keep the field descriptions for schema generation.As per coding guidelines, public APIs and complex logic in Python code must be documented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py` around lines 18 - 27, Add a concise class-level docstring to the public NumGuardToolInput model describing its purpose and role as the NumGuard tool input contract. Preserve the existing kind and params field descriptions unchanged for schema generation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py`:
- Around line 55-62: Update the failure handling in NumGuardTool’s verification
flow to return JSON for both missing-dependency and malformed-claim paths,
matching the documented error envelope. Clearly mark failures as unverified so
they cannot be confused with a verified result where survives is false, while
preserving the existing success response shape and error details.
- Around line 51-54: Update the dependency import and invocation in the NumGuard
tool to use the supported top-level numguard.verify_claim API instead of
numguard.guard.check. Preserve the existing ImportError response for missing
numguard and pass the tool’s claim input through the documented API.
---
Nitpick comments:
In `@lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py`:
- Around line 18-27: Add a concise class-level docstring to the public
NumGuardToolInput model describing its purpose and role as the NumGuard tool
input contract. Preserve the existing kind and params field descriptions
unchanged for schema generation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 41d1c982-177e-4e07-becb-90ef3a99fee2
📒 Files selected for processing (4)
lib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/numguard_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py
| try: | ||
| from numguard.guard import check | ||
| except ImportError: | ||
| return "numguard is not installed. Run: pip install numguard" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
python - <<'PY'
from numguard import verify_claim
result = verify_claim("backtest", sr=0.12, T=250, n_trials=100)
assert isinstance(result, dict)
assert "survives" in result
print(result)
PYRepository: crewAIInc/crewAI
Length of output: 275
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file context =="
sed -n '1,120p' lib/crewai-tools/src/crewapi_tools/tools/numguard_tool/numguard_tool.py 2>/dev/null || sed -n '1,120p' lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py
echo
echo "== locate numguard_tool.py =="
fd -a 'numguard_tool\.py$' . || true
echo
echo "== package metadata clues for numguard =="
rg -n "numguard|numguard_tool|guard\.check|verify_claim|from numguard" pyproject.toml requirements* setup.py setup.cfg poe.lock pnpm-lock . 2>/dev/null | head -120 || true
echo
echo "== installed numguard probe =="
python3 - <<'PY'
import subprocess, json, sys
for pkg in ["numguard"]:
r = subprocess.run([sys.executable, "-m", "pip", "show", pkg], capture_output=True, text=True)
print(pkg, "available:", r.returncode == 0)
if r.returncode == 0:
print(r.stdout.strip())
print(r.stderr.strip())
PYRepository: crewAIInc/crewAI
Length of output: 4064
🌐 Web query:
numguard python package verify_claim import numguard.guard check
💡 Result:
There is no single Python package that combines the specific import numguard.guard and a verify_claim function as described. Several different projects have names similar to or associated with the term "numguard" or claim-verification functionality, which may be causing confusion. 1. NumGuard (Numeric Integrity): The term "NumGuard" refers to a numeric integrity utility used in the doc2dataset project [1][2][3]. It is primarily implemented in Rust [4][2], with Python bindings available via the three_dcf_py crate [1][3]. It is used for tracking and hashing numeric data in document pipelines to detect corruption [2][3], rather than acting as a general-purpose claim verification library. 2. numguard (PyPI package): There is a Python package named numguard on PyPI (version 0.1.1) [5]. It includes functions such as verify_claim, issue_receipt, and verify_receipt [5]. Based on its documentation, this package is designed for verifying statistical claims (e.g., related to the "Deflated Sharpe Ratio") and handling cryptographically signed receipts for agentic workflows [5]. It is not related to the "NumGuard" numeric integrity tool mentioned above. 3. NuGuard (Security Suite): There is also a distinct, unrelated project called nuguard (NuGuard AI), which is an AI application security toolkit used for SBOM generation, red-teaming, and behavioral validation [6][7][8]. It does not use the import numguard.guard. If you are working with the numguard package from PyPI, the correct imports are typically from numguard import verify_claim [5]. If you are looking for claim verification in other agentic contexts, other libraries such as multimodal-evidence-sdk (which also provides a verify_claim function) [9] or verity-guard [10] offer similar but distinct functionality.
Citations:
- 1: https://github.com/3DCF-Labs/doc2dataset
- 2: https://dev.to/yevh/i-built-an-open-source-pipeline-to-convert-documents-into-llm-training-data-37pb
- 3: https://users.rust-lang.org/t/3dcf-doc2dataset-rust-based-document-dataset-pipeline-30-formats-token-compression-numeric-integrity/136773
- 4: https://docs.rs/three-dcf-core/latest/three_dcf_core/struct.NumGuard.html
- 5: https://pypi.org/project/numguard/
- 6: https://nuguardai.github.io/nuguard/
- 7: https://pypi.org/project/nuguard/
- 8: https://pypi.org/project/nuguard/0.7.4/
- 9: https://pypi.org/project/multimodal-evidence-sdk/
- 10: https://pypi.org/project/verity-guard/
🌐 Web query:
PyPI numguard 0.1.1 source __init__.py verify_claim
💡 Result:
In the context of the PyPI package numguard (version 0.1.1), the function verify_claim is a tool used for validating statistical claims, such as those related to backtesting performance (e.g., Deflated Sharpe Ratio) [1][2]. It is designed to be used in conjunction with other functions like issue_receipt and verify_receipt to create and verify signed, tamper-evident records of claim verdicts [1][2]. Example usage from the numguard library: from numguard import verify_claim, issue_receipt, verify_receipt, keypair priv, pub = keypair result = verify_claim("backtest", sr=0.12, T=250, n_trials=100) receipt = issue_receipt(result, priv, pub) verify_receipt(receipt) # Returns True if the signed receipt is valid Note that while there is an Elixir library called Guardian that also defines a verify_claim callback for token authentication [3][4][5], it is distinct from the Python package numguard [1][2]. The mention of init.py typically refers to the package's entry point where these functions are exposed for import [1][2].
Citations:
- 1: https://pypi.org/project/numguard/0.1.1/
- 2: https://pypi.org/project/numguard/
- 3: https://hexdocs.pm/guardian/Guardian.Token.Verify.html
- 4: https://guardian.hexdocs.pm/Guardian.Token.Jwt.html
- 5: https://github.com/ueberauth/guardian/blob/master/lib/guardian/token/jwt/verify.ex
🌐 Web query:
raw github ipezygj numguard master numguard __init__.py verify_claim
💡 Result:
The Python package numguard by ipezygj, available on PyPI [1], is a verification layer designed for the AI agent economy [1]. It functions as an agent-callable MCP (Model Context Protocol) server [1] that allows agents to check numeric claims, such as eval scores or backtest results, before asserting them [1]. Regarding the specific reference to a verify_claim function in numguard, the numguard package is primarily structured as an MCP server [1]. While the project is intended to provide tools for agents to verify numbers [1], the specific implementation of verify_claim (or similar tools) within the Python codebase follows the MCP architecture. Agents interact with these verification functions by calling them through the MCP interface [1]. It is important to distinguish this Python-based numguard (created by ipezygj) [1] from the "NumGuard" numeric integrity system found in the 3DCF-Labs/doc2dataset project [2]. The 3DCF-Labs implementation is a Rust-native system [2] that stores NumGuard records as structs containing numeric coordinates and SHA-1 hashes to detect data corruption in documents [3][4]. These are two distinct projects that happen to share the same name [1][2]. For the ipezygj/numguard library, you can install and use it as follows [1]: pip install git+https://github.com/ipezygj/numguard python -m numguard.mcp_server Agents can then invoke verification tools exposed by this server to obtain signed receipts for their numeric claims [1]. For specific API details, including the exact naming of functions like verify_claim, you should consult the source code in the numguard/init.py file within the ipezygj/numguard GitHub repository.
Citations:
- 1: https://pypi.org/project/numguard/
- 2: https://github.com/3dcf-labs/doc2dataset
- 3: https://github.com/3DCF-Labs/doc2dataset/blob/main/3dcf_doc2dataset_spec.md
- 4: https://docs.rs/three-dcf-core/latest/three_dcf_core/struct.NumGuard.html
🌐 Web query:
github ipezygj numguard numguard .py numguard __init__.py verify_claim
💡 Result:
The term NumGuard appears in two distinct, unrelated technical contexts: a numeric integrity tool within a document processing pipeline, and an AI evaluation/verification library. 1. NumGuard (Numeric Integrity): This is a component of the 3DCF/doc2dataset pipeline [1][2][3]. It is designed to maintain data accuracy when converting financial or legal documents into training data [4][2]. It extracts numbers from documents, computes SHA-1 hashes (stored in a NumGuard record containing coordinates and units), and tracks them to detect numeric corruption [3][5]. It is implemented in Rust with Python and Node.js bindings [1][2]. 2. numguard (AI Evaluation): This is a Python library (available on PyPI) created by user ipezygj [6]. It functions as an Model Context Protocol (MCP) server designed to verify AI evaluation claims, such as benchmark scores, leaderboard rankings, and LLM-judge verdicts [6]. It builds upon the evalgate library and provides tools to check for statistical artifacts like multiple-comparisons errors, judge bias, and overfitting (e.g., via the Deflated Sharpe Ratio) [6]. The function verify_claim (or similar verification tools like verify_backtest) is part of its MCP interface, allowing AI agents to perform integrity checks before trusting or publishing an evaluation result [6]. Note: Do not confuse these with the unrelated project NuGuard (nuguard), which is an AI application security and red-teaming toolkit [7].
Citations:
- 1: https://github.com/3dcf-labs/doc2dataset
- 2: https://users.rust-lang.org/t/3dcf-doc2dataset-rust-based-document-dataset-pipeline-30-formats-token-compression-numeric-integrity/136773
- 3: https://github.com/3DCF-Labs/doc2dataset/blob/main/3dcf_doc2dataset_spec.md
- 4: https://dev.to/yevh/i-built-an-open-source-pipeline-to-convert-documents-into-llm-training-data-37pb
- 5: https://docs.rs/three-dcf-core/latest/three_dcf_core/struct.NumGuard.html
- 6: https://pypi.org/project/numguard/
- 7: https://pypi.org/project/nuguard/
🌐 Web query:
site:github.com/ipezygj/numguard "numguard.guard" "verify_claim" "__init__"
💡 Result:
I found the repo, but I didn’t find any matches for numguard.guard, verify_claim, or __init__ in the GitHub page contents I could access. The repository README does show verify_claim as part of its public API example, though. (github.com)
If you want, I can dig further into the actual source files in the repo.
Citations:
Call the numguard API by its real name.
The package exposes verify_claim top-level use; from numguard.guard import check is not part of the supported API, so the import fails before the tool can even test the installed-dependency path. Import and call the documented verify_claim API instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py`
around lines 51 - 54, Update the dependency import and invocation in the
NumGuard tool to use the supported top-level numguard.verify_claim API instead
of numguard.guard.check. Preserve the existing ImportError response for missing
numguard and pass the tool’s claim input through the documented API.
| try: | ||
| verdict = check(kind, **(params or {})) | ||
| except Exception as e: | ||
| logger.error(f"NumGuardTool error: {e}") | ||
| return f"Could not verify claim '{kind}': {e}" | ||
| survives = bool(verdict.get("survives")) | ||
| line = verdict.get("verdict") or ("survives" if survives else "flagged") | ||
| return json.dumps({"survives": survives, "verdict": line, "detail": verdict}, default=str) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Keep failure responses in the documented JSON shape.
Success returns JSON, but missing dependencies and malformed claims return plain text. This contradicts README.md line 54 and forces consumers to special-case errors. Return a consistent JSON error envelope for both failure paths, clearly distinguishing “unverified” from a verified survives: false result.
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 61-61: use jsonify instead of json.dumps for JSON output
Context: json.dumps({"survives": survives, "verdict": line, "detail": verdict}, default=str)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/numguard_tool/numguard_tool.py`
around lines 55 - 62, Update the failure handling in NumGuardTool’s verification
flow to return JSON for both missing-dependency and malformed-claim paths,
matching the documented error envelope. Clearly mark failures as unverified so
they cannot be confused with a verified result where survives is false, while
preserving the existing success response shape and error details.
1 participant
NumGuardTool — verify a number before an agent asserts it
Adds a tool (under
lib/crewai-tools) that routes a numeric claim to the statistical check that most directly tests it and returns whether it survives, plus an honest verdict. It lets a crew gate a backtest Sharpe (Deflated Sharpe Ratio), an A/B accuracy gap, a cherry-picked subset win, or an LLM-judge preference before reporting it as real.Backed by the open-source
numguardlibrary.Usage
Follows the repo conventions
src/crewai_tools/tools/numguard_tool/withnumguard_tool.py+README.md__all__) in bothtools/__init__.pyandcrewai_tools/__init__.pypackage_dependencies = ["numguard"](same pattern as ArxivPaperTool) — no hard dependency added to the package_runfails soft (returns a message) ifnumguardisn't installed or the claim is malformedNote
tool.specs.jsonis a generated artifact (generate_tool_specs.py) that imports the full tool suite; I wasn't able to run the full-suite generator in my environment, so I've lefttool.specs.jsonfor regeneration by the maintainers/CI (python -m crewai_tools.generate_tool_specs). Happy to update if preferred.