conformance

Agentic Resource Discovery Conformance Testing Tool

Automated compliance checking CLI to validate capability manifests and Registry REST API implementations.

This directory contains the official, zero-dependency Conformance Testing CLI Tool designed to verify that catalogs and discovery registries conform strictly to the Agentic Resource Discovery and ai-catalog specifications.

---

🚀 Getting Started

The CLI is written in Python and is completely zero-dependency, running out-of-the-box on any machine with standard Python 3.

1. Make Executable

Ensure the test and demo runner scripts have execution permissions:

chmod +x bin/conformance-test bin/run-conformance-demo

2. Run the Automated One-Click Demo

You can instantly verify both manifest validation and Registry API probing using our pre-bundled mock assets. Run the automated demo:

./bin/run-conformance-demo

This script validates a mock spec-compliant ai-catalog.json manifest, starts a lightweight mock registry API server in the background, executes live conformance probes against it, and automatically cleans up on completion.

3. Run Manual CLI Checks

You can also run the tester directly without arguments to view its manual commands:

./bin/conformance-test

---

📖 Usage Instructions

The tool operates in two distinct verification modes:

A. Manifest Validation Mode

Validates a local file or a remote live HTTP URL pointing to an ai-catalog.json capability manifest.

# Validate a local catalog manifest file
./bin/conformance-test manifest examples/basic/ai-catalog.json

# Validate a remote well-known catalog manifest hosted on a web server
./bin/conformance-test manifest https://example.com/.well-known/ai-catalog.json

B. Registry API Validation Mode

Probes and validates a live running Agent Registry REST API server.

./bin/conformance-test registry http://localhost:9010/api

For private or self-hosted registries that require request headers, pass one or more --header options. Headers are applied to all Registry API probes (GET /agents, POST /search, and POST /explore):

ARD_REGISTRY_TOKEN=...
./bin/conformance-test registry https://registry.example.com/api/ard \
  --header "Authorization: Bearer ${ARD_REGISTRY_TOKEN}"

This is a conformance tooling option only; it does not require authentication for public registries or define a Registry API security model.

---

🔍 What It Validates

1. Manifest Validation Mode

When checking a capability manifest (ai-catalog.json), the tool executes the following validations:

• JSON Structural Integrity: Parses the manifest payload to ensure it is valid, uncorrupted JSON.
• JSON Schema Draft 2020-12 Conformance: If the Python jsonschema library is installed (pip install jsonschema), the tool automatically runs a strict schema-level validation using the official ai-catalog.schema.json file.
• Strict URN Pattern Matching: Enforces that each entry's identifier adheres strictly to the domain-anchored URN namespace format defined in the spec: urn:air:<publisher>:<namespace>:<agent-name> (RFC 8141).
• Value-or-Reference Delivery: Enforces the mutual exclusivity constraint of the specification. Each entry MUST contain precisely one of either "url" (remote reference) or "data" (embedded payload), and will fail if both or neither are provided.
• Ecosystem Attributes Validation:
  • Checks that "representativeQueries" contains between 2 to 5 natural-language queries to ensure high-performance semantic vector embeddings.
  • Verifies that standard properties (specVersion, entries, displayName, type) are correctly typed.
• Deprecated Field Detection: Ensures the catalog does NOT use the deprecated "collections" root property (removed under ADR-0003 in favor of recursive catalog entries using type: application/ai-catalog+json).

---

2. Registry API Validation Mode

When checking a live Agent Registry server, the tool executes the following probes:

• GET /agents (Optional Listing Probe):
  • Contacts the GET endpoint to check if the registry supports deterministic browsing.
  • If supported (returns 200 OK), it validates that the response contains the paginated "items" structure.
  • If not supported (returns 404 or 501), it marks this as compliant since deterministic listing is optional.
• POST /search (Mandated Search Probe):
  • Probes the search route which is required for dynamic semantic capability discovery.
  • Sends a mock natural-language query payload with required query string and optional filter / limit parameters.
  • Verifies a 200 OK response structure containing a "results" array.
  • Validates that the nested payload structure is a valid CatalogEntry (contains identifier, displayName, type, and url or data).
• POST /explore (Optional Introspection Probe):
  • Probes the dynamic introspection route for facet and bucketing generation.
  • Sends a mock request requesting facet counts grouped by type.
  • Verifies a 200 OK response returning a valid "resultType" and "facets" Object.
  • If unsupported (returns 404 or 501), it marks this as compliant since explore is optional.

---

📌 Exit Codes

The tool outputs standard exit codes, making it ideal for integration into CI/CD pipelines, automated git hooks, or test rigs:

• 0: PASS. The manifest or registry conforms perfectly to the Agentic Resource Discovery specifications without errors.
• 1: FAIL. The target violates one or more specification constraints. Details of the violations are printed in red to stderr.

---

💡 Tip: Enhanced Validation

For strict schema-level checking, install the Python jsonschema validator in your local development workspace:

pip install jsonschema

If present, the tool will automatically activate JSON Schema checking alongside its custom semantic validations.