Skip to content

feat: validation settings docs #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
akijakya wants to merge 3 commits into
base: main
Choose a base branch
from
+355 −3
Draft

feat: validation settings docs #342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3010e98
feat: update oasf-sdk validation docs
akijakya Jan 12, 2026
2814672
feat: add validation comparison page to OASF docs
akijakya Jan 12, 2026
3720ab9
feat: add validation settings page to directory docs
akijakya Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/dir/.index
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ nav:
- Getting Started: getting-started.md
- Architecture: architecture.md
- Records: ads-records.md
- Validation Settings: validation-settings.md
- Trust Model: trust-model.md
- Features and Usage: scenarios.md
- Event Streaming: events.md
Expand Down
222 changes: 222 additions & 0 deletions docs/dir/validation-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
# Directory Validation Settings

The Directory server validates all records server-side before accepting them. The validation configuration determines which records are accepted and how compatible your directory instance is with other directory instances using different validation settings.

## Validation Implementation

The Directory server performs validation using the [OASF SDK](https://github.com/agntcy/oasf-sdk). This has important implications:

- **Supported OASF Versions**: The OASF SDK only supports a specific set of OASF schema versions. Records using unsupported versions will fail validation regardless of the validation mode.

- **Embedded Schema Validation**: When using embedded JSON Schema Draft-07 validation, the SDK can only validate against the schemas that have been embedded into it. These embedded schemas match the content of the official OASF instance at the time the SDK was built. This means:
- The embedded validator cannot validate against custom OASF instances or extensions
- It can only validate using the official OASF schema definitions that were included in the SDK

- **API Validation**: When using API validation, the SDK makes HTTP requests to the configured OASF server, allowing validation against custom instances and extensions.

## Validation Configuration Options

The Directory server supports three main validation modes:

### 1. OASF API Validation (Strict Mode - Default)

By default, the Directory server uses OASF API validation with strict mode enabled. This provides the most comprehensive validation and forms the official network baseline. Records validated here will pass all laxer validation modes.

**Configuration:**

```yaml
oasf_api_validation:
schema_url: "https://schema.oasf.outshift.com"
disable: false
strict_mode: true
```

### 2. OASF API Validation (Non-Strict)

Using the official OASF instance with strict mode disabled allows records with warnings (but not errors). Contains all records from Option 1, plus records with warnings. Overlaps with Option 3 but has edge case differences.

**Configuration:**

```yaml
oasf_api_validation:
schema_url: "https://schema.oasf.outshift.com"
disable: false
strict_mode: false
```

### 3. Embedded JSON Schema Validation

Using embedded JSON schemas for validation (no HTTP calls to OASF server). Overlaps with Option 2 but has edge case differences. Similar to Option 2 in that it doesn't return errors for extra modules. Cannot pull from Option 2 due to edge cases with placeholder classes (this may change in future versions).

**Configuration:**

```yaml
oasf_api_validation:
disable: true
```

## Using Custom OASF Instances

The validation modes above can also be used with custom OASF instances instead of the official OASF instance. This allows you to use OASF extensions or modified schemas. Note that using custom instances affects compatibility with other directory instances.

### 4. Private OASF Instance (Additional Modules)

Using a private OASF instance with extensions that only add additional module classes. Fully bidirectional compatible with Options 2 and 3, as these modes accept records with additional modules without validation errors.

**Configuration:**

```yaml
oasf_api_validation:
schema_url: "https://your-custom-oasf-instance.com"
disable: false
strict_mode: true # or false - affects compatibility
```

### 5. Private OASF Instance (Additional Taxonomy)

Using a private OASF instance with extensions that add to the skill/domain taxonomy (extends but doesn't change or remove). Records using the extended taxonomy can only be pulled by nodes using the exact same extended taxonomy.

**Configuration:**

```yaml
oasf_api_validation:
schema_url: "https://your-custom-oasf-instance.com"
disable: false
strict_mode: true # or false
```

### 6. Private OASF Instance (Changed Taxonomy)

Using a private OASF instance with extensions that modify the skill/domain taxonomy (changes or removes from taxonomy). Completely incompatible with all other options. Can only work with nodes using the exact same changed taxonomy.

**Configuration:**

```yaml
oasf_api_validation:
schema_url: "https://your-custom-oasf-instance.com"
disable: false
strict_mode: true # or false
```

## Network Compatibility: Record Pulling

In a distributed directory network, nodes can pull records from other nodes and have their records pulled by other nodes. The validation settings determine bidirectional compatibility:

- **Can Pull From**: Nodes from which this option can successfully pull and validate records
- **Can Be Pulled By**: Nodes that can successfully pull and validate records from this option

### Compatibility Matrix

| Option | Can Pull From | Can Be Pulled By |
| --------------------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------- |
| **1. API Validation (Strict)** | Option 1 only | Options 1, 2, 3, 4, 5 |
| **2. API Validation (Non-Strict)** | Options 1, 2 | Options 2, 4 (non-strict mode) |
| **3. Embedded JSON Schema** | Options 1, 3<br/>_Note: Cannot pull from Option 2 due to edge cases with placeholder classes_ | Options 2, 4 (non-strict mode) |
| **4. Additional Modules (Strict)** | Options 1, 4 (strict mode) | Options 2, 3, 4 (any mode) |
| **4. Additional Modules (Non-Strict)** | Options 1, 2, 3, 4 (any mode) | Options 2, 3, 4 (non-strict mode) |
| **5. Additional Taxonomy (Strict)** | Options 1, 5 (same taxonomy, strict) | Options 5 (same taxonomy, any mode) |
| **5. Additional Taxonomy (Non-Strict)** | Options 1, 2, 3, 4, 5 (same taxonomy, any mode) | Options 5 (same taxonomy, any mode) |
| **6. Changed Taxonomy** | Option 6 (same taxonomy only) | Option 6 (same taxonomy only) |

## Compatibility Diagram

The following diagram illustrates the compatibility relationships between different validation settings. Options 5 and 6 (with custom taxonomies) are shown separately as they have limited compatibility.

```mermaid
flowchart TB
subgraph Official["Official OASF Instance"]
A1["1. API Validation<br/>(Strict Mode)<br/>MOST STRICT"]
style A1 fill:#ffffff,stroke:#000000,stroke-width:3px

B2["2. API Validation<br/>(Non-Strict Mode)"]
style B2 fill:#e6f3ff,stroke:#0066cc,stroke-width:2px

C3["3. Embedded JSON Schema"]
style C3 fill:#e6ffe6,stroke:#00cc00,stroke-width:2px

A1 -->|"All records pass"| B2
B2 <-.->|"Overlap with<br/>edge case differences"| C3
end

subgraph Private["Private OASF (Additional Modules)"]
D4S["4a. Strict Mode"]
style D4S fill:#fff9e6,stroke:#ff9900,stroke-width:2px

D4N["4b. Non-Strict Mode"]
style D4N fill:#fff9e6,stroke:#ff9900,stroke-width:2px

B2 <-->|"Fully compatible"| D4S
B2 <-->|"Fully compatible"| D4N
C3 <-->|"Fully compatible"| D4S
C3 <-->|"Fully compatible"| D4N
end

subgraph CustomTax["Custom Taxonomy"]
E5S["5a. Additional Taxonomy<br/>(Strict Mode)"]
style E5S fill:#fff0e6,stroke:#ff6600,stroke-width:2px

E5N["5b. Additional Taxonomy<br/>(Non-Strict Mode)"]
style E5N fill:#fff0e6,stroke:#ff6600,stroke-width:2px

E6["6. Changed Taxonomy<br/>(INCOMPATIBLE)"]
style E6 fill:#ffe6e6,stroke:#cc0000,stroke-width:3px
end

A1 -->|"Can pull from"| E5S
A1 -->|"Can pull from"| E5N
E5S -.->|"Same taxonomy only"| E5S
E5N -.->|"Same taxonomy only"| E5N
E5S -.->|"Same taxonomy only"| E5N
E6 -.->|"Same taxonomy only"| E6
```

## Configuration Examples

### Environment Variables

The directory server requires validation configuration to start. You must either provide an OASF schema URL or disable API validation to use embedded schemas.

```bash
# API validation with strict mode (default when URL is provided)
DIRECTORY_SERVER_OASF_API_VALIDATION_SCHEMA_URL=https://schema.oasf.outshift.com ./dirctl-apiserver

# Custom OASF server
DIRECTORY_SERVER_OASF_API_VALIDATION_SCHEMA_URL=http://localhost:8080 ./dirctl-apiserver

# Non-strict API validation
DIRECTORY_SERVER_OASF_API_VALIDATION_SCHEMA_URL=https://schema.oasf.outshift.com DIRECTORY_SERVER_OASF_API_VALIDATION_STRICT_MODE=false ./dirctl-apiserver

# Embedded schema validation (no API calls)
DIRECTORY_SERVER_OASF_API_VALIDATION_DISABLE=true ./dirctl-apiserver
```

### YAML Configuration

```yaml
# server.config.yml
oasf_api_validation:
schema_url: "https://schema.oasf.outshift.com"
disable: false
strict_mode: true
listen_address: "0.0.0.0:8888"
```

## Choosing a Validation Mode

- **Production/Public Directories**: Use Option 1 (API validation, strict mode) for maximum compatibility and quality assurance.

- **Development/Testing**: Use Option 2 (non-strict) or Option 3 (embedded) for faster iteration while still catching errors.

- **Private Deployments with Additional Modules**: Use Option 4 if you only need additional module classes.
- Use strict mode if you want maximum compatibility with Option 1
- Use non-strict mode if you want compatibility with Options 2 and 3

- **Private Deployments with Additional Taxonomy**: Use Option 5 if you need to add to the skill/domain taxonomy without changing or removing existing entries.

- **Private Deployments with Changed Taxonomy**: Use Option 6 only if you need to modify (change or remove from) the core skill/domain taxonomy. Note that this creates an incompatible directory instance that can only work with nodes using the exact same taxonomy.

## Related Documentation

- [OASF Validation Service](../oasf/validation.md) - Detailed validation service documentation
- [Validation Comparison](../oasf/validation-comparison.md) - Comparison between API validator and JSON Schema
- [OASF Extensions](https://github.com/agntcy/oasf/blob/main/CONTRIBUTING.md#oasf-extensions) - Information about creating OASF extensions
1 change: 1 addition & 0 deletions docs/oasf/.index
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,6 @@ nav:
- Decoding Service: decoding.md
- Translation Service: translation.md
- Validation Service: validation.md
- Validation Comparison: validation-comparison.md
- OASF Record Guide: agent-record-guide.md
- OASF Contribution Guide: contributing.md
98 changes: 98 additions & 0 deletions docs/oasf/validation-comparison.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Validation Comparison: API Validator vs JSON Schema Draft-07

This table compares validation outcomes between the OASF API validator and JSON Schema Draft-07 validator.

## Legend
- **API Error**: API validator returns ERROR
- **API Warning**: API validator returns WARNING
- **JSON Schema Pass**: JSON Schema Draft-07 validation passes
- **JSON Schema Fail**: JSON Schema Draft-07 validation fails

## Comparison Table

| Case | Example | API Validator | JSON Schema | Notes |
|------|---------|---------------|-------------|-------|
| **1. Version Incompatibility (Later)** | `schema_version: "1.0.0"` when server is `0.8.0` | ERROR (`version_incompatible_later`) | PASS | JSON Schema validates format only, not semantic compatibility * |
| **2. Version Incompatibility (Initial Dev)** | `schema_version: "0.1.0"` when server is `0.8.0` | ERROR (`version_incompatible_initial_development`) | PASS | JSON Schema validates format only, not SemVer rules * |
| **3. Version Incompatibility (Prerelease)** | `schema_version: "0.8.0-alpha"` when server is `0.8.0` | ERROR (`version_incompatible_prerelease`) | PASS | JSON Schema validates format only, not prerelease compatibility * |
| **4. Version Earlier (Compatible)** | `schema_version: "0.7.0"` when server is `0.8.0` | WARNING (`version_earlier`) | PASS | JSON Schema validates format only, not version comparison * |
| **5. Regex Pattern Mismatch** | `previous_record_cid: "invalid"` (doesn't match CID regex) | WARNING (`attribute_value_regex_not_matched`) | PASS | JSON Schema may export regex but validation is lenient; API treats as warning |
| **6. Regex Pattern Mismatch (Super Type)** | String doesn't match super type regex | WARNING (`attribute_value_super_type_regex_not_matched`) | PASS | Same as above for super types |
| **7. Using Base Class (Placeholder)** | `skills: [{"name": "base_skill"}]` | WARNING (`placeholder_used`) | FAIL | JSON Schema `oneOf` doesn't include parent classes, so base_skill is not in the list |
| **8. Using Deprecated Class** | Using a deprecated skill/domain/module | WARNING (`class_deprecated`) | PASS | JSON Schema doesn't track deprecation |
| **9. Using Deprecated Attribute** | Using a deprecated attribute | WARNING (`attribute_deprecated`) | PASS | JSON Schema doesn't track deprecation |
| **10. Using Deprecated Object** | Using a deprecated object | WARNING (`object_deprecated`) | PASS | JSON Schema doesn't track deprecation |
| **11. Using Deprecated Enum Value** | Using a deprecated enum value | WARNING (`attribute_enum_value_deprecated`) | PASS | JSON Schema doesn't track deprecation |
| **12. Using Deprecated Enum Array Value** | Using a deprecated enum array value | WARNING (`attribute_enum_array_value_deprecated`) | PASS | JSON Schema doesn't track deprecation |
| **13. Enum Sibling Mismatch** | Enum value doesn't match sibling caption | WARNING (`attribute_enum_sibling_incorrect`) | PASS | JSON Schema doesn't validate enum siblings |
| **14. Enum Sibling Suspicious (Other)** | Enum value 99 with matching sibling caption | WARNING (`attribute_enum_sibling_suspicious_other`) | PASS | JSON Schema doesn't validate enum siblings |
| **15. Missing Recommended Attribute** | Required attribute missing (if `warn_on_missing_recommended` enabled) | WARNING (`attribute_recommended_missing`) | PASS | JSON Schema only validates `required`, not `recommended` |
| **16. Empty Required Array** | `skills: []` when skills is required | ERROR (`attribute_required_empty`) | FAIL | JSON Schema now includes `minItems: 1` for required arrays (after fix) |
| **17. Wrong Type** | `name: 123` when name should be string | ERROR (`attribute_wrong_type`) | FAIL | Both catch type mismatches |
| **18. Missing Required Attribute** | Missing required `name` field | ERROR (`attribute_required_missing`) | FAIL | Both catch missing required fields |
| **19. Unknown Attribute** | Extra attribute not in schema | ERROR (`attribute_unknown`) | FAIL | JSON Schema uses `additionalProperties: false` |
| **20. Unknown Enum Value** | Enum value not in allowed list | ERROR (`attribute_enum_value_unknown`) | FAIL | JSON Schema uses `enum` constraint |
| **21. Unknown Enum Array Value** | Enum array value not in allowed list | ERROR (`attribute_enum_array_value_unknown`) | FAIL | JSON Schema validates array items |
| **22. Value Outside Range** | Number outside type's range | ERROR (`attribute_value_exceeds_range`) | FAIL | JSON Schema uses `minimum`/`maximum` |
| **23. Value Exceeds Max Length** | String exceeds type's max length | ERROR (`attribute_value_exceeds_max_len`) | FAIL | JSON Schema uses `maxLength` |
| **24. Value Not in Type Values** | Value not in type's allowed values list | ERROR (`attribute_value_not_in_type_values`) | FAIL | JSON Schema uses `enum` constraint |
| **25. Unknown Class ID** | `id: 99999` doesn't exist | ERROR (`id_unknown`) | FAIL | JSON Schema uses `oneOf` with `const` values |
| **26. Unknown Class Name** | `name: "nonexistent"` doesn't exist | ERROR (`name_unknown`) | FAIL | JSON Schema uses `oneOf` with `const` values |
| **27. ID/Name Mismatch** | `id: 2` and `name: "different"` refer to different classes | ERROR (`id_name_mismatch`) | FAIL | JSON Schema `oneOf` with `const` prevents mismatches |
| **28. Constraint Failed (at_least_one)** | None of the constraint fields present | ERROR (`constraint_failed`) | FAIL | JSON Schema uses `anyOf` for `at_least_one` |
| **29. Constraint Failed (just_one)** | Both fields in `just_one` constraint present | ERROR (`constraint_failed`) | FAIL | JSON Schema `oneOf` pattern correctly prevents both fields from being present |
| **30. Enum Array Sibling Missing** | Enum array sibling array too short | ERROR (`attribute_enum_array_sibling_missing`) | PASS | JSON Schema doesn't validate enum array siblings |
| **31. Enum Array Sibling Incorrect** | Enum array sibling value doesn't match | ERROR (`attribute_enum_array_sibling_incorrect`) | PASS | JSON Schema doesn't validate enum array siblings |
| **32. Enum Object Not Matched** | Enum object doesn't match any allowed objects | ERROR (`enum_object_not_matched`) | FAIL | JSON Schema uses `oneOf` for enum objects |
| **33. Unknown Profile** | Profile in `metadata.profiles` doesn't exist | ERROR (`profile_unknown`) | PASS | JSON Schema validates structure but not profile existence (only for classes, not record) |
| **34. Invalid Version Format** | `schema_version: "invalid"` | ERROR (`version_invalid_format`) | FAIL | JSON Schema validates semantic versioning format * |
| **35. Non-OASF Module** | Module name not an OASF module | WARNING (`module_unknown`) | PASS | JSON Schema doesn't distinguish OASF vs non-OASF modules |

\* **Note on schema_version validation**: When using oasf-sdk, any schema version that is not explicitly supported (currently: 0.3.1, 0.7.0, 0.8.0) will result in a failure, regardless of whether the API validator returns an error or warning. The SDK uses a switch statement that only handles specific versions, and any non-supported version will return an error: `unsupported OASF version: <version>`.

## Summary by Category

### API ERROR, JSON Schema PASSES (Gaps in JSON Schema)
1. Version incompatibility (later, initial dev, prerelease)
2. Enum array sibling validation (missing, incorrect)
3. Unknown profile (for classes)

### API WARNING, JSON Schema PASSES (Non-blocking Issues)
1. Version earlier (compatible)
2. Regex pattern mismatches
3. All deprecation warnings (class, attribute, object, enum values)
4. Enum sibling mismatches
5. Missing recommended attributes
6. Using base classes (placeholders)
7. Non-OASF modules

### Both FAIL (Both Catch the Issue)
1. Empty required arrays
2. Wrong types
3. Missing required attributes
4. Unknown attributes
5. Unknown enum values
6. Value range/length violations
7. Unknown class ID/name
8. ID/name mismatches
9. Constraint violations (`just_one`, `at_least_one`)
10. Enum object mismatches
11. Invalid version format

### API WARNING, JSON Schema FAILS (JSON Schema More Strict)
1. Using base classes (base_skill, base_domain, base_module) - JSON Schema `oneOf` doesn't include parent classes

## Key Insights

1. **JSON Schema is comprehensive** for structural and value validation
2. **Main gaps in JSON Schema**:
- Semantic version compatibility (only validates format)
- Enum array sibling validation
- Deprecation tracking
- Regex validation (treated as warnings, not errors)
- Profile existence (for classes)

3. **JSON Schema is more strict** in one case:
- Base classes are not included in `oneOf` lists, so using them fails JSON Schema but only warns in API validator

4. **Both validators catch** most structural and type validation issues
Loading
Loading