Rdf ingestion mvp #15741

stephengoldbaum · 2025-12-19T18:36:23Z

Summary

This PR introduces a new RDF ingestion source for DataHub, enabling ingestion of RDF/OWL ontologies (Turtle, RDF/XML, JSON-LD, N3, N-Triples) with a focus on business glossaries. The source extracts glossary terms, term hierarchies, and relationships from RDF files using standard vocabularies like SKOS, OWL, and RDFS.

What's New

Core Features

RDF Ingestion Source (type: rdf) - Native DataHub plugin for RDF/OWL ontologies
Multiple Format Support - Turtle, RDF/XML, JSON-LD, N3, N-Triples
Flexible Source Loading - Files, directories (with recursive option), URLs, and comma-separated file lists
Glossary Term Extraction - Converts skos:Concept and owl:Class to DataHub GlossaryTerms
Glossary Node Hierarchy - Auto-creates glossary nodes from IRI path hierarchies
Term Relationships - Extracts skos:broader and skos:narrower relationships as isRelatedTerms
Stateful Ingestion - Full support for stale entity removal via stateful_ingestion config
Platform Instance Support - Configurable platform instances via platform_instance config

Architecture

Modular Entity Processing - Clean separation with extractors, converters, and MCP builders
Dependency-Based Processing - Topological sort for correct entity processing order
Test Connection Support - Implements test_connection() for connection validation

Capabilities

The source supports the following DataHub capabilities:

Capability	Status	Notes
Glossary Terms	✅	Enabled by default
Glossary Nodes	✅	Auto-created from IRI path hierarchies
Term Relationships	✅	Supports `skos:broader` and `skos:narrower`
Detect Deleted Entities	✅	Requires `stateful_ingestion.enabled: true`
Platform Instance	✅	Supported via `platform_instance` config
Extract Descriptions	✅	Enabled by default (from `skos:definition` or `rdfs:comment`)
Data Domain	❌	Not applicable (domains used internally for hierarchy)
Dataset Profiling	❌	Not applicable
Extract Lineage	❌	Not in MVP
Extract Ownership	❌	Not supported
Extract Tags	❌	Not supported

Testing

Test Coverage

126 unit tests - Comprehensive coverage of core functionality, error handling, and edge cases
16 integration tests - End-to-end testing with golden file validation
Test scenarios include:
- Simple glossary ingestion
- Glossary with relationships
- Glossary with domains
- Multiple RDF formats (Turtle, RDF/XML, JSON-LD)
- Recursive directory ingestion
- Export filtering (export_only, skip_export)
- Stateful ingestion with stale entity removal
- Error handling (missing files, malformed RDF, invalid formats)
- Large file performance warnings
- Path traversal protection

Test Files

tests/unit/rdf/ - Unit tests for individual components
tests/integration/rdf/ - Integration tests with golden file validation
All tests passing ✅

Documentation

User Documentation

docs/sources/rdf/rdf.md - Comprehensive user guide (489 lines)
- Quickstart guide
- Configuration reference
- RDF format and source types
- Dialects and selective export
- Stateful ingestion guide
- Example RDF files
- IRI-to-URN mapping
- Glossary node hierarchy
- Supported vocabularies
- Limitations and troubleshooting

Recipe Examples

docs/sources/rdf/rdf_recipe.yml - Example recipes for basic and stateful ingestion

Integration Test Documentation

tests/integration/rdf/README.md - Detailed guide for running integration tests

Configuration Example

source:
type: rdf
config:
source: ./glossary.ttl
format: turtle
environment: PROD
stateful_ingestion:
enabled: true
remove_stale_metadata: true
export_only:
- glossary## Files Changed

Technical Notes

Security & Performance

URL Loading Security - Timeout limits, size limits, and redirect limits for safe URL loading
Path Traversal Protection - Configurable enforcement to prevent access outside intended directories
Memory Efficiency - Generator patterns for work unit generation and streaming for large URL downloads
Format Validation - Validates RDF formats before processing

Code Quality

Thread-Safe Registry - Entity registry uses double-checked locking pattern for thread safety
Component Validation - Validates registered components for entity type consistency
Type Safety - Complete type hints with proper forward references for MCP return types
Error Handling - Granular error reporting with structured logs and context
URN Generation - Standardized URN format using dot notation, proper encoding, and GUID fallback for non-ASCII characters

New Files

src/datahub/ingestion/source/rdf/ingestion/rdf_source.py - Main source implementation
src/datahub/ingestion/source/rdf/core/rdf_loader.py - RDF loading utilities with security
src/datahub/ingestion/source/rdf/core/urn_generator.py - URN generation with encoding
src/datahub/ingestion/source/rdf/entities/base.py - Base interfaces for entity processing
src/datahub/ingestion/source/rdf/entities/registry.py - Thread-safe entity registry
docs/sources/rdf/rdf.md - User documentation
docs/sources/rdf/rdf_recipe.yml - Recipe examples
tests/integration/rdf/test_rdf_source.py - Integration tests
tests/unit/rdf/ - Unit tests (multiple files)

Modified Files

setup.py - Added RDF source to entry points (line 862)

Breaking Changes

None - This is a new feature addition with no breaking changes to existing functionality.

Support Status

The RDF source is marked as INCUBATING (SupportStatus.INCUBATING), indicating it's ready for community adoption but may have minor version changes in future releases based on feedback.

Checklist

- Add RDF ingestion source for glossary terms, domains, and relationships - Streamlined architecture: extractors return DataHub AST directly - Removed unnecessary abstraction layers (RDF AST, converters where not needed) - Support for SKOS, OWL, and other RDF vocabularies - Comprehensive test coverage with 128 passing tests - UI integration for RDF source configuration

- Remove build_relationship_mcps() method from GlossaryTermMCPBuilder - Update tests to use RelationshipMCPBuilder directly - Clean separation: glossary_term handles terms, relationship handles relationships - Refactor _generate_workunits_from_ast to reduce complexity

…ance - Enhance error handling in RDFSource to provide actionable messages for missing files, malformed RDF, and invalid formats. - Implement unit tests to verify error handling behavior and ensure graceful degradation. - Update glossary term URN generation to use dot notation for hierarchical paths. - Improve logging for large file processing and ensure consistent URN formats across glossary nodes and terms. - Refactor methods to yield MCPs for memory efficiency during processing.

…uidance - Added helper text for various RDF source fields including source, format, extensions, recursive processing, environment, and dialect. - These enhancements aim to provide clearer instructions and examples for users configuring RDF ingestion settings.

- Introduced new RDF platform entry in capability_summary.json with detailed capabilities including deletion detection, tags, ownership, lineage, data profiling, domains, descriptions, and platform instance support. - Each capability includes a description and support status to enhance clarity for users configuring RDF ingestion.

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

…ters - Introduced a new documentation file for the RDF ingestion source. - Updated type hints across various classes to use `Optional` for context parameters, enhancing code clarity and type safety. - Adjusted method signatures in `EntityExtractor`, `EntityConverter`, `EntityMCPBuilder`, and related classes to reflect these changes.

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

…_summary.json

This reverts commit 48a0118.

- Added unit tests for duplicate term definition handling, ensuring correct extraction behavior for same URIs and properties. - Implemented comprehensive validation tests for RDF source configuration, covering required fields, type checks, and value constraints. - Introduced connection testing unit tests to verify functionality and error handling for various scenarios, including file existence and RDF format validation. - Developed edge case tests to handle scenarios like empty files, circular relationships, and special characters in paths. - Enhanced error handling tests to ensure actionable feedback for file not found, invalid format, and unsupported extensions.

- Updated test_duplicate_handling.py to use URIRef and Literal for RDF terms, enhancing readability and maintainability. - Modified test_rdf_config.py to specify type for config_dict, improving type safety. - Adjusted test_rdf_source_errors.py to refine error assertion logic for better clarity in failure messages.

- Updated test_rdf_config.py to clarify format and environment validation logic, ensuring accurate assertions for invalid formats and environments. - Refined test cases to reflect changes in default values and type coercion for recursive configurations. - Improved test_rdf_connection.py and test_rdf_edge_cases.py by adding namespace prefixes for better RDF structure clarity. - Enhanced error handling in test_rdf_source_errors.py to ensure actionable feedback for various failure scenarios.

- Introduced a new RDFSourceConfig class for managing RDF ingestion configurations, including options for source paths, formats, and stateful ingestion. - Updated the GlossaryTermMCPBuilder to dynamically set the termSource based on the presence of the term's source, ensuring accurate metadata representation for RDF terms. - Improved documentation within the code to clarify the behavior and expectations for term source handling in RDF ingestion.

- Added a new entry for the Airflow data platform in the sources.json file, including its URN, name, and display name. - This update enhances the ingestion capabilities by incorporating Airflow as a supported source.

- Added support for loading RDF data from local and remote zip files. - Implemented functionality to handle web folder URLs with HTML directory listings. - Updated documentation examples to include zip file usage. - Added unit tests for zip file and web folder loading, including edge cases for error handling and security checks.

… be regenerated by CI)

github-actions · 2025-12-19T18:36:36Z

Linear: ING-1308

codecov · 2025-12-19T18:38:58Z

Codecov Report

❌ Patch coverage is 66.74280% with 728 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...rc/datahub/ingestion/source/rdf/core/rdf_loader.py	67.17%	150 Missing ⚠️
...tahub/ingestion/source/rdf/ingestion/rdf_source.py	70.85%	116 Missing ⚠️
...datahub/ingestion/source/rdf/core/urn_generator.py	22.38%	104 Missing ⚠️
...ion/source/rdf/entities/glossary_term/extractor.py	55.86%	94 Missing ⚠️
.../datahub/ingestion/source/rdf/entities/registry.py	66.85%	60 Missing ⚠️
...rc/datahub/ingestion/source/rdf/dialects/router.py	27.86%	44 Missing ⚠️
...c/datahub/ingestion/source/rdf/dialects/generic.py	25.49%	38 Missing ⚠️
.../src/datahub/ingestion/source/rdf/dialects/fibo.py	28.26%	33 Missing ⚠️
...c/datahub/ingestion/source/rdf/dialects/bcbs239.py	29.54%	31 Missing ⚠️
...source/rdf/entities/glossary_term/urn_generator.py	68.18%	14 Missing ⚠️
... and 8 more

❌ Your patch check has failed because the patch coverage (66.74%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.

📢 Thoughts on this report? Let us know!

alwaysmeticulous · 2025-12-19T18:42:27Z

✅ Meticulous spotted 0 visual differences across 979 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 83de506. This comment will update as new commits are pushed.}

codecov · 2025-12-19T18:45:28Z

Bundle Report

Changes will increase total bundle size by 3.34kB (0.01%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	31.27MB	3.34kB (0.01%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	3.34kB	19.24MB	0.02%

Files in assets/index-*.js:

./src/app/ingestV2/source/builder/RecipeForm/rdf.ts → Total Size: 2.86kB
./src/app/ingestV2/source/builder/sources.json → Total Size: 40.49kB
./src/app/ingestV2/source/builder/RecipeForm/constants.ts → Total Size: 10.11kB
./src/app/ingestV2/source/builder/constants.ts → Total Size: 7.01kB

stephengoldbaum added 22 commits December 19, 2025 13:19

Bringing RDF ingestion to DataHub standards compliance

63e7a02

fix(ingestion): update warning filters and add RDF dependency

e6245b9

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

chore(setup): add RDF dependency to base and test requirements

65f001a

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

Update capability_summary.json with RDF plugin

39d51e9

Update RDF documentation link and ensure newline at end of capability…

c072bd0

…_summary.json

Fix capability_summary.py to ensure consistent plugin ordering

14f6d58

Revert "Fix capability_summary.py to ensure consistent plugin ordering"

8f649d2

This reverts commit 48a0118.

Fix broken markdown links in RDF documentation

a5fbd9d

Add Airflow source configuration to sources.json

98656ed

- Added a new entry for the Airflow data platform in the sources.json file, including its URN, name, and display name. - This update enhances the ingestion capabilities by incorporating Airflow as a supported source.

Restore capability_summary.json from master to include dataplex (will…

10de0a7

… be regenerated by CI)

Regenerate capability_summary.json to include RDF source

83de506

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX labels Dec 19, 2025

github-actions bot added the community-contribution PR or Issue raised by member(s) of DataHub Community label Dec 19, 2025

github-actions bot deployed to datahub-wheels (Preview) December 19, 2025 18:38 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) December 19, 2025 18:41 View deployment

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 19, 2025

vercel bot deployed to Preview December 19, 2025 18:48 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rdf ingestion mvp #15741

Rdf ingestion mvp #15741

stephengoldbaum commented Dec 19, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 19, 2025

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rdf ingestion mvp #15741

Are you sure you want to change the base?

Rdf ingestion mvp #15741

Conversation

stephengoldbaum commented Dec 19, 2025

Summary

What's New

Core Features

Architecture

Capabilities

Testing

Test Coverage

Test Files

Documentation

User Documentation

Recipe Examples

Integration Test Documentation

Configuration Example

Technical Notes

Security & Performance

Code Quality

New Files

Modified Files

Breaking Changes

Support Status

Checklist

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 19, 2025

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Dec 19, 2025 •

edited

Loading

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading