diff --git a/DEVELOPERS.md b/DEVELOPERS.md new file mode 100644 index 0000000..4b2bfba --- /dev/null +++ b/DEVELOPERS.md @@ -0,0 +1,113 @@ +# Developers + +How to run the examples in this repository locally and contribute new ones. + +## Prerequisites + +- **Python 3.9+** (3.10 or 3.11 recommended) +- **Docker** and **Docker Compose** — to run OpenMetadata locally +- **Git** + +## Run OpenMetadata locally + +The fastest path is Docker Compose: + +```bash +mkdir openmetadata-docker && cd openmetadata-docker +curl -sL -o docker-compose.yml https://github.com/open-metadata/OpenMetadata/releases/latest/download/docker-compose.yml +docker compose up -d +``` + +The UI comes up at **http://localhost:8585** and the API at **http://localhost:8585/api**. First boot takes a few minutes while Elasticsearch and the database initialize. + +Full deployment options (Kubernetes, Helm, bare metal): https://docs.open-metadata.org/latest/deployment + +## Get an authentication token + +Most examples need a JWT. + +1. Open http://localhost:8585 +2. Go to **Settings → Bots → ingestion-bot** +3. Copy the token + +Export it along with your host: + +```bash +export OPENMETADATA_HOST="http://localhost:8585/api" +export OPENMETADATA_JWT_TOKEN="" +``` + +Never commit tokens. Use environment variables or a local `.env` that stays untracked. + +## Set up a Python environment + +```bash +python -m venv .venv +source .venv/bin/activate # Windows: .venv\Scripts\activate + +pip install "openmetadata-ingestion~=1.13.1.0" +``` + +**Version matching matters.** The SDK version should match your OpenMetadata server version. A 1.13 server with a 1.11 SDK will fail on schema differences. Check your server version at the bottom of the UI, or: + +```bash +curl -s http://localhost:8585/api/v1/system/version +``` + +## Run an example + +```bash +python example_apis.py +``` + +Read-only operations run by default. Write operations (create, update, delete, lineage) are commented out in `main()` — uncomment the ones you want. + +Individual example folders have their own README with setup specific to that integration. Some need extra dependencies: + +```bash +cd custom-connector +pip install -r requirements.txt # where present +``` + +## Repository layout + +Each top-level folder is a self-contained example. Folders do not depend on each other — copy the one you need. + +| Folder | Purpose | +|--------|---------| +| `sdk-examples/` | Python SDK usage — entities, lineage, search, ownership | +| `mcp/` | Connect AI assistants and agents over MCP | +| `custom-connector/` | Build a custom ingestion connector | +| `ingestion-automation/`, `ingestion-github-actions/` | Programmatic and CI-driven ingestion | +| `api-lineage-cicd/`, `hive-lineage/` | Lineage from pipelines and query history | +| `test-suite/` | Data quality tests and assertions | +| `keycloak-sso/`, `sso-with-ssl/` | Authentication and TLS | +| `postgres/`, `redpanda/`, `grafana/` | Source-specific integrations | + +## Troubleshooting + +**`Could not connect to OpenMetadata`** — Confirm the server is up (`docker compose ps`) and that `OPENMETADATA_HOST` ends in `/api`. + +**`401 Unauthorized`** — The token expired or is wrong. Regenerate it under Settings → Bots. + +**Pydantic validation errors on create** — Almost always an SDK/server version mismatch. Align the versions. + +**`Table not found`** — The sample-data FQNs assume the bundled sample connector ran. Point `OPENMETADATA_SAMPLE_TABLE` at a table that exists in your instance. + +## Contributing + +1. Fork and branch from `main`. +2. Keep each example self-contained — no cross-folder imports. +3. Include a `README.md` in any new folder: what it demonstrates, prerequisites, and how to run it. +4. Use environment variables for hosts and tokens; never hard-code credentials. +5. Verify your example runs against a current OpenMetadata release before opening a PR. + +Issues and pull requests: https://github.com/open-metadata/openmetadata-demo + +## Links + +- **Docs:** https://docs.open-metadata.org/latest +- **Python SDK:** https://docs.open-metadata.org/latest/sdk/python +- **MCP guide:** https://docs.open-metadata.org/latest/how-to-guides/mcp +- **Main repo:** https://github.com/open-metadata/OpenMetadata +- **Slack community:** https://slack.open-metadata.org diff --git a/README.md b/README.md index 0efbf9b..2ea1297 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,97 @@ -# OpenMetadata Demo +# OpenMetadata Demo — Runnable Examples -In this repo you can find resources to try out OpenMetadata with different connectors. +Runnable examples for **[OpenMetadata](https://github.com/open-metadata/OpenMetadata)**, the open-source data catalog and metadata platform for data discovery, lineage, governance, data quality, and AI context. Each folder is self-contained: copy it, set your host and token, run it. + +## Quickstart + +The Python SDK ships in the `openmetadata-ingestion` package. Match the version to your OpenMetadata server: + +```bash +pip install "openmetadata-ingestion~=1.13.1.0" +``` + +Connect and verify: + +```python +from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import ( + OpenMetadataConnection, AuthProvider, +) +from metadata.generated.schema.security.client.openMetadataJWTClientConfig import ( + OpenMetadataJWTClientConfig, +) +from metadata.ingestion.ometa.ometa_api import OpenMetadata + +metadata = OpenMetadata(OpenMetadataConnection( + hostPort="http://localhost:8585/api", + authProvider=AuthProvider.openmetadata, + securityConfig=OpenMetadataJWTClientConfig(jwtToken=""), +)) +assert metadata.health_check() # True when connected +``` + +Read an entity: + +```python +from metadata.generated.schema.entity.data.table import Table + +table = metadata.get_by_name( + entity=Table, + fqn="sample_data.ecommerce_db.shopify.raw_product_catalog", +) +print(table.description) +print([c.name.root for c in table.columns]) +``` + +**Entity hierarchy:** a Table belongs to a Schema, which belongs to a Database, which belongs to a DatabaseService. Every entity references its parent by `fullyQualifiedName` — start at the service and work down. + +## Connect an AI assistant (MCP) + +OpenMetadata includes a built-in **MCP server**, so AI assistants and agents can search your catalog, traverse lineage, and read governed context over the Model Context Protocol. See **[`mcp/`](mcp/)** for a working setup. + +```bash +pip install data-ai-sdk +``` + +```python +from ai_sdk import AISdk, AISdkConfig + +client = AISdk.from_config(AISdkConfig.from_env()) +tools = client.mcp.as_langchain_tools() # LangChain-ready in one line +result = client.mcp.call_tool("search_metadata", {"query": "customers"}) +``` + +## What's in here + +| Folder | What it shows | +|--------|---------------| +| [`mcp/`](mcp/) | Connect AI assistants and agents to OpenMetadata over MCP | +| [`custom-connector/`](custom-connector/) | Build a custom ingestion connector (Python, protobuf, Docker) | +| [`ingestion-automation/`](ingestion-automation/) | Deploy ingestion pipelines programmatically | +| [`ingestion-github-actions/`](ingestion-github-actions/) | Run metadata ingestion from CI | +| [`api-lineage-cicd/`](api-lineage-cicd/) | Push lineage from a CI/CD pipeline via the API | +| [`hive-lineage/`](hive-lineage/) | Lineage for Hive workloads | +| [`test-suite/`](test-suite/) | Data quality tests and assertions | +| [`custom-graphql/`](custom-graphql/) | Query metadata through GraphQL | +| [`dynamic_csv_importer/`](dynamic_csv_importer/) | Bulk-import metadata from CSV | +| [`mlmodel-cicd/`](mlmodel-cicd/) | Register and track ML models | +| [`webhook_endpoint/`](webhook_endpoint/) | Receive OpenMetadata change events | +| [`keycloak-sso/`](keycloak-sso/), [`sso-with-ssl/`](sso-with-ssl/) | SSO and TLS configuration | +| [`postgres/`](postgres/), [`postgres-foreign/`](postgres-foreign/), [`redpanda/`](redpanda/), [`grafana/`](grafana/) | Source-specific integrations | +| [`rename-service/`](rename-service/), [`scripts/`](scripts/) | Operational utilities | + +## SDKs + +| SDK | Package | Install | +|-----|---------|---------| +| Python (metadata operations) | [`openmetadata-ingestion`](https://pypi.org/project/openmetadata-ingestion/) | `pip install "openmetadata-ingestion~=1.13.1.0"` | +| Python (AI / MCP / agents) | [`data-ai-sdk`](https://pypi.org/project/data-ai-sdk/) | `pip install data-ai-sdk` | +| TypeScript | [`@openmetadata/ai-sdk`](https://www.npmjs.com/package/@openmetadata/ai-sdk) | `npm install @openmetadata/ai-sdk` | +| Java | [`org.open-metadata:ai-sdk`](https://central.sonatype.com/artifact/org.open-metadata/ai-sdk) | Maven / Gradle | + +## Documentation + +- **Python SDK reference:** https://docs.open-metadata.org/latest/sdk/python +- **MCP server guide:** https://docs.open-metadata.org/latest/how-to-guides/mcp +- **AI SDK repo:** https://github.com/open-metadata/ai-sdk +- **Main project:** https://github.com/open-metadata/OpenMetadata +- **Docs home:** https://docs.open-metadata.org/latest