Skip to content

Conversation

@kyungsoo-datahub
Copy link
Contributor

@kyungsoo-datahub kyungsoo-datahub commented Dec 20, 2025

Summary

  • Fix column-level lineage for PowerBI ODBC connections to Athena/Glue
  • Strip awsdatacatalog. catalog prefix from column lineage URNs to match existing entities

Problem

When PowerBI connects to Athena via ODBC, SQL queries reference tables as awsdatacatalog.database.table. The previous fix (#15728) only stripped this prefix from table-level lineage URNs, but column-level lineage URNs still contained the prefix, causing lineage to not connect to existing Athena/Glue entities.

Solution

Updated _strip_athena_catalog_from_lineage() in OdbcLineage to strip catalog prefix from both:

  • lineage.upstreams[*].urn (table-level)
  • lineage.column_lineage[*].upstreams[*].table (column-level)

Scope

  • Only affects: PowerBI → ODBC → Athena/Glue

@github-actions
Copy link
Contributor

Linear: ING-1311

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Dec 20, 2025
@codecov
Copy link

codecov bot commented Dec 20, 2025

Codecov Report

❌ Patch coverage is 89.18919% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ngestion/source/powerbi/m_query/pattern_handler.py 89.18% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 20, 2025
@kyungsoo-datahub kyungsoo-datahub force-pushed the mm--powerbi-athena-catalog-strip branch 2 times, most recently from 6041772 to 86cac14 Compare December 20, 2025 03:29
@kyungsoo-datahub kyungsoo-datahub changed the title fix(powerbi): Strip AwsDataCatalog prefix from ODBC Athena/Glue linea… fix(ingest/powerbi): strip Athena catalog prefix from ODBC column lineage URNs Dec 20, 2025
@kyungsoo-datahub kyungsoo-datahub force-pushed the mm--powerbi-athena-catalog-strip branch from 86cac14 to cccc5ee Compare December 20, 2025 15:56
@kyungsoo-datahub kyungsoo-datahub force-pushed the mm--powerbi-athena-catalog-strip branch from cccc5ee to b683276 Compare December 20, 2025 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants