Add multi schema support for databricks #754
Open
+393
−138
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for documenting a full databricks catalog (full table ref in dbx nomenclature is
catalog.schema.table) by not specifying a schema in the DSN.This is a non-breaking change, if a schema is specified (as previously required) behaviour is identical, verified this by comparing new outputs against the contents of
samples/databricks/which were generated without this change.If the schema is not specified then the dbx driver simply documents all tables under the catalog. To avoid ambiguity/name-collisions each table reference is given as
schema_name.table_namewhen ran in this mode. I think this is the same behaviour as the postgres adaptor.Attached is sample output when ran in this mode (just captures the tpch tables used in the single-schema output along with tables in the databricks information schema). This could be committed to the repo if desired but no other dbs include multiple versions of sample output at the moment so haven't done that yet.
tbls-dbx-multi-schema-output.zip
couple little changes in the implementation to try and align with wider project idioms also included here and there but all minor.