This repository has been archived by the owner on Jan 19, 2023. It is now read-only.
Sync with upstream #2
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update documentation * Backwards-compatible rename special_type -> semantic_type
* Added support for aliased models and source blocks Added alias support in model parsing. Also added searching for "sources" yml blocks pushing table/column documentation to metabase (defining sources is recommended by dbt best practices and often done in tandem with creating staging models). Allows using "metabase.fk_ref" in sources or models .yml in order explicitly define a metabase relationship to a table which allows us to propagate the FK relationship properly to Metabase regardless of aliasing. metabase.fk_ref is looked for first in meta before falling back to default parser. * Update dbt.py Replace print with log and ensure multiple sources in a single YML can be looped through- also formatted according to black * black formatting update replace single quoting with double to conform to style guideline * logging for empty or invalid yml useful when project structure is defined and ymls created but not yet populated * logging updates replaced last prints with info level logs and made invalid yml logger msg match rest * more detailed model logging Re-added processing model logger info msg * log msg matching recommendation updated log to match recommendation
|
I'll have a look at the modifications introduced. |
* Dummy unit test and CI workflow * Fix dummy test * Fix lint issues and include lint in CI * Fix type issues and include type check in CI * Basic mocked tests and requirements for type checking
* Fix 400 Error on empty-string descriptions * Applying formatting with black
…as support (#19) * Add .idea to gitignore * merge changes from read-from-artifacts fork integrating with latest version of dbt-metabase having improved yaml parsing. large formatting update to conform to black. * fix grammatical error and update readme to say semantic type * fix error in field lookup key setting and continue to support either special or semantic until ready to deprecate * internally use semantic but support meta refs to special which will be properly used if found in metabase api response. * safe importing of dependent modules in addition to bugfixes, logical synchronization, and formatting * added more verbose comments tracking intent as well as ensuring nodes without depends on/test_metadata dont throw * setting fields to PK type is worthy of info logging * handle cross schema foreign keys with explicit metabase.fk_ref with expected input format to be defined in readme. otherwise automatic resolution of target field using relation test will prepend target run schema which should be fine in 95% of use cases. cases outside that can use manifest.json parsing or set fk_ref in yml. * added debug log for validating parsed schema/fields for fk targets and ensured support for schema agnostic fk targets (schema resolved from manifest.json) * all necessary args added for cli usage. some basic assertions added. sync will now only fail hard if timeout is explicit, otherwise default behaviour if --sync is true is to attempt sync for 30 seconds and proceed with aligning what can be aligned successfully. more formatting and a few comments for clarity of intent. also added option to pass custom cert bundle to verify. * corrected typo in exclude var and added support for verbosity flag. translated some args to store action based. * updates to handle aliases when ran via dbt_path (yml parser) ubiquitously with seemless function alongside primary artifact parser (manifest.json). * support for aliased fk refs parsed via parse_ref. backoff of parse ref regex to permit catching last arg of either ref or source always being the target table. if pointing to an alias, we are collecting aliases during yml parsing to be passed to metabased client and translated to metabase table names as needed. this functionality should be unnoticed by the user but provide more resiliency as well as more user friendly outcome whilst still being very specific in our logging. * a blank dict attribute is okay here since we know our refs our clean the getter calls will just return none * correct referencing of semantic type and not special * arg var renamed back to --database to ensure seamless compatibility/use with prev version. * improve typing ensuring python 3.6+ compatibility, expanduser called on path strings allowing relative paths for --dbt_path or --dbt_manifest_path simplified * explicit Any type hint for consistency * Use more generic and compatible typing operators. parse ref now guarantees us `schema.table` format. This allows us to guarantee the incorrectly formatted ref (which should be `schema.table`) is originating from yml. Log the warning and infer correct schema for our users using target schema which covers the 90% use case. * use mapping for column and express last bit of typing for bool args * re added clarification on semantic types being formerly known as special types * Following best practices, declare default args for lists as none setting as empty list in function call * docstrings to reflect default arg is None * updates to ensure use of warning instead of warn on logger, added __init__ to nonemptystr class, cleaned some logging calls to use lazy interpolation * simplified typing and ensure type tests pass * more typing updates * added mutablemapping types * explicit type for reader as being either manifest or yml * update tests to run on updated structure * point to right module for manifest reader * use same keyword arg as other reader for uniformity * format and fix req dev pointer * tests include assertions and dummy data from jaffle shop dbt classic example project * include a compiled manifest.json for test * ensure schema cast to upper for uniformity. * added last upper() call in folder parser. * move tests to fixtures folder and remove dummy CI tests * tests point to correct sample proj path * maintain expected default for sync and https, normalize help str text to be uniform (no ending periods) * make verify typing more specific * union typing with commas, not pipe * remove double import * let timeout type check as optional setting val if None in func * remove unecessary os path join * remove unecessary "" defaults for gets compared to literals * use f strings for readability * removed two end of line periods for uniformity in console help msg * fixed unintentional changes to readme * move ternary to separate statement for cleanliness * update dev req txt to name referred to in setup * fix misnamed test requirements to one specified in CI pipeline * update integrated str checks to use class model attributes * fix ref to requirements-test * use class attribute to populate variable so we can set to None or str for metabase api * added some nice debug logging messages to any model passed over in manifest reader for visibility * include sources propagation as provided in yml parser with the benefit of implicit schema handling provided by manifest. Co-authored-by: falador_wiz1 <[email protected]> Co-authored-by: Mike Gouline <[email protected]>
* Add Argument --mb_http When calling dbt-metabase export from the command line, any argument supplied to --mb_https evaluates to True. * Format __init__.py with Black Co-authored-by: erika-e <[email protected]>
* Make non-default boolean CLI flags consistent * Big refactor/reorder of arguments and names for consistency
* DbtFolder parser: collect schema info for source tables * PR Review tweaks * formatting tweaks * Removing schema argument
* Setup version improvements * Release trigger and PyPI publish
* Add .idea to gitignore * merge changes from read-from-artifacts fork integrating with latest version of dbt-metabase having improved yaml parsing. large formatting update to conform to black. * fix grammatical error and update readme to say semantic type * fix error in field lookup key setting and continue to support either special or semantic until ready to deprecate * internally use semantic but support meta refs to special which will be properly used if found in metabase api response. * safe importing of dependent modules in addition to bugfixes, logical synchronization, and formatting * added more verbose comments tracking intent as well as ensuring nodes without depends on/test_metadata dont throw * setting fields to PK type is worthy of info logging * handle cross schema foreign keys with explicit metabase.fk_ref with expected input format to be defined in readme. otherwise automatic resolution of target field using relation test will prepend target run schema which should be fine in 95% of use cases. cases outside that can use manifest.json parsing or set fk_ref in yml. * added debug log for validating parsed schema/fields for fk targets and ensured support for schema agnostic fk targets (schema resolved from manifest.json) * all necessary args added for cli usage. some basic assertions added. sync will now only fail hard if timeout is explicit, otherwise default behaviour if --sync is true is to attempt sync for 30 seconds and proceed with aligning what can be aligned successfully. more formatting and a few comments for clarity of intent. also added option to pass custom cert bundle to verify. * corrected typo in exclude var and added support for verbosity flag. translated some args to store action based. * updates to handle aliases when ran via dbt_path (yml parser) ubiquitously with seemless function alongside primary artifact parser (manifest.json). * support for aliased fk refs parsed via parse_ref. backoff of parse ref regex to permit catching last arg of either ref or source always being the target table. if pointing to an alias, we are collecting aliases during yml parsing to be passed to metabased client and translated to metabase table names as needed. this functionality should be unnoticed by the user but provide more resiliency as well as more user friendly outcome whilst still being very specific in our logging. * a blank dict attribute is okay here since we know our refs our clean the getter calls will just return none * correct referencing of semantic type and not special * arg var renamed back to --database to ensure seamless compatibility/use with prev version. * improve typing ensuring python 3.6+ compatibility, expanduser called on path strings allowing relative paths for --dbt_path or --dbt_manifest_path simplified * explicit Any type hint for consistency * Use more generic and compatible typing operators. parse ref now guarantees us `schema.table` format. This allows us to guarantee the incorrectly formatted ref (which should be `schema.table`) is originating from yml. Log the warning and infer correct schema for our users using target schema which covers the 90% use case. * use mapping for column and express last bit of typing for bool args * re added clarification on semantic types being formerly known as special types * Following best practices, declare default args for lists as none setting as empty list in function call * docstrings to reflect default arg is None * updates to ensure use of warning instead of warn on logger, added __init__ to nonemptystr class, cleaned some logging calls to use lazy interpolation * simplified typing and ensure type tests pass * more typing updates * added mutablemapping types * explicit type for reader as being either manifest or yml * update tests to run on updated structure * point to right module for manifest reader * use same keyword arg as other reader for uniformity * format and fix req dev pointer * tests include assertions and dummy data from jaffle shop dbt classic example project * include a compiled manifest.json for test * ensure schema cast to upper for uniformity. * added last upper() call in folder parser. * move tests to fixtures folder and remove dummy CI tests * tests point to correct sample proj path * maintain expected default for sync and https, normalize help str text to be uniform (no ending periods) * make verify typing more specific * union typing with commas, not pipe * remove double import * let timeout type check as optional setting val if None in func * remove unecessary os path join * remove unecessary "" defaults for gets compared to literals * use f strings for readability * removed two end of line periods for uniformity in console help msg * fixed unintentional changes to readme * move ternary to separate statement for cleanliness * update dev req txt to name referred to in setup * fix misnamed test requirements to one specified in CI pipeline * update integrated str checks to use class model attributes * fix ref to requirements-test * use class attribute to populate variable so we can set to None or str for metabase api * added some nice debug logging messages to any model passed over in manifest reader for visibility * include sources propagation as provided in yml parser with the benefit of implicit schema handling provided by manifest. * since we share so many args, make primary export function agnostic differentiating executed client method with getattr for the most DRY/clean approach * initial fully functional auto exposure export func built out * added some model fields to support differentiating between source / ref and another for constructing a dbt ref during model parsing for use downstream * build dbt ref/source jinja during model parsing as well as logging model source * placeholder for dataset query to be base64 encoded in exploratory link generation with compiled sql pulled from manifest parser * add assertions for commands, uniform dbt docs url arg * add recursions for extract card capable of extracting saved questions used as tables for sources or joins robustly. * uniformity in argument dbt_docs -> dbt_docs_url * much more detailed documentation provided for metabase analytics. * added new model_key and ref to unit tests * some typing updates * schema arg is unused for now * preparing to update the docs * add missing argvars for exposure parsing, default output args in working dir, fixed ref to dashboard created at key * remove default schema arg so we propagate down into manifest reader scema agnostically if unspecified * remove alpha sorting so we preserve our constructed format * give folder reader a default schema since it should have a separate invocation per schema * properly allow schema agnostic parsing if schema is not passed since we cprrectly resolce all schemas in manifest parsing * typing fix for schema arg * remove unused type * populating mock api for current and future testing * pre ran jaffle shop with artifacts * moved mock api to api sub dir to play very nicely with mock client api method and added baseline yml for testing * exposure extractor will return its output dict * test suite utils for dbt-metabase for recreating any test files * unit tests for exposure parser complete * fixed test to conform to freshly ran manifest on fresh db, old manifest was created via compile in fixture dir * ensure arg to the fresh db name used when compiling artifacts * leave the artifact for user introspection * added lookup artifacts generated from api calls * util func to rebuild lookup artifacts * added unit test for metadata lookups * remove func call at bottom * updates to include exposure parsing documentation usage * mock api clean up of user field * simplified personal collection check to use personal id key * removed dbt docs param from exposure parse, and small grammatical fixes * removed uneccessary mb prefix * overhaul to dbt-metabase cli interface and programmatic interface using config objects * removed extra arg, fixed missing t in command, and tested CLI in production * use get instead of key because root does not have personal id in keys * use dbt_models as arg to preserve models as main func name synonymously with cli * dropped empty init to lint properly, we have an init in parent and access these via namespace * pass bool to personal collection arg default, re-add init * add default to exclude arg * relative imports make more sense here namespace wise and pass linter * show defaults for command specific optional args * last bits of updates to readme to match updated programmatic interface * fix type in our * more accurate help string * rename args prefixing dbt_ and add schema arg back to cli * name iterable in such a way not conflict with var name in same scope * use DbtConfig as class name, docs_url when in dbt only context, and use context for open files in tests * all config args now match cli dest args exactly * updated readme to reflect args in programmatic section * update args to configs to not have prefix, unneeded * added const resource version to exposure func, made output path default "." * simplify nested if with a force cast so we can call startswith regardless of int/str * updated dumper to indent sequences as dbt documents show * refactor for readability improvements and easier flow tracing * final polishing improvements, documentation, and large readability improvements Co-authored-by: falador_wiz1 <[email protected]> Co-authored-by: Mike Gouline <[email protected]>
* Set explicit Python version on GitHub Actions * Install build requirements on GitHub Actions * feat: use Enum instead of Literal for Python 3.6+ compatibility Also wrap the use of importlib * feat: use Enum instead of Literal for Python 3.6+ compatibility Also wrap the use of importlib * Move build dependencies to requirements-test.txt * Fix typo * Refactor check for importlib existence Co-authored-by: Mike Gouline <[email protected]>
Co-authored-by: Falador_wiz1 <[email protected]>
…mmand (#65) * large refactor to use click as cli interface, improve and further abstract interface complexity, allow env vars implicitly * ephemeral model skip should be debug level like rest * full implementation of yaml config support in tandem to env var support, iteractive config generation and updates * add config load func to utils * add config global, override option process method to include config lookup which lets us reimplement required args * utility function to list env vars, validate in use env vars, inject env vars to render runtime config * last linting and typing updates and more robust type handler for config prompt lists * specify encoding for all file handlers * explicitly specify lru cache size * remove incorrect underscore in kwarg * update interface to inherit from config objects * little comment for clarity * readme updates to programmatic invocation and config * fix typo * fixed same type * remove uneeded sentence * add small note for manual config yml create and insight into layout. * rst needs a break between code-block decl and code * add method to use the resolved parser prop and execute read_models directly from interface * reflect dbt interface simplification in readme Co-authored-by: Falador_wiz1 <[email protected]>
Co-authored-by: Matthew Moy de Vitry <[email protected]>
|
As we discussed, this is not worth the effort since we are moving away from Metabase. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.