Skip to content

Refactor CI scripts and test fixes#304

Merged
99Lys merged 10 commits into
dremio:mainfrom
99Lys:DX-104286-refactor-dbt-pipelines
Aug 19, 2025
Merged

Refactor CI scripts and test fixes#304
99Lys merged 10 commits into
dremio:mainfrom
99Lys:DX-104286-refactor-dbt-pipelines

Conversation

@99Lys
Copy link
Copy Markdown
Contributor

@99Lys 99Lys commented Aug 5, 2025

Summary

Refactoring CI scripts and fixing some tests.

Description

[CI Refactoring]

  • Deleted create_dbt_test_users.sh → users are only needed for grants tests, which are not being ran in the current CI
  • Created extract_auth_token.sh and updated create_and_format_sources.sh → split authentication token extraction from sources creation
  • Updated create_env_file.sh → removed users and roles, as they are only needed for grants tests
  • Display the exit status of each test/job → this gives us more visibility on what group of tests failed exactly, unfortunately the pipeline as a whole will be considered as "failed", but we only need to consider the results of Verify Expected Test Failures job
  • Fixed erroring tests not being considered as failing tests → look for ERROR tests in artifacts as well, not only FAILED tests

[Tests Fixing]

  • Grants → override apply_grants macro to include user: if no prefix is defined, otherwise any rerun of the same model will lead to revoking and regranting; override tests to include prefix in the expected results
  • Hooks → add test prefix in dataset folder creation to avoid existing folder error

Test Results

  • All tests are passing/failing as expected

Changelog

  • Added a summary of what this PR accomplishes to CHANGELOG.md

@99Lys 99Lys force-pushed the DX-104286-refactor-dbt-pipelines branch 5 times, most recently from cec16c1 to 1b39e7a Compare August 7, 2025 21:13
@99Lys 99Lys marked this pull request as ready for review August 8, 2025 16:34
@99Lys 99Lys marked this pull request as draft August 8, 2025 16:35
@99Lys 99Lys force-pushed the DX-104286-refactor-dbt-pipelines branch 7 times, most recently from 849295b to 4b0cb8b Compare August 12, 2025 14:52
@99Lys 99Lys marked this pull request as ready for review August 12, 2025 14:57
bcmeireles
bcmeireles previously approved these changes Aug 12, 2025
Copy link
Copy Markdown
Contributor

@bcmeireles bcmeireles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test

@99Lys 99Lys marked this pull request as draft August 12, 2025 15:05
@99Lys 99Lys force-pushed the DX-104286-refactor-dbt-pipelines branch 2 times, most recently from 91a8452 to 7e7efa9 Compare August 12, 2025 15:28
@99Lys 99Lys force-pushed the DX-104286-refactor-dbt-pipelines branch from 7e7efa9 to 9d3c7b0 Compare August 12, 2025 16:01
@99Lys 99Lys marked this pull request as ready for review August 13, 2025 14:17
@99Lys 99Lys changed the title Adjust CI scripts Refactor CI scripts and test fixes Aug 13, 2025
Copy link
Copy Markdown
Contributor

@valterfrancisco-dremio valterfrancisco-dremio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just have one question

Comment thread .github/scripts/create_env_file.sh
@99Lys 99Lys merged commit e20e92c into dremio:main Aug 19, 2025
16 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants