Container image tagging strategy with dbt versions

[GideonStowell]

Related Slack thread

What changes are being made and why?

1. Building the container images every week and tagging them with latest causes the underlying code being run by consumers of these images to never be stable (dbt version is constantly being updated and is never predictable). Removing the scheduled build will cause the images to stabilize, even if it is not clear what version of dbt is being used.
2. Tagging the container images by dbt-version allows consumers to pin to a specific working version and manage upgrading to newer versions intentionally.

---

How the changes have been QAed?

id: dbt_version
namespace: test

tasks:
  - id: dbt_version
    type: io.kestra.plugin.dbt.cli.DbtCLI
    containerImage: ghcr.io/kestra-io/dbt-bigquery:latest #TODO change to specific version
    commands:
      - dbt --version

Contributor Checklist ✅

• PR Title and commits follows conventional commits
• Add a closes #ISSUE_ID or fixes #ISSUE_ID in the description if the PR relates to an opened issue.
• Documentation updated (plugin docs from @Schema for properties and outputs, @Plugin with examples, README.md file with basic knowledge and specifics).
• Setup instructions included if needed (API keys, accounts, etc.).
• Prefix all rendered properties by r not rendered (eg: rHost).
• Use runContext.logger() to log enough important infos where it's needed and with the best level (DEBUG, INFO, WARN or ERROR).

⚙️ Properties

• Properties are declared with Property<T> carrier type, do not use @PluginProperty.
• Mandatory properties must be annotated with @NotNull and checked during the rendering.
• You can model a JSON thanks to a simple Property<Map<String, Object>>.

🌐 HTTP

• Must use Kestra’s internal HTTP client from io.kestra.core.http.client

📦 JSON

• If you are serializing response from an external API, you may have to add a @JsonIgnoreProperties(ignoreUnknown = true) at the mapped class level. So that we will avoid to crash the plugin if the provider add a new field suddenly.
• Must use Jackson mappers provided by core (io.kestra.core.serializers)

✨ New plugins / subplugins

• Make sure your new plugin is configured like mentioned here.
• Add a package-info.java under each sub package respecting this format and choosing the right category.
• Every time you use runContext.metric(...) you have to add a @Metric (see this doc)
• Docs don't support to have both tasks/triggers in the root package (e.g. io.kestra.plugin.kubernetes) and in a sub package (e.g. io.kestra.plugin.kubernetes.kubectl), whether it's: all tasks/triggers in the root package OR only tasks/triggers in sub packages.
• Icons added in src/main/resources/icons in SVG format and not in thumbnail (keep it big):
  • plugin-icon.svg
  • One icon per package, e.g. io.kestra.plugin.aws.svg
  • For subpackages, e.g. io.kestra.plugin.aws.s3, add io.kestra.plugin.aws.s3.svg
    See example here.
• Use "{{ secret('YOUR_SECRET') }}" in the examples for sensible infos such as an API KEY.
• If you are fetching data (one, many or too many), you must add a Property<FetchType> fetchType to be able to use FETCH_ONE, FETCH and even STORE to store big amount of data in the internal storage.
• Align the """ to close examples blocks with the flow id.
• Update the existing index.yaml for the main plugin, and for each new subpackage add a metadata file named exactly after the subpackage (e.g. s3.yaml for io.kestra.plugin.aws.s3) under src/main/resources/metadata/, following the same schema.

🧪 Tests

• Unit Tests added or updated to cover the change (using the RunContext to actually run tasks).
• Add sanity checks if possible with a YAML flow inside src/test/resources/flows.
• Avoid disabling tests for CI. Instead, configure a local environment whenever it's possible with .github/setup-unit.sh (to be set executable with chmod +x setup-unit.sh) (which can be executed locally and in the CI) all along with a new docker-compose-ci.yml file (do not edit the existing docker-compose.yml). If needed, create an executable (chmod +x cleanup-unit.sh) cleanup-unit.sh to remove the potential costly resources (tables, datasets, etc).
• Provide screenshots from your QA / tests locally in the PR description. The goal here is to use the JAR of the plugin and directly test it locally in Kestra UI to ensure it integrates well.

📤 Outputs

• Do not send back as outputs the same infos you already have in your properties.
• If you do not have any output use VoidOutput.
• Do not output twice the same infos (eg: a status code, an error code saying the same thing...).

[fdelbrayelle]

Maybe not remove it but make it more frequent?

[GideonStowell]

No, I want it less frequent.

Respectfully, did you read the description? The point of removing the schedule is to prevent new images from being built every week and breaking our dbt workflows every monday morning.

[fdelbrayelle]

Then the filename shouldn't contain -scheduled anymore to fit with this logic ;-)

[fdelbrayelle]

Should be changed to main now that the main branch is main by the way

[fdelbrayelle]

@GideonStowell Did you have time to get back on the PR? Thanks 🙏

[MilosPaunovic]

Hey @GideonStowell, are there any updates on this?