Skip to content

feat(cli): Add image build command to airbyte-cdk CLI #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 51 commits into
base: aj/feat/add-standard-tests-cli
Choose a base branch
from

Conversation

devin-ai-integration[bot]
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Apr 16, 2025

Notes from AJ (@aaronsteers):

This PR allows you to quickly build a docker image for your connector. This is a follow-on to the previous PR that added connector test and secrets fetch capabilities. It uses most of the same practices as the official airbyte-ci pipeline, but it is faster and has some modest improvements.

See the base PR for more context on the CLI in general:

Other notes:

  • For now, this always does a multi-arch build, so (assuming connector is sources-s3 you'll have images:
    • airbyte/source-s3:dev-linux-amd64
    • airbyte/source-s3:dev-linux-arm64
    • airbyte/source-s3:dev (same as the ARM64 one, assuming for now that local dev is happening on an M-series Mac).
  • This uses uv instead of pip to do the install. This speeds up the builds significantly.
  • As of now, this only supports Python connectors.
    • Manifest-only connectors will be added to this PR before merging.
    • Java/Kotlin connectors may be added, but are not currently the priority.
  • As with the other airbyte-cdk CLI commands, you can run this statelessly using uvx airbyce-cdk image build. This doesn't require pre-installing the CDK, and will use the latest CDK version by default.
  • There's no current plan to replace/remove airbyte-ci for build+publish. We can discuss this, but currently, this functionality is only for dev/test workflows.

https://www.loom.com/share/5ca23374a5e24af1a4163597e3048154?sid=47e4cfa3-3a03-4eea-beb3-9a9dca3cb0ea

Notes From Devin:

Adds a new airbyte-cdk image build command to the Python CDK that allows building connector Docker images directly using Docker, without requiring the Dagger pipeline used in airbyte-ci build.

This PR:

  • Adds a new CLI entrypoint for airbyte-cdk image build
  • Implements Docker image building using the host's Docker daemon
  • Supports different connector types based on metadata and file structure
  • Verifies built images by running the spec command
  • Includes documentation for the new command

Requested by: Aaron ("AJ") Steers ([email protected])
Link to Devin run: https://app.devin.ai/sessions/6a63c18a1da44463a87b2099bcd1c092

Co-Authored-By: Aaron <AJ> Steers <[email protected]>
Copy link
Contributor Author

Original prompt from Aaron:

@Devin - Can you analyze the airbyte-ci build command in the main airbyte repo? I'd like to migrate this logic into the Python CDK, under a new cli entrypoint and command called "airbyte-cdk build". Exclude the process of building base images.

Copy link
Contributor Author

devin-ai-integration bot commented Apr 16, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added the enhancement New feature or request label Apr 16, 2025
@devin-ai-integration devin-ai-integration bot changed the title feat: Add build command to CDK CLI (do not merge) feat(cli): Add build command to CDK CLI (do not merge) Apr 16, 2025
@aaronsteers aaronsteers marked this pull request as draft April 16, 2025 22:23
@devin-ai-integration devin-ai-integration bot changed the title feat(cli): Add build command to CDK CLI (do not merge) feat(cli): Add build command to CDK CLI Apr 16, 2025
@devin-ai-integration devin-ai-integration bot marked this pull request as ready for review April 16, 2025 22:29
@aaronsteers aaronsteers force-pushed the devin/1744841809-add-build-command branch from d2c9fc9 to 71652ef Compare April 18, 2025 04:34
@aaronsteers aaronsteers changed the title feat(cli): Add build command to CDK CLI feat(cli): Add image build command to airbyte-cdk CLI Apr 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant