Skip to content

feat(cli): Add image build command to airbyte-cdk CLI #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 51 commits into
base: aj/feat/add-standard-tests-cli
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
c65d832
feat: Add build command to CDK CLI
devin-ai-integration[bot] Apr 16, 2025
bd7a765
fix: Update formatting for build command files
devin-ai-integration[bot] Apr 16, 2025
2a32e27
feat: Add Click-based CLI interface
devin-ai-integration[bot] Apr 17, 2025
653bda1
chore: Update poetry.lock
devin-ai-integration[bot] Apr 17, 2025
9510f2d
fix: Add type annotations to CLI functions
devin-ai-integration[bot] Apr 17, 2025
84d649b
feat: Always build for both AMD64 and ARM64 architectures
devin-ai-integration[bot] Apr 17, 2025
f3e1743
fix: Format code with ruff
devin-ai-integration[bot] Apr 17, 2025
2bc8848
fix: Address PR comments - remove infer_connector_language, use build…
devin-ai-integration[bot] Apr 17, 2025
ca1bbbe
fix: Update Dockerignore to use * pattern and fix formatting
devin-ai-integration[bot] Apr 17, 2025
373f263
fix: Format all files with ruff
devin-ai-integration[bot] Apr 17, 2025
430dd8d
docs: Move build.md content to __init__.py docstring
devin-ai-integration[bot] Apr 17, 2025
fe946ac
docs: Update __init__.py docstring with build command documentation
devin-ai-integration[bot] Apr 17, 2025
af3e56e
Delete airbyte_cdk/cli/entrypoint/run.py
aaronsteers Apr 17, 2025
7b6dc5d
fix: Clean up duplicate code in _run.py and update imports
devin-ai-integration[bot] Apr 17, 2025
2b1db11
refactor: Move build functionality to utils/docker and metadata model…
devin-ai-integration[bot] Apr 17, 2025
4d45bab
fix: Remove unsupported --ignorefile flag and implement proper cleanu…
devin-ai-integration[bot] Apr 17, 2025
7e68925
fix: Add fallback to single platform build when multi-platform not av…
devin-ai-integration[bot] Apr 17, 2025
df4b315
fix: Update dockerignore to include source_* directories
devin-ai-integration[bot] Apr 17, 2025
83eaca9
enhancement: Improve build process to match airbyte-ci's approach wit…
devin-ai-integration[bot] Apr 18, 2025
789f109
fix: Apply ruff formatting to fix CI checks
devin-ai-integration[bot] Apr 18, 2025
ba09434
fix: Add run_command function to build.py and update imports
devin-ai-integration[bot] Apr 18, 2025
b25671b
fix: Add run_command function to build.py
devin-ai-integration[bot] Apr 18, 2025
d7ee4f1
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers Apr 18, 2025
e1c7c13
delete unnecessay
aaronsteers Apr 18, 2025
ba0ab6b
move docstring
aaronsteers Apr 18, 2025
61fdced
fully move model to models.connector_metadata
aaronsteers Apr 18, 2025
f293e7e
clean up
aaronsteers Apr 18, 2025
7daa7f0
fix docstring
aaronsteers Apr 18, 2025
4ba7e5c
clean up cli module
aaronsteers Apr 18, 2025
49c9045
revert format update
aaronsteers Apr 18, 2025
d9c1ced
lean into pydantic model
aaronsteers Apr 18, 2025
3103d0c
clean up
aaronsteers Apr 18, 2025
52546be
clean up
aaronsteers Apr 18, 2025
63cbc9b
fix: Address PR feedback - fix typo in __all__, update docstring, and…
devin-ai-integration[bot] Apr 18, 2025
c8f91cf
fix: Correct CLI entry point in pyproject.toml
devin-ai-integration[bot] Apr 18, 2025
71652ef
Apply suggestions from code review
aaronsteers Apr 18, 2025
78b173c
add-back the help text
aaronsteers Apr 18, 2025
598d454
refactor, tidy up
aaronsteers Apr 18, 2025
bcf5d9f
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers Apr 19, 2025
871cb71
refactor docker implementation
aaronsteers Apr 21, 2025
42af39d
toggle default arch, fix return value for verify step
aaronsteers Apr 21, 2025
964a215
better err handling
aaronsteers Apr 21, 2025
d7f9c7b
clean up
aaronsteers Apr 21, 2025
3e9183b
tidy docstring
aaronsteers Apr 21, 2025
6ea0623
ruff fix
aaronsteers Apr 21, 2025
ea5fd85
poe lock
aaronsteers Apr 21, 2025
0229421
fix mypy
aaronsteers Apr 21, 2025
297a98c
fix base image
aaronsteers Apr 21, 2025
f7b19e1
use uv instead of pip for installs
aaronsteers Apr 21, 2025
7a87b62
use rich click
aaronsteers Apr 21, 2025
f7e4ddf
refactor, use build args
aaronsteers Apr 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion airbyte_cdk/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,9 @@
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""The `airbyte-cdk.cli` module provides command-line interfaces for the Airbyte CDK.

As of now, it includes the following commands:

- `airbyte-cdk`: Commands for working with connectors.
- `source-declarative-manifest`: Directly invoke the declarative manifests connector.

"""
133 changes: 128 additions & 5 deletions airbyte_cdk/cli/airbyte_cdk/_image.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,138 @@
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""Docker image commands."""
"""Airbyte CDK 'image' commands.

The `airbyte-cdk` command provides a simplified way to build connector Docker images without requiring the full Airbyte CI pipeline.

```bash
pip install airbyte-cdk

pipx run airbyte-cdk image [arguments]
```


```bash
airbyte-cdk image build /path/to/connector

airbyte-cdk image build /path/to/connector --tag custom_tag

airbyte-cdk image build /path/to/connector --no-verify

airbyte-cdk image build /path/to/connector --verbose
```


- `connector_dir`: Path to the connector directory (required)
- `--tag`: Tag to apply to the built image (default: "dev")
- `--no-verify`: Skip verification of the built image
- `--verbose`, `-v`: Enable verbose logging


The command reads the connector's metadata from the `metadata.yaml` file, builds a Docker image using the connector's Dockerfile, and verifies the image by running the `spec` command. The image is tagged according to the repository name specified in the metadata and the provided tag.

This command is designed to be a simpler alternative to the `airbyte-ci build` command, using Docker directly on the host machine instead of Dagger.
"""

import sys
from pathlib import Path

import click

from airbyte_cdk.models.connector_metadata import MetadataFile
from airbyte_cdk.utils.docker import (
build_from_base_image,
build_from_dockerfile,
verify_docker_installation,
verify_image,
)


@click.group(name="image")
def image_cli_group() -> None:
"""Docker image commands."""
pass
"""Commands for working with connector Docker images."""


@image_cli_group.command()
@click.argument(
"connector_directory",
type=click.Path(exists=True, file_okay=False, dir_okay=True, path_type=Path),
)
@click.option("--tag", default="dev", help="Tag to apply to the built image (default: dev)")
@click.option("--no-verify", is_flag=True, help="Skip verification of the built image")
@click.option("--verbose", "-v", is_flag=True, help="Enable verbose logging")
def build(
connector_directory: Path,
tag: str = "dev",
no_verify: bool = False,
verbose: bool = False,
) -> None:
"""Build a connector Docker image.

This command builds a Docker image for a connector, using either
the connector's Dockerfile or a base image specified in the metadata.
The image is built for both AMD64 and ARM64 architectures.
"""
if not verify_docker_installation():
click.echo(
"Docker is not installed or not running. Please install Docker and try again.", err=True
)
sys.exit(1)

try:
metadata = MetadataFile.from_file(connector_directory / "metadata.yaml")
click.echo(f"Connector: {metadata.data.dockerRepository}")
click.echo(f"Version: {metadata.data.dockerImageTag}")

if metadata.data.language:
click.echo(f"Connector language from metadata: {metadata.data.language}")
else:
click.echo("Connector language not specified in metadata")

try:
import subprocess

result = subprocess.run(
["docker", "buildx", "inspect"], capture_output=True, text=True, check=False
)

if "linux/amd64" in result.stdout and "linux/arm64" in result.stdout:
platforms = "linux/amd64,linux/arm64"
click.echo(f"Building for platforms: {platforms}")
else:
platforms = "linux/amd64"
click.echo(
f"Multi-platform build not available. Building for platform: {platforms}"
)
click.echo(
"To enable multi-platform builds, configure Docker buildx with: docker buildx create --use"
)
except Exception:
platforms = "linux/amd64"
click.echo(f"Multi-platform build check failed. Building for platform: {platforms}")

if metadata.data.connectorBuildOptions and metadata.data.connectorBuildOptions.baseImage:
image_name = build_from_base_image(connector_directory, metadata, tag, platforms)
else:
image_name = build_from_dockerfile(connector_directory, metadata, tag, platforms)

if not no_verify:
if verify_image(image_name):
click.echo(f"Build completed successfully: {image_name}")
sys.exit(0)
else:
click.echo(f"Built image failed verification: {image_name}", err=True)
sys.exit(1)
else:
click.echo(f"Build completed successfully (without verification): {image_name}")
sys.exit(0)

except Exception as e:
click.echo(f"Error: {str(e)}", err=True)
if verbose:
import traceback

click.echo(traceback.format_exc(), err=True)
sys.exit(1)


__all__ = [
__all___ = [
"image_cli_group",
]
78 changes: 78 additions & 0 deletions airbyte_cdk/models/connector_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
"""Models to represent the structure of a `metadata.yaml` file."""

from __future__ import annotations

from enum import Enum
from pathlib import Path
from typing import Any

import yaml
from pydantic import BaseModel, Field


class ConnectorLanguage(str, Enum):
"""Connector implementation language."""

PYTHON = "python"
JAVA = "java"
LOW_CODE = "low-code"
MANIFEST_ONLY = "manifest-only"
UNKNOWN = "unknown"


class ConnectorBuildOptions(BaseModel):
"""Connector build options from metadata.yaml."""

model_config = {"extra": "allow"}

baseImage: str | None = Field(
None,
description="Base image to use for building the connector",
)
path: str | None = Field(
None,
description="Path to the connector code within the repository",
)


class ConnectorMetadata(BaseModel):
"""Connector metadata from metadata.yaml."""

model_config = {"extra": "allow"}

dockerRepository: str = Field(..., description="Docker repository for the connector image")
dockerImageTag: str = Field(..., description="Docker image tag for the connector")
language: ConnectorLanguage | None = Field(
None, description="Language of the connector implementation"
)
connectorBuildOptions: ConnectorBuildOptions | None = Field(
None, description="Options for building the connector"
)


class MetadataFile(BaseModel):
"""Represents the structure of a metadata.yaml file."""

model_config = {"extra": "allow"}

data: ConnectorMetadata = Field(..., description="Connector metadata")

@classmethod
def from_file(
cls,
file_path: Path,
) -> MetadataFile:
"""Load metadata from a YAML file."""
if not file_path.exists():
raise FileNotFoundError(f"Metadata file not found: {file_path!s}")

metadata_content = file_path.read_text()
metadata_dict = yaml.safe_load(metadata_content)

if not metadata_dict or "data" not in metadata_dict:
raise ValueError(
"Invalid metadata format: missing 'data' field in YAML file '{file_path!s}'"
)

metadata_file = MetadataFile.model_validate(metadata_dict)
return metadata_file
Loading
Loading