Skip to content

feat(cli): Add image build command to airbyte-cdk CLI #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 51 commits into
base: aj/feat/add-standard-tests-cli
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
c65d832
feat: Add build command to CDK CLI
devin-ai-integration[bot] Apr 16, 2025
bd7a765
fix: Update formatting for build command files
devin-ai-integration[bot] Apr 16, 2025
2a32e27
feat: Add Click-based CLI interface
devin-ai-integration[bot] Apr 17, 2025
653bda1
chore: Update poetry.lock
devin-ai-integration[bot] Apr 17, 2025
9510f2d
fix: Add type annotations to CLI functions
devin-ai-integration[bot] Apr 17, 2025
84d649b
feat: Always build for both AMD64 and ARM64 architectures
devin-ai-integration[bot] Apr 17, 2025
f3e1743
fix: Format code with ruff
devin-ai-integration[bot] Apr 17, 2025
2bc8848
fix: Address PR comments - remove infer_connector_language, use build…
devin-ai-integration[bot] Apr 17, 2025
ca1bbbe
fix: Update Dockerignore to use * pattern and fix formatting
devin-ai-integration[bot] Apr 17, 2025
373f263
fix: Format all files with ruff
devin-ai-integration[bot] Apr 17, 2025
430dd8d
docs: Move build.md content to __init__.py docstring
devin-ai-integration[bot] Apr 17, 2025
fe946ac
docs: Update __init__.py docstring with build command documentation
devin-ai-integration[bot] Apr 17, 2025
af3e56e
Delete airbyte_cdk/cli/entrypoint/run.py
aaronsteers Apr 17, 2025
7b6dc5d
fix: Clean up duplicate code in _run.py and update imports
devin-ai-integration[bot] Apr 17, 2025
2b1db11
refactor: Move build functionality to utils/docker and metadata model…
devin-ai-integration[bot] Apr 17, 2025
4d45bab
fix: Remove unsupported --ignorefile flag and implement proper cleanu…
devin-ai-integration[bot] Apr 17, 2025
7e68925
fix: Add fallback to single platform build when multi-platform not av…
devin-ai-integration[bot] Apr 17, 2025
df4b315
fix: Update dockerignore to include source_* directories
devin-ai-integration[bot] Apr 17, 2025
83eaca9
enhancement: Improve build process to match airbyte-ci's approach wit…
devin-ai-integration[bot] Apr 18, 2025
789f109
fix: Apply ruff formatting to fix CI checks
devin-ai-integration[bot] Apr 18, 2025
ba09434
fix: Add run_command function to build.py and update imports
devin-ai-integration[bot] Apr 18, 2025
b25671b
fix: Add run_command function to build.py
devin-ai-integration[bot] Apr 18, 2025
d7ee4f1
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers Apr 18, 2025
e1c7c13
delete unnecessay
aaronsteers Apr 18, 2025
ba0ab6b
move docstring
aaronsteers Apr 18, 2025
61fdced
fully move model to models.connector_metadata
aaronsteers Apr 18, 2025
f293e7e
clean up
aaronsteers Apr 18, 2025
7daa7f0
fix docstring
aaronsteers Apr 18, 2025
4ba7e5c
clean up cli module
aaronsteers Apr 18, 2025
49c9045
revert format update
aaronsteers Apr 18, 2025
d9c1ced
lean into pydantic model
aaronsteers Apr 18, 2025
3103d0c
clean up
aaronsteers Apr 18, 2025
52546be
clean up
aaronsteers Apr 18, 2025
63cbc9b
fix: Address PR feedback - fix typo in __all__, update docstring, and…
devin-ai-integration[bot] Apr 18, 2025
c8f91cf
fix: Correct CLI entry point in pyproject.toml
devin-ai-integration[bot] Apr 18, 2025
71652ef
Apply suggestions from code review
aaronsteers Apr 18, 2025
78b173c
add-back the help text
aaronsteers Apr 18, 2025
598d454
refactor, tidy up
aaronsteers Apr 18, 2025
bcf5d9f
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers Apr 19, 2025
871cb71
refactor docker implementation
aaronsteers Apr 21, 2025
42af39d
toggle default arch, fix return value for verify step
aaronsteers Apr 21, 2025
964a215
better err handling
aaronsteers Apr 21, 2025
d7f9c7b
clean up
aaronsteers Apr 21, 2025
3e9183b
tidy docstring
aaronsteers Apr 21, 2025
6ea0623
ruff fix
aaronsteers Apr 21, 2025
ea5fd85
poe lock
aaronsteers Apr 21, 2025
0229421
fix mypy
aaronsteers Apr 21, 2025
297a98c
fix base image
aaronsteers Apr 21, 2025
f7b19e1
use uv instead of pip for installs
aaronsteers Apr 21, 2025
7a87b62
use rich click
aaronsteers Apr 21, 2025
f7e4ddf
refactor, use build args
aaronsteers Apr 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion airbyte_cdk/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,9 @@
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""The `airbyte-cdk.cli` module provides command-line interfaces for the Airbyte CDK.

As of now, it includes the following CLI entry points:

- `airbyte-cdk`: Commands for working with connectors.
- `source-declarative-manifest`: Directly invoke the declarative manifests connector.

"""
76 changes: 67 additions & 9 deletions airbyte_cdk/cli/airbyte_cdk/_image.py
Original file line number Diff line number Diff line change
@@ -1,22 +1,80 @@
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""Docker image commands.
"""Airbyte CDK 'image' commands.

Coming soon.

This module is planned to provide a command line interface (CLI) for building
Docker images for Airbyte CDK connectors.
The `airbyte-cdk image build` command provides a simple way to work with Airbyte
connector images.
"""

import click
import sys
from pathlib import Path

import rich_click as click

from airbyte_cdk.cli.airbyte_cdk._util import resolve_connector_name_and_directory
from airbyte_cdk.models.connector_metadata import MetadataFile
from airbyte_cdk.utils.docker import (
build_connector_image,
verify_docker_installation,
)


@click.group(
name="image",
help=__doc__.replace("\n", "\n\n"), # Render docstring as help text (markdown)
)
def image_cli_group() -> None:
"""Docker image commands."""
pass
"""Commands for working with connector Docker images."""


@image_cli_group.command()
@click.option(
"--connector-name",
type=str,
help="Name of the connector to test. Ignored if --connector-directory is provided.",
)
@click.option(
"--connector-directory",
type=click.Path(exists=True, file_okay=False, path_type=Path),
help="Path to the connector directory.",
)
@click.option("--tag", default="dev", help="Tag to apply to the built image (default: dev)")
@click.option("--no-verify", is_flag=True, help="Skip verification of the built image")
def build(
connector_name: str | None = None,
connector_directory: Path | None = None,
*,
tag: str = "dev",
no_verify: bool = False,
) -> None:
"""Build a connector Docker image.

This command builds a Docker image for a connector, using either
the connector's Dockerfile or a base image specified in the metadata.
The image is built for both AMD64 and ARM64 architectures.
"""
if not verify_docker_installation():
click.echo(
"Docker is not installed or not running. Please install Docker and try again.", err=True
)
sys.exit(1)

connector_name, connector_directory = resolve_connector_name_and_directory(
connector_name=connector_name,
connector_directory=connector_directory,
)

metadata = MetadataFile.from_file(connector_directory / "metadata.yaml")
click.echo(
f"Building Image for Connector: {metadata.data.dockerRepository} "
f"(v{metadata.data.dockerImageTag})"
)
build_connector_image(
connector_directory=connector_directory,
connector_name=connector_name,
metadata=metadata,
tag=tag,
arch="linux/amd64",
no_verify=no_verify,
)


__all__ = [
Expand Down
78 changes: 78 additions & 0 deletions airbyte_cdk/models/connector_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
"""Models to represent the structure of a `metadata.yaml` file."""

from __future__ import annotations

from enum import Enum
from pathlib import Path
from typing import Any

import yaml
from pydantic import BaseModel, Field


class ConnectorLanguage(str, Enum):
"""Connector implementation language."""

PYTHON = "python"
JAVA = "java"
LOW_CODE = "low-code"
MANIFEST_ONLY = "manifest-only"
UNKNOWN = "unknown"


class ConnectorBuildOptions(BaseModel):
"""Connector build options from metadata.yaml."""

model_config = {"extra": "allow"}

baseImage: str | None = Field(
None,
description="Base image to use for building the connector",
)
path: str | None = Field(
None,
description="Path to the connector code within the repository",
)


class ConnectorMetadata(BaseModel):
"""Connector metadata from metadata.yaml."""

model_config = {"extra": "allow"}

dockerRepository: str = Field(..., description="Docker repository for the connector image")
dockerImageTag: str = Field(..., description="Docker image tag for the connector")
language: ConnectorLanguage | None = Field(
None, description="Language of the connector implementation"
)
connectorBuildOptions: ConnectorBuildOptions | None = Field(
None, description="Options for building the connector"
)


class MetadataFile(BaseModel):
"""Represents the structure of a metadata.yaml file."""

model_config = {"extra": "allow"}

data: ConnectorMetadata = Field(..., description="Connector metadata")

@classmethod
def from_file(
cls,
file_path: Path,
) -> MetadataFile:
"""Load metadata from a YAML file."""
if not file_path.exists():
raise FileNotFoundError(f"Metadata file not found: {file_path!s}")

metadata_content = file_path.read_text()
metadata_dict = yaml.safe_load(metadata_content)

if not metadata_dict or "data" not in metadata_dict:
raise ValueError(
"Invalid metadata format: missing 'data' field in YAML file '{file_path!s}'"
)

metadata_file = MetadataFile.model_validate(metadata_dict)
return metadata_file
Loading
Loading