-queue`
+- Messages are processed one at a time (`BatchSize: 1`)
+- `VisibilityTimeout` is set to 960s (Lambda timeout + buffer)
diff --git a/docs/nav/cloud/lambda.md b/docs/nav/cloud/lambda.md
new file mode 100644
index 00000000..d40ff64f
--- /dev/null
+++ b/docs/nav/cloud/lambda.md
@@ -0,0 +1,79 @@
+# AWS Lambda
+
+Deploy your dotflow pipeline as a container-based AWS Lambda function.
+
+## Create project
+
+```bash
+dotflow init
+# Select cloud: lambda
+```
+
+Or generate files for an existing project:
+
+```bash
+dotflow cloud generate --platform lambda
+```
+
+## Generated files
+
+| File | Description |
+|------|-------------|
+| `Dockerfile` | Lambda container image |
+| `handler.py` | Lambda entry point that calls your workflow |
+| `template.yaml` | SAM template defining the function |
+| `samconfig.toml` | Pre-configured deployment settings |
+
+## Prerequisites
+
+- `pip install dotflow[aws]`
+- AWS CLI configured (`aws configure`)
+- Docker
+
+## Deploy
+
+### Option 1: dotflow deploy
+
+```bash
+dotflow deploy --platform lambda --project my_pipeline
+```
+
+### Option 2: SAM CLI
+
+```bash
+aws ecr create-repository --repository-name my_pipeline --region us-east-1
+sam build
+sam deploy
+```
+
+## Invoke
+
+```bash
+# Via dotflow deploy
+aws lambda invoke --function-name my_pipeline --region us-east-1 /dev/stdout
+
+# Via SAM
+sam remote invoke
+```
+
+## View logs
+
+```bash
+sam logs --stack-name my_pipeline --tail
+```
+
+## Variants
+
+| Platform | Trigger | Deploy method |
+|----------|---------|---------------|
+| `lambda` | Manual invocation | `dotflow deploy` or `sam deploy` |
+| `lambda-scheduled` | EventBridge cron/rate | `dotflow deploy --schedule` or `sam deploy` |
+| `lambda-s3-trigger` | S3 file upload | `dotflow deploy` or `sam deploy` |
+| `lambda-sqs-trigger` | SQS message | `dotflow deploy` or `sam deploy` |
+| `lambda-api-trigger` | HTTP POST via API Gateway | `dotflow deploy` or `sam deploy` |
+
+## Important
+
+- Do not rename `handler.py` or the `handler()` function — the Dockerfile CMD is `handler.handler`
+- Do not rename `workflow.py` or the `main()` function — `handler.py` imports it
+- The `samconfig.toml` has stack name, region, and ECR repository pre-configured
diff --git a/docs/nav/cloud/nomad.md b/docs/nav/cloud/nomad.md
new file mode 100644
index 00000000..6b5075aa
--- /dev/null
+++ b/docs/nav/cloud/nomad.md
@@ -0,0 +1,5 @@
+# Nomad
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/pubsub-trigger.md b/docs/nav/cloud/pubsub-trigger.md
new file mode 100644
index 00000000..3dec67c7
--- /dev/null
+++ b/docs/nav/cloud/pubsub-trigger.md
@@ -0,0 +1,5 @@
+# Pubsub Trigger
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/railway.md b/docs/nav/cloud/railway.md
new file mode 100644
index 00000000..69d3e9b6
--- /dev/null
+++ b/docs/nav/cloud/railway.md
@@ -0,0 +1,5 @@
+# Railway
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/render.md b/docs/nav/cloud/render.md
new file mode 100644
index 00000000..c73d18eb
--- /dev/null
+++ b/docs/nav/cloud/render.md
@@ -0,0 +1,5 @@
+# Render
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/step-functions.md b/docs/nav/cloud/step-functions.md
new file mode 100644
index 00000000..5cb33588
--- /dev/null
+++ b/docs/nav/cloud/step-functions.md
@@ -0,0 +1,5 @@
+# Step Functions
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/terraform.md b/docs/nav/cloud/terraform.md
new file mode 100644
index 00000000..66946037
--- /dev/null
+++ b/docs/nav/cloud/terraform.md
@@ -0,0 +1,5 @@
+# Terraform
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/cloud/vercel.md b/docs/nav/cloud/vercel.md
new file mode 100644
index 00000000..a920ed4f
--- /dev/null
+++ b/docs/nav/cloud/vercel.md
@@ -0,0 +1,5 @@
+# Vercel
+
+:soon: **Coming soon**
+
+This platform is not yet available. Check the [template repository](https://github.com/dotflow-io/template) for updates.
diff --git a/docs/nav/concepts/concept-providers.md b/docs/nav/concepts/concept-providers.md
index f07baf6e..6c9779de 100644
--- a/docs/nav/concepts/concept-providers.md
+++ b/docs/nav/concepts/concept-providers.md
@@ -1,26 +1,51 @@
# Providers
-**Providers** are pluggable backends wired through [`Config`](../reference/config.md). They let the same workflow code use different storage, notifications, logging, and scheduling without changing task logic.
+**Providers** are pluggable backends wired through [`Config`](../reference/config.md). They let the same workflow code use different storage, notifications, logging, tracing, metrics, and scheduling without changing task logic.
## Families
-| Provider | Purpose |
-|----------|---------|
-| **Storage** | Persists task context and checkpoints (memory, file, S3, GCS, …). |
-| **Notify** | Sends alerts or summaries (default no-op, Telegram, …). |
-| **Log** | Structured logging for runs. |
-| **Scheduler** | Drives recurring execution (default, cron with `dotflow[scheduler]`). |
-
-You pass instances into `Config`, for example `Config(storage=StorageFile(path=".output"), scheduler=SchedulerCron(...))`. Built-in providers use **core** dependencies; cloud and cron integrations often need **pip extras**—see [Use integrations](../integrations/use-integrations.md) and the [Overview](../integrations/index.md) hub.
+| Provider | Purpose | Built-in | Integrations |
+|----------|---------|----------|--------------|
+| **Storage** | Persists task context and checkpoints | Default (memory), File | S3, GCS |
+| **Notify** | Sends alerts on status changes | Default (no-op) | Telegram, Discord |
+| **Log** | Structured logging for runs | Default (console/file) | OpenTelemetry, Sentry |
+| **Tracer** | Distributed traces per workflow/task | Default (no-op) | OpenTelemetry, Sentry |
+| **Metrics** | Counters and histograms | Default (no-op) | OpenTelemetry |
+| **Scheduler** | Drives recurring execution | Default (no-op) | Cron |
+
+You pass instances into `Config`:
+
+```python
+from dotflow import Config
+from dotflow.providers import StorageFile, LogDefault, TracerOpenTelemetry
+
+config = Config(
+ storage=StorageFile(path=".output"),
+ log=LogDefault(output="console", format="json"),
+ tracer=TracerOpenTelemetry(service_name="my-pipeline"),
+)
+```
+
+Built-in providers use **core** dependencies. Cloud and observability integrations need **pip extras**:
+
+| Extra | Install | Providers |
+|-------|---------|-----------|
+| `dotflow[scheduler]` | croniter | SchedulerCron |
+| `dotflow[aws]` | boto3 | StorageS3 |
+| `dotflow[gcp]` | google-cloud-storage | StorageGCS |
+| `dotflow[otel]` | opentelemetry-api, opentelemetry-sdk | LogOpenTelemetry, TracerOpenTelemetry, MetricsOpenTelemetry |
+| `dotflow[sentry]` | sentry-sdk | LogSentry, TracerSentry |
## Why it matters
- **Durability**: storage + [checkpoints](../tutorial/checkpoint.md) enable resume after failure.
-- **Observability**: log and notify providers surface failures and retries.
+- **Observability**: log, tracer, and metrics providers surface failures, retries, and performance.
+- **Notifications**: notify providers alert teams via Telegram, Discord, or custom webhooks.
- **Operations**: swap file storage in development for S3 or GCS in production with one config change.
## References
- [`Config`](../reference/config.md)
- [Custom providers](../development/custom-providers.md)
-- Abstract bases: [`Storage`](../reference/abc-storage.md), [`Notify`](../reference/abc-notify.md), [`Log`](../reference/abc-log.md), [`Scheduler`](../reference/abc-scheduler.md)
+- [Integrations](../integrations/index.md)
+- Abstract bases: [`Storage`](../reference/abc-storage.md), [`Notify`](../reference/abc-notify.md), [`Log`](../reference/abc-log.md), [`Scheduler`](../reference/abc-scheduler.md), [`Tracer`](../reference/abc-tracer.md), [`Metrics`](../reference/abc-metrics.md)
diff --git a/docs/nav/how-to/index.md b/docs/nav/how-to/index.md
index ff9bbd2f..6ff96c5a 100644
--- a/docs/nav/how-to/index.md
+++ b/docs/nav/how-to/index.md
@@ -2,14 +2,6 @@
-- :material-cog-outline: __Providers__
-
- ---
-
- Connect **storage**, **notify**, **log**, and **scheduler** backends. Submenus group tutorials by provider type.
-
- [:octicons-arrow-right-24: How-to guides](../tutorial/storage-default.md)
-
- :material-graph-outline: __Workflows__
---
diff --git a/dotflow/__init__.py b/dotflow/__init__.py
index e22e62be..cdfd4e8e 100644
--- a/dotflow/__init__.py
+++ b/dotflow/__init__.py
@@ -1,6 +1,6 @@
"""Dotflow __init__ module."""
-__version__ = "0.14.1.dev2"
+__version__ = "0.15.0.dev2"
__description__ = "🎲 Dotflow turns an idea into flow!"
from .core.action import Action as action
diff --git a/dotflow/cli/commands/__init__.py b/dotflow/cli/commands/__init__.py
index 1c2a7fc0..0f66e5e1 100644
--- a/dotflow/cli/commands/__init__.py
+++ b/dotflow/cli/commands/__init__.py
@@ -1,8 +1,18 @@
"""Commands __init__ module."""
+from dotflow.cli.commands.cloud import CloudGenerateCommand, CloudListCommand
+from dotflow.cli.commands.deploy import DeployCommand
from dotflow.cli.commands.init import InitCommand
from dotflow.cli.commands.log import LogCommand
from dotflow.cli.commands.schedule import ScheduleCommand
from dotflow.cli.commands.start import StartCommand
-__all__ = ["InitCommand", "LogCommand", "ScheduleCommand", "StartCommand"]
+__all__ = [
+ "CloudGenerateCommand",
+ "CloudListCommand",
+ "DeployCommand",
+ "InitCommand",
+ "LogCommand",
+ "ScheduleCommand",
+ "StartCommand",
+]
diff --git a/dotflow/cli/commands/cloud.py b/dotflow/cli/commands/cloud.py
new file mode 100644
index 00000000..44916741
--- /dev/null
+++ b/dotflow/cli/commands/cloud.py
@@ -0,0 +1,201 @@
+"""Command cloud module"""
+
+import json
+import re
+from pathlib import Path
+from subprocess import run
+
+from rich import print # type: ignore
+from rich.console import Console
+from rich.table import Table
+
+from dotflow.cli.command import Command
+from dotflow.settings import Settings as settings
+
+
+def _get_template_dir() -> Path | None:
+ import tempfile
+
+ target = Path(tempfile.gettempdir()) / "dotflow-template"
+ cloud_dir = target / settings.TEMPLATE_CLOUD_DIR
+
+ if cloud_dir.exists():
+ run(
+ ["git", "-C", str(target), "pull", "--ff-only"],
+ capture_output=True,
+ text=True,
+ )
+ return cloud_dir
+
+ try:
+ result = run(
+ [
+ "git",
+ "clone",
+ "--depth",
+ "1",
+ "--branch",
+ settings.TEMPLATE_BRANCH,
+ settings.TEMPLATE_REPO,
+ str(target),
+ ],
+ capture_output=True,
+ text=True,
+ )
+ if result.returncode == 0:
+ return cloud_dir
+ except Exception as err:
+ print(
+ settings.ERROR_ALERT,
+ f"Failed to clone template repository: {err}",
+ )
+
+ return None
+
+
+def _load_registry(cloud_dir: Path) -> dict | None:
+ registry_path = cloud_dir / "registry.json"
+ if not registry_path.exists():
+ print(settings.ERROR_ALERT, "Registry not found in template.")
+ return None
+
+ try:
+ return json.loads(registry_path.read_text())
+ except Exception as err:
+ print(settings.ERROR_ALERT, f"Failed to read registry: {err}")
+ return None
+
+
+def _get_platforms(registry: dict) -> dict | None:
+ platforms = registry.get("platforms")
+ if not isinstance(platforms, dict):
+ print(settings.ERROR_ALERT, "Unexpected registry format.")
+ return None
+
+ return platforms
+
+
+def _read_project_name(pyproject: Path) -> str | None:
+ try:
+ in_project = False
+ for line in pyproject.read_text().splitlines():
+ stripped = line.strip()
+ if stripped.startswith("["):
+ in_project = stripped in ("[project]", "[tool.poetry]")
+ if in_project and re.match(r"^name\s*=", stripped):
+ return stripped.split("=", 1)[1].strip().strip('"').strip("'")
+ except Exception:
+ return None
+ return None
+
+
+class CloudGenerateCommand(Command):
+ def setup(self):
+ platform = self.params.platform
+
+ cloud_dir = _get_template_dir()
+ if cloud_dir is None:
+ return
+
+ registry = _load_registry(cloud_dir)
+ if registry is None:
+ return
+
+ platforms = _get_platforms(registry)
+ if platforms is None:
+ return
+
+ if platform not in platforms:
+ available = ", ".join(sorted(platforms.keys()))
+ print(
+ settings.ERROR_ALERT,
+ f"Unknown platform '{platform}'. Available: {available}",
+ )
+ return
+
+ platform_info = platforms[platform]
+ files = platform_info.get("files", [])
+ output = Path(self.params.output).resolve()
+ output.mkdir(parents=True, exist_ok=True)
+
+ project_name = self.params.project
+ if not project_name:
+ pyproject = Path.cwd() / "pyproject.toml"
+ if pyproject.exists():
+ project_name = _read_project_name(pyproject)
+ if not project_name:
+ project_name = Path.cwd().name
+
+ module_name = project_name.replace("-", "_")
+
+ print(
+ settings.INFO_ALERT,
+ f"Generating {platform_info.get('name', platform)} files"
+ f" for '{project_name}'...",
+ )
+
+ source_dir = cloud_dir / platform
+
+ for filename in files:
+ filepath = (output / filename).resolve()
+ try:
+ filepath.relative_to(output)
+ except ValueError:
+ print(
+ settings.ERROR_ALERT,
+ f" Skipped (unsafe filename): {filename}",
+ )
+ continue
+
+ if filepath.exists():
+ print(
+ settings.WARNING_ALERT,
+ f" Skipped (already exists): {filepath}",
+ )
+ continue
+
+ source = source_dir / filename
+ if not source.exists():
+ print(
+ settings.ERROR_ALERT,
+ f" Skipped (template not found): {filename}",
+ )
+ continue
+
+ content = source.read_text()
+ content = content.replace("{{PROJECT_NAME}}", project_name)
+ content = content.replace("{{MODULE_NAME}}", module_name)
+
+ filepath.write_text(content)
+ print(f" Created: {filepath}")
+
+ print(settings.INFO_ALERT, "Done.")
+
+
+class CloudListCommand(Command):
+ def setup(self):
+ cloud_dir = _get_template_dir()
+ if cloud_dir is None:
+ return
+
+ registry = _load_registry(cloud_dir)
+ if registry is None:
+ return
+
+ platforms = _get_platforms(registry)
+ if platforms is None:
+ return
+
+ table = Table(title="Available Platforms")
+ table.add_column("Platform", style="bold cyan")
+ table.add_column("Name", style="bold")
+ table.add_column("Description")
+
+ for key, info in platforms.items():
+ table.add_row(
+ key,
+ info.get("name", ""),
+ info.get("description", ""),
+ )
+
+ Console().print(table)
diff --git a/dotflow/cli/commands/deploy.py b/dotflow/cli/commands/deploy.py
new file mode 100644
index 00000000..1da7f241
--- /dev/null
+++ b/dotflow/cli/commands/deploy.py
@@ -0,0 +1,110 @@
+"""Command deploy module."""
+
+from rich import print # type: ignore
+
+from dotflow.cli.command import Command
+from dotflow.settings import Settings as settings
+
+SAM_PLATFORMS = {
+ "ecs-scheduled",
+}
+
+DEFAULT_REGIONS = {
+ "lambda": "us-east-1",
+ "lambda-scheduled": "us-east-1",
+ "lambda-s3-trigger": "us-east-1",
+ "lambda-sqs-trigger": "us-east-1",
+ "lambda-api-trigger": "us-east-1",
+ "ecs": "us-east-1",
+ "cloud-run": "us-central1",
+ "cloud-run-scheduled": "us-central1",
+}
+
+
+class DeployCommand(Command):
+ def setup(self):
+ platform = self.params.platform
+ name = self.params.project
+ region = self.params.region or DEFAULT_REGIONS.get(
+ platform, "us-east-1"
+ )
+ schedule = getattr(self.params, "schedule", None)
+
+ if platform in SAM_PLATFORMS:
+ print(
+ settings.INFO_ALERT,
+ f"'{platform}' requires SAM for full trigger setup.",
+ )
+ print(" Run: sam build && sam deploy")
+ return
+
+ deployers = {
+ "lambda": self._deploy_lambda,
+ "lambda-scheduled": self._deploy_lambda,
+ "lambda-s3-trigger": self._deploy_lambda_s3,
+ "lambda-sqs-trigger": self._deploy_lambda_sqs,
+ "lambda-api-trigger": self._deploy_lambda_api,
+ "ecs": self._deploy_ecs,
+ "cloud-run": self._deploy_cloud_run,
+ "cloud-run-scheduled": self._deploy_cloud_run,
+ "github-actions": self._deploy_github_actions,
+ }
+
+ handler = deployers.get(platform)
+ if not handler:
+ print(
+ settings.ERROR_ALERT,
+ f"Deploy not supported for '{platform}'.",
+ )
+ return
+
+ handler(name=name, region=region, schedule=schedule)
+
+ def _deploy_lambda(self, name: str, region: str, schedule: str = None):
+ """Deploy to AWS Lambda."""
+ from dotflow.cloud.aws import LambdaDeployer
+
+ deployer = LambdaDeployer(region=region)
+ deployer.deploy(name, schedule=schedule)
+
+ def _deploy_lambda_api(self, name: str, region: str, **kwargs):
+ """Deploy to AWS Lambda + API Gateway."""
+ from dotflow.cloud.aws import LambdaApiDeployer
+
+ deployer = LambdaApiDeployer(region=region)
+ deployer.deploy(name)
+
+ def _deploy_lambda_s3(self, name: str, region: str, **kwargs):
+ """Deploy to AWS Lambda + S3 Trigger."""
+ from dotflow.cloud.aws import LambdaS3Deployer
+
+ deployer = LambdaS3Deployer(region=region)
+ deployer.deploy(name)
+
+ def _deploy_lambda_sqs(self, name: str, region: str, **kwargs):
+ """Deploy to AWS Lambda + SQS Trigger."""
+ from dotflow.cloud.aws import LambdaSQSDeployer
+
+ deployer = LambdaSQSDeployer(region=region)
+ deployer.deploy(name)
+
+ def _deploy_ecs(self, name: str, region: str, **kwargs):
+ """Deploy to AWS ECS Fargate."""
+ from dotflow.cloud.aws import ECSDeployer
+
+ deployer = ECSDeployer(region=region)
+ deployer.deploy(name)
+
+ def _deploy_cloud_run(self, name: str, region: str, **kwargs):
+ """Deploy to Google Cloud Run."""
+ from dotflow.cloud.gcp import CloudRunDeployer
+
+ deployer = CloudRunDeployer(region=region)
+ deployer.deploy(name)
+
+ def _deploy_github_actions(self, name: str, **kwargs):
+ """Deploy to GitHub Actions."""
+ from dotflow.cloud.github import ActionsDeployer
+
+ deployer = ActionsDeployer()
+ deployer.deploy(name)
diff --git a/dotflow/cli/setup.py b/dotflow/cli/setup.py
index 67f45f5b..3d37be94 100644
--- a/dotflow/cli/setup.py
+++ b/dotflow/cli/setup.py
@@ -4,6 +4,9 @@
from dotflow import __description__, __version__
from dotflow.cli.commands import (
+ CloudGenerateCommand,
+ CloudListCommand,
+ DeployCommand,
InitCommand,
LogCommand,
ScheduleCommand,
@@ -39,6 +42,8 @@ def __init__(self, parser):
self.setup_logs()
self.setup_start()
self.setup_schedule()
+ self.setup_cloud()
+ self.setup_deploy()
self.command()
def setup_init(self):
@@ -145,6 +150,66 @@ def setup_schedule(self):
self.cmd_schedule.set_defaults(exec=ScheduleCommand)
+ def setup_cloud(self):
+ self.cmd_cloud = self.subparsers.add_parser(
+ "cloud",
+ help="Generate cloud infrastructure files for a target platform",
+ )
+ cloud_subparsers = self.cmd_cloud.add_subparsers()
+
+ cmd_generate = cloud_subparsers.add_parser(
+ "generate", help="Generate infrastructure files"
+ )
+ cmd_generate.add_argument(
+ "--platform",
+ required=True,
+ help="Target platform (e.g. docker, lambda, cloud-run, ecs, kubernetes)",
+ )
+ cmd_generate.add_argument(
+ "--project",
+ default=None,
+ help="Project name (defaults to current directory name)",
+ )
+ cmd_generate.add_argument(
+ "--output",
+ default=".",
+ help="Output directory (defaults to current directory)",
+ )
+ cmd_generate.set_defaults(exec=CloudGenerateCommand)
+
+ cmd_list = cloud_subparsers.add_parser(
+ "list", help="List available cloud platforms"
+ )
+ cmd_list.set_defaults(exec=CloudListCommand)
+
+ def setup_deploy(self):
+ self.cmd_deploy = self.subparsers.add_parser(
+ "deploy",
+ help="Deploy pipeline to a cloud platform",
+ )
+ self.cmd_deploy.add_argument(
+ "--platform",
+ required=True,
+ help="Target platform (e.g. lambda, ecs)",
+ )
+ self.cmd_deploy.add_argument(
+ "--project",
+ default=None,
+ required=True,
+ help="Project name",
+ )
+ self.cmd_deploy.add_argument(
+ "--region",
+ default=None,
+ help="Cloud region (e.g. us-east-1 for AWS, us-central1 for GCP)",
+ )
+ self.cmd_deploy.add_argument(
+ "--schedule",
+ default=None,
+ help="Schedule expression (e.g. 'rate(6 hours)')",
+ )
+ self.cmd_deploy.set_defaults(exec=DeployCommand)
+
def setup_logs(self):
self.cmd_logs = self.subparsers.add_parser("logs", help="Logs")
self.cmd_logs = self.cmd_logs.add_argument_group(
diff --git a/dotflow/cloud/__init__.py b/dotflow/cloud/__init__.py
new file mode 100644
index 00000000..4ba0a8c6
--- /dev/null
+++ b/dotflow/cloud/__init__.py
@@ -0,0 +1 @@
+"""Deploy module."""
diff --git a/dotflow/cloud/aws/__init__.py b/dotflow/cloud/aws/__init__.py
new file mode 100644
index 00000000..9d5d2de3
--- /dev/null
+++ b/dotflow/cloud/aws/__init__.py
@@ -0,0 +1,15 @@
+"""AWS deploy module."""
+
+from dotflow.cloud.aws.ecs_deployer import ECSDeployer
+from dotflow.cloud.aws.lambda_api_deployer import LambdaApiDeployer
+from dotflow.cloud.aws.lambda_deployer import LambdaDeployer
+from dotflow.cloud.aws.lambda_s3_deployer import LambdaS3Deployer
+from dotflow.cloud.aws.lambda_sqs_deployer import LambdaSQSDeployer
+
+__all__ = [
+ "LambdaDeployer",
+ "LambdaApiDeployer",
+ "LambdaS3Deployer",
+ "LambdaSQSDeployer",
+ "ECSDeployer",
+]
diff --git a/dotflow/cloud/aws/base_lambda_deployer.py b/dotflow/cloud/aws/base_lambda_deployer.py
new file mode 100644
index 00000000..25402c50
--- /dev/null
+++ b/dotflow/cloud/aws/base_lambda_deployer.py
@@ -0,0 +1,96 @@
+"""Base Lambda deployment — shared logic for all Lambda variants."""
+
+from __future__ import annotations
+
+from rich import print # type: ignore
+
+from dotflow.cloud.aws.constants import BOTO3_NOT_FOUND, CREDENTIALS_NOT_FOUND
+from dotflow.cloud.aws.services.cloudwatch import CloudWatch
+from dotflow.cloud.aws.services.ecr import ECR
+from dotflow.cloud.aws.services.iam import IAM
+from dotflow.cloud.core import Deployer
+from dotflow.settings import Settings as settings
+
+
+class BaseLambdaDeployer(Deployer):
+ """Shared logic for all Lambda deployers."""
+
+ def __init__(self, region: str = "us-east-1"):
+ try:
+ import boto3
+ except ImportError as err:
+ raise SystemExit(BOTO3_NOT_FOUND) from err
+
+ try:
+ sts = boto3.client("sts", region_name=region)
+ self._account_id = sts.get_caller_identity()["Account"]
+ except Exception as err:
+ raise SystemExit(CREDENTIALS_NOT_FOUND) from err
+
+ self._region = region
+ self._boto3 = boto3
+ self._lambda = boto3.client("lambda", region_name=region)
+
+ self._ecr = ECR(
+ boto3.client("ecr", region_name=region),
+ self._account_id,
+ region,
+ )
+ self._iam = IAM(boto3.client("iam", region_name=region))
+ self._logs = CloudWatch(boto3.client("logs", region_name=region))
+
+ @property
+ def _function_arn(self) -> str:
+ return (
+ f"arn:aws:lambda:{self._region}:{self._account_id}"
+ f":function:{self._name}"
+ )
+
+ def setup(self, name: str) -> None:
+ """Create IAM role and CloudWatch log group."""
+ self._name = name
+ self._role_arn = self._iam.ensure_lambda_role(name)
+ self._logs.ensure_log_group(f"/aws/lambda/{name}")
+
+ def ensure_roles(self, name: str) -> str:
+ """Create or get Lambda execution role."""
+ return self._iam.ensure_lambda_role(name)
+
+ def ensure_logs(self, name: str) -> None:
+ """Create CloudWatch log group."""
+ self._logs.ensure_log_group(f"/aws/lambda/{name}")
+
+ def deploy(self, name: str, **kwargs) -> None:
+ """Deploy Lambda and configure trigger."""
+ print(settings.INFO_ALERT, f"Deploying '{name}'...")
+
+ self.setup(name)
+ image_uri = self._ecr.push(name)
+ self._create_or_update_lambda(name, image_uri, self._role_arn)
+ self._configure_trigger(name, **kwargs)
+
+ print(settings.INFO_ALERT, "Done.")
+
+ def _create_or_update_lambda(
+ self, name: str, image_uri: str, role_arn: str
+ ):
+ """Create or update Lambda function."""
+ try:
+ self._lambda.get_function(FunctionName=name)
+ print(" Updating Lambda function...")
+ self._lambda.update_function_code(
+ FunctionName=name, ImageUri=image_uri
+ )
+ except self._lambda.exceptions.ResourceNotFoundException:
+ print(" Creating Lambda function...")
+ self._lambda.create_function(
+ FunctionName=name,
+ PackageType="Image",
+ Code={"ImageUri": image_uri},
+ Role=role_arn,
+ Timeout=900,
+ MemorySize=512,
+ )
+
+ def _configure_trigger(self, name: str, **kwargs) -> None:
+ """Override in subclasses to configure the specific trigger."""
diff --git a/dotflow/cloud/aws/constants.py b/dotflow/cloud/aws/constants.py
new file mode 100644
index 00000000..bb1f02bf
--- /dev/null
+++ b/dotflow/cloud/aws/constants.py
@@ -0,0 +1,9 @@
+"""AWS messages."""
+
+CREDENTIALS_NOT_FOUND = (
+ "AWS credentials not found. "
+ "Run 'aws configure' or set AWS_ACCESS_KEY_ID "
+ "and AWS_SECRET_ACCESS_KEY environment variables."
+)
+
+BOTO3_NOT_FOUND = "boto3 is required: pip install dotflow[deploy-aws]"
diff --git a/dotflow/cloud/aws/ecs_deployer.py b/dotflow/cloud/aws/ecs_deployer.py
new file mode 100644
index 00000000..e105ed97
--- /dev/null
+++ b/dotflow/cloud/aws/ecs_deployer.py
@@ -0,0 +1,108 @@
+"""ECS deployment."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from rich import print # type: ignore
+
+from dotflow.cloud.aws.constants import BOTO3_NOT_FOUND, CREDENTIALS_NOT_FOUND
+from dotflow.cloud.aws.services.cloudwatch import CloudWatch
+from dotflow.cloud.aws.services.ecr import ECR
+from dotflow.cloud.aws.services.iam import IAM
+from dotflow.cloud.core import Deployer
+from dotflow.settings import Settings as settings
+
+
+class ECSDeployer(Deployer):
+ """Deploy dotflow pipelines to AWS ECS Fargate."""
+
+ def __init__(self, region: str = "us-east-1"):
+ try:
+ import boto3
+ except ImportError as err:
+ raise SystemExit(BOTO3_NOT_FOUND) from err
+
+ try:
+ sts = boto3.client("sts", region_name=region)
+ self._account_id = sts.get_caller_identity()["Account"]
+ except Exception as err:
+ raise SystemExit(CREDENTIALS_NOT_FOUND) from err
+
+ self._region = region
+ self._ecs = boto3.client("ecs", region_name=region)
+
+ self._ecr = ECR(
+ boto3.client("ecr", region_name=region),
+ self._account_id,
+ region,
+ )
+ self._iam = IAM(boto3.client("iam", region_name=region))
+ self._logs = CloudWatch(boto3.client("logs", region_name=region))
+
+ def setup(self, name: str) -> None:
+ """Create IAM role, CloudWatch log group, and ECS cluster."""
+ self._execution_role_arn = self._iam.ensure_ecs_execution_role()
+ self._logs.ensure_log_group(f"/ecs/{name}")
+ self._ensure_cluster(name)
+
+ def deploy(self, name: str, **kwargs) -> None:
+ """Deploy an ECS Fargate task from the current directory."""
+ print(settings.INFO_ALERT, f"Deploying ECS task '{name}'...")
+
+ self.setup(name)
+ self._ecr.push(name)
+
+ if (Path.cwd() / "task-definition.json").exists():
+ self._register_from_file()
+ else:
+ self._create_task_definition(name, self._execution_role_arn)
+
+ print(settings.INFO_ALERT, "Done.")
+
+ def _register_from_file(self):
+ """Register task definition from task-definition.json."""
+ content = json.loads((Path.cwd() / "task-definition.json").read_text())
+ print(" Registering task definition...")
+ self._ecs.register_task_definition(**content)
+
+ def _create_task_definition(self, name: str, execution_role_arn: str):
+ """Create a basic Fargate task definition."""
+ image_uri = (
+ f"{self._account_id}.dkr.ecr.{self._region}"
+ f".amazonaws.com/{name}:latest"
+ )
+ print(" Creating task definition...")
+ self._ecs.register_task_definition(
+ family=name,
+ networkMode="awsvpc",
+ requiresCompatibilities=["FARGATE"],
+ cpu="256",
+ memory="512",
+ executionRoleArn=execution_role_arn,
+ containerDefinitions=[
+ {
+ "name": name,
+ "image": image_uri,
+ "essential": True,
+ "logConfiguration": {
+ "logDriver": "awslogs",
+ "options": {
+ "awslogs-group": f"/ecs/{name}",
+ "awslogs-region": self._region,
+ "awslogs-stream-prefix": "ecs",
+ },
+ },
+ }
+ ],
+ )
+
+ def _ensure_cluster(self, name: str):
+ """Create ECS cluster if it doesn't exist."""
+ cluster_name = f"{name}-cluster"
+ clusters = self._ecs.describe_clusters(clusters=[cluster_name])
+ active = [c for c in clusters["clusters"] if c["status"] == "ACTIVE"]
+ if not active:
+ print(f" Creating ECS cluster '{cluster_name}'...")
+ self._ecs.create_cluster(clusterName=cluster_name)
diff --git a/dotflow/cloud/aws/lambda_api_deployer.py b/dotflow/cloud/aws/lambda_api_deployer.py
new file mode 100644
index 00000000..ce5387e9
--- /dev/null
+++ b/dotflow/cloud/aws/lambda_api_deployer.py
@@ -0,0 +1,67 @@
+"""Lambda + API Gateway deployment."""
+
+from __future__ import annotations
+
+import contextlib
+
+from dotflow.cloud.aws.base_lambda_deployer import BaseLambdaDeployer
+
+
+class LambdaApiDeployer(BaseLambdaDeployer):
+ """Deploy dotflow pipeline as Lambda + API Gateway."""
+
+ def _configure_trigger(self, name: str, **kwargs) -> None:
+ """Create HTTP API with Lambda integration."""
+ apigw = self._boto3.client("apigatewayv2", region_name=self._region)
+ api_name = f"{name}-api"
+
+ api_id = self._find_api(apigw, api_name)
+
+ if not api_id:
+ print(" Creating API Gateway...")
+ response = apigw.create_api(Name=api_name, ProtocolType="HTTP")
+ api_id = response["ApiId"]
+
+ integration = apigw.create_integration(
+ ApiId=api_id,
+ IntegrationType="AWS_PROXY",
+ IntegrationUri=self._function_arn,
+ PayloadFormatVersion="2.0",
+ )
+
+ apigw.create_route(
+ ApiId=api_id,
+ RouteKey="POST /workflow",
+ Target=f"integrations/{integration['IntegrationId']}",
+ )
+
+ apigw.create_stage(
+ ApiId=api_id, StageName="$default", AutoDeploy=True
+ )
+
+ with contextlib.suppress(
+ self._lambda.exceptions.ResourceConflictException
+ ):
+ self._lambda.add_permission(
+ FunctionName=name,
+ StatementId=f"{api_name}-invoke",
+ Action="lambda:InvokeFunction",
+ Principal="apigateway.amazonaws.com",
+ SourceArn=(
+ f"arn:aws:execute-api:{self._region}:"
+ f"{self._account_id}:{api_id}/*"
+ ),
+ )
+
+ url = (
+ f"https://{api_id}.execute-api."
+ f"{self._region}.amazonaws.com/workflow"
+ )
+ print(f" Endpoint: {url}")
+
+ def _find_api(self, apigw, name: str) -> str | None:
+ """Find existing API by name."""
+ for api in apigw.get_apis().get("Items", []):
+ if api["Name"] == name:
+ return api["ApiId"]
+ return None
diff --git a/dotflow/cloud/aws/lambda_deployer.py b/dotflow/cloud/aws/lambda_deployer.py
new file mode 100644
index 00000000..e0a2c1db
--- /dev/null
+++ b/dotflow/cloud/aws/lambda_deployer.py
@@ -0,0 +1,49 @@
+"""Lambda deployment."""
+
+from __future__ import annotations
+
+import contextlib
+
+from dotflow.cloud.aws.base_lambda_deployer import BaseLambdaDeployer
+
+
+class LambdaDeployer(BaseLambdaDeployer):
+ """Deploy dotflow pipelines to AWS Lambda."""
+
+ def _configure_trigger(self, name: str, **kwargs) -> None:
+ """Create EventBridge schedule if provided."""
+ schedule = kwargs.get("schedule")
+ if not schedule:
+ return
+
+ rule_name = f"{name}-schedule"
+
+ print(f" Creating EventBridge rule '{rule_name}'...")
+ events = self._boto3.client("events", region_name=self._region)
+
+ events.put_rule(
+ Name=rule_name,
+ ScheduleExpression=schedule,
+ State="ENABLED",
+ )
+
+ with contextlib.suppress(
+ self._lambda.exceptions.ResourceConflictException
+ ):
+ self._lambda.add_permission(
+ FunctionName=name,
+ StatementId=rule_name,
+ Action="lambda:InvokeFunction",
+ Principal="events.amazonaws.com",
+ SourceArn=(
+ f"arn:aws:events:{self._region}:{self._account_id}"
+ f":rule/{rule_name}"
+ ),
+ )
+
+ events.put_targets(
+ Rule=rule_name,
+ Targets=[{"Id": "1", "Arn": self._function_arn}],
+ )
+
+ print(f" Schedule: {schedule}")
diff --git a/dotflow/cloud/aws/lambda_s3_deployer.py b/dotflow/cloud/aws/lambda_s3_deployer.py
new file mode 100644
index 00000000..6595bc11
--- /dev/null
+++ b/dotflow/cloud/aws/lambda_s3_deployer.py
@@ -0,0 +1,56 @@
+"""Lambda + S3 Trigger deployment."""
+
+from __future__ import annotations
+
+import contextlib
+
+from dotflow.cloud.aws.base_lambda_deployer import BaseLambdaDeployer
+
+
+class LambdaS3Deployer(BaseLambdaDeployer):
+ """Deploy dotflow pipeline as Lambda + S3 Trigger."""
+
+ def _configure_trigger(self, name: str, **kwargs) -> None:
+ """Create S3 bucket with Lambda notification."""
+ s3 = self._boto3.client("s3", region_name=self._region)
+ bucket_name = f"{name}-source"
+
+ print(f" Creating S3 bucket '{bucket_name}'...")
+ try:
+ if self._region == "us-east-1":
+ s3.create_bucket(Bucket=bucket_name)
+ else:
+ s3.create_bucket(
+ Bucket=bucket_name,
+ CreateBucketConfiguration={
+ "LocationConstraint": self._region
+ },
+ )
+ except s3.exceptions.BucketAlreadyOwnedByYou:
+ pass
+
+ with contextlib.suppress(
+ self._lambda.exceptions.ResourceConflictException
+ ):
+ self._lambda.add_permission(
+ FunctionName=name,
+ StatementId=f"{name}-s3-invoke",
+ Action="lambda:InvokeFunction",
+ Principal="s3.amazonaws.com",
+ SourceArn=f"arn:aws:s3:::{bucket_name}",
+ )
+
+ print(" Configuring S3 notification...")
+ s3.put_bucket_notification_configuration(
+ Bucket=bucket_name,
+ NotificationConfiguration={
+ "LambdaFunctionConfigurations": [
+ {
+ "LambdaFunctionArn": self._function_arn,
+ "Events": ["s3:ObjectCreated:*"],
+ }
+ ]
+ },
+ )
+
+ print(f" Trigger: s3://{bucket_name}/*")
diff --git a/dotflow/cloud/aws/lambda_sqs_deployer.py b/dotflow/cloud/aws/lambda_sqs_deployer.py
new file mode 100644
index 00000000..fd848f55
--- /dev/null
+++ b/dotflow/cloud/aws/lambda_sqs_deployer.py
@@ -0,0 +1,43 @@
+"""Lambda + SQS Trigger deployment."""
+
+from __future__ import annotations
+
+from dotflow.cloud.aws.base_lambda_deployer import BaseLambdaDeployer
+
+
+class LambdaSQSDeployer(BaseLambdaDeployer):
+ """Deploy dotflow pipeline as Lambda + SQS Trigger."""
+
+ def _configure_trigger(self, name: str, **kwargs) -> None:
+ """Create SQS queue and event source mapping."""
+ sqs = self._boto3.client("sqs", region_name=self._region)
+ queue_name = f"{name}-queue"
+
+ print(f" Creating SQS queue '{queue_name}'...")
+ try:
+ response = sqs.create_queue(
+ QueueName=queue_name,
+ Attributes={"VisibilityTimeout": "960"},
+ )
+ queue_url = response["QueueUrl"]
+ except sqs.exceptions.QueueNameAlreadyExists:
+ queue_url = sqs.get_queue_url(QueueName=queue_name)["QueueUrl"]
+
+ queue_arn = sqs.get_queue_attributes(
+ QueueUrl=queue_url, AttributeNames=["QueueArn"]
+ )["Attributes"]["QueueArn"]
+
+ existing = self._lambda.list_event_source_mappings(
+ FunctionName=name, EventSourceArn=queue_arn
+ )["EventSourceMappings"]
+
+ if not existing:
+ print(" Creating event source mapping...")
+ self._lambda.create_event_source_mapping(
+ FunctionName=name,
+ EventSourceArn=queue_arn,
+ BatchSize=1,
+ Enabled=True,
+ )
+
+ print(f" Queue: {queue_url}")
diff --git a/dotflow/cloud/aws/services/__init__.py b/dotflow/cloud/aws/services/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/dotflow/cloud/aws/services/cloudwatch.py b/dotflow/cloud/aws/services/cloudwatch.py
new file mode 100644
index 00000000..d9045556
--- /dev/null
+++ b/dotflow/cloud/aws/services/cloudwatch.py
@@ -0,0 +1,22 @@
+"""CloudWatch Logs operations."""
+
+from __future__ import annotations
+
+import contextlib
+
+from dotflow.cloud.core import LogManager
+
+
+class CloudWatch(LogManager):
+ """AWS CloudWatch log manager."""
+
+ def __init__(self, logs_client):
+ self._logs = logs_client
+
+ def ensure_log_group(self, name: str):
+ """Create log group if it doesn't exist."""
+ with contextlib.suppress(
+ self._logs.exceptions.ResourceAlreadyExistsException
+ ):
+ self._logs.create_log_group(logGroupName=name)
+ print(f" Created log group '{name}'")
diff --git a/dotflow/cloud/aws/services/ecr.py b/dotflow/cloud/aws/services/ecr.py
new file mode 100644
index 00000000..fb721d90
--- /dev/null
+++ b/dotflow/cloud/aws/services/ecr.py
@@ -0,0 +1,72 @@
+"""ECR operations — build, push and authenticate Docker images."""
+
+from __future__ import annotations
+
+import base64
+import contextlib
+from subprocess import run
+
+from dotflow.cloud.core import Registry
+
+
+class ECR(Registry):
+ """Amazon ECR container registry."""
+
+ def __init__(self, ecr_client, account_id: str, region: str):
+ self._ecr = ecr_client
+ self._account_id = account_id
+ self._region = region
+
+ def push(self, name: str) -> str:
+ """Build, tag and push a Docker image. Returns the image URI."""
+ self._ensure_repository(name)
+ self.login()
+
+ image_uri = (
+ f"{self._account_id}.dkr.ecr.{self._region}"
+ f".amazonaws.com/{name}:latest"
+ )
+
+ print(" Building image...")
+ run(["docker", "build", "-t", name, "."], check=True)
+
+ print(" Tagging image...")
+ run(["docker", "tag", f"{name}:latest", image_uri], check=True)
+
+ print(" Pushing image...")
+ run(["docker", "push", image_uri], check=True)
+
+ return image_uri
+
+ def _ensure_repository(self, name: str):
+ """Create ECR repository if it doesn't exist."""
+ print(" Creating ECR repository...")
+ with contextlib.suppress(
+ self._ecr.exceptions.RepositoryAlreadyExistsException
+ ):
+ self._ecr.create_repository(repositoryName=name)
+
+ def login(self):
+ """Authenticate Docker with ECR."""
+ print(" Logging in to ECR...")
+ token = self._ecr.get_authorization_token()
+ auth = token["authorizationData"][0]
+ registry = auth["proxyEndpoint"]
+ username, password = (
+ base64.b64decode(auth["authorizationToken"]).decode().split(":")
+ )
+
+ run(
+ [
+ "docker",
+ "login",
+ "--username",
+ username,
+ "--password-stdin",
+ registry,
+ ],
+ input=password,
+ text=True,
+ capture_output=True,
+ check=True,
+ )
diff --git a/dotflow/cloud/aws/services/iam.py b/dotflow/cloud/aws/services/iam.py
new file mode 100644
index 00000000..46307b5a
--- /dev/null
+++ b/dotflow/cloud/aws/services/iam.py
@@ -0,0 +1,79 @@
+"""IAM operations — create and manage execution roles."""
+
+from __future__ import annotations
+
+import json
+import time
+
+from dotflow.cloud.core import RoleManager
+
+
+class IAM(RoleManager):
+ """AWS IAM role manager."""
+
+ def __init__(self, iam_client):
+ self._iam = iam_client
+
+ def ensure_lambda_role(self, name: str) -> str:
+ """Create or get Lambda execution role."""
+ return self.ensure_role(
+ name=f"{name}-lambda-role",
+ service="lambda.amazonaws.com",
+ policy=(
+ "arn:aws:iam::aws:policy/service-role/"
+ "AWSLambdaBasicExecutionRole"
+ ),
+ wait=True,
+ )
+
+ def ensure_ecs_execution_role(self) -> str:
+ """Create or get ECS task execution role."""
+ return self.ensure_role(
+ name="ecsTaskExecutionRole",
+ service="ecs-tasks.amazonaws.com",
+ policy=(
+ "arn:aws:iam::aws:policy/service-role/"
+ "AmazonECSTaskExecutionRolePolicy"
+ ),
+ )
+
+ def ensure_role(
+ self,
+ name: str,
+ service: str,
+ policy: str,
+ wait: bool = False,
+ ) -> str:
+ """Create role if it doesn't exist, attach policy, return ARN."""
+ role_name = name
+ try:
+ response = self._iam.get_role(RoleName=role_name)
+ return response["Role"]["Arn"]
+ except self._iam.exceptions.NoSuchEntityException:
+ pass
+
+ print(f" Creating IAM role '{role_name}'...")
+ response = self._iam.create_role(
+ RoleName=role_name,
+ AssumeRolePolicyDocument=json.dumps(
+ {
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Principal": {"Service": service},
+ "Action": "sts:AssumeRole",
+ }
+ ],
+ }
+ ),
+ )
+ self._iam.attach_role_policy(
+ RoleName=role_name,
+ PolicyArn=policy,
+ )
+
+ if wait:
+ time.sleep(10)
+
+ return response["Role"]["Arn"]
diff --git a/dotflow/cloud/aws/services/s3.py b/dotflow/cloud/aws/services/s3.py
new file mode 100644
index 00000000..afe6540f
--- /dev/null
+++ b/dotflow/cloud/aws/services/s3.py
@@ -0,0 +1,47 @@
+"""S3 operations — read and write objects."""
+
+from __future__ import annotations
+
+from json import dumps, loads
+
+from dotflow.cloud.core import ObjectStorage
+from dotflow.core.exception import ModuleNotFound
+
+
+class S3(ObjectStorage):
+ """Amazon S3 object storage."""
+
+ def __init__(self, bucket: str, prefix: str = "", region: str = None):
+ try:
+ import boto3
+ except ImportError:
+ raise ModuleNotFound(
+ module="boto3", library="dotflow[aws]"
+ ) from None
+
+ self._s3 = boto3.client("s3", region_name=region)
+ self.bucket = bucket
+ self.prefix = prefix
+
+ self._s3.head_bucket(Bucket=self.bucket)
+
+ def read(self, key: str) -> list:
+ """Read a JSON list from S3. Returns empty list if key not found."""
+ try:
+ response = self._s3.get_object(
+ Bucket=self.bucket,
+ Key=f"{self.prefix}{key}",
+ )
+ data = response["Body"].read().decode("utf-8")
+ return loads(data)
+ except self._s3.exceptions.NoSuchKey:
+ return []
+
+ def write(self, key: str, data: list) -> None:
+ """Write a JSON list to S3."""
+ self._s3.put_object(
+ Bucket=self.bucket,
+ Key=f"{self.prefix}{key}",
+ Body=dumps(data),
+ ContentType="application/json",
+ )
diff --git a/dotflow/cloud/core.py b/dotflow/cloud/core.py
new file mode 100644
index 00000000..f6c84a8e
--- /dev/null
+++ b/dotflow/cloud/core.py
@@ -0,0 +1,89 @@
+"""Deploy ABCs."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+
+
+class Registry(ABC):
+ """Base class for container registries (ECR, Artifact Registry, Docker Hub, etc.)."""
+
+ @abstractmethod
+ def push(self, name: str) -> str:
+ """Build and push the Docker image. Returns the image URI."""
+
+ @abstractmethod
+ def login(self) -> None:
+ """Authenticate with the registry."""
+
+
+class RoleManager(ABC):
+ """Base class for IAM/service account management."""
+
+ @abstractmethod
+ def ensure_role(self, name: str, service: str, policy: str) -> str:
+ """Create or get a role for the given service. Returns the role ARN/ID."""
+
+
+class LogManager(ABC):
+ """Base class for log group/resource management."""
+
+ @abstractmethod
+ def ensure_log_group(self, name: str) -> None:
+ """Create log group if it doesn't exist."""
+
+
+class ObjectStorage(ABC):
+ """Base class for object storage (S3, GCS, Azure Blob, etc.)."""
+
+ @abstractmethod
+ def read(self, key: str) -> list:
+ """Read data from a key. Returns empty list if not found."""
+
+ @abstractmethod
+ def write(self, key: str, data: list) -> None:
+ """Write data to a key."""
+
+
+class Queue(ABC):
+ """Base class for message queues (SQS, Pub/Sub, Azure Queue, etc.)."""
+
+ @abstractmethod
+ def create(self, name: str) -> str:
+ """Create queue. Returns queue URL/ID."""
+
+ @abstractmethod
+ def get_arn(self, name: str) -> str:
+ """Get queue ARN/ID."""
+
+
+class EventScheduler(ABC):
+ """Base class for event schedulers (EventBridge, Cloud Scheduler, etc.)."""
+
+ @abstractmethod
+ def create_schedule(
+ self, name: str, expression: str, target_arn: str
+ ) -> None:
+ """Create a scheduled rule targeting a resource."""
+
+
+class Deployer(ABC):
+ """Base class for all cloud deployers."""
+
+ @abstractmethod
+ def deploy(self, name: str, **kwargs) -> None:
+ """Deploy the pipeline to the target platform."""
+
+ def setup(self, name: str) -> None:
+ """Create required cloud resources (roles, log groups, etc.).
+
+ Optional — override when the platform needs pre-deploy setup.
+ No-op by default.
+ """
+
+ def teardown(self, name: str) -> None:
+ """Remove cloud resources created by deploy.
+
+ Optional — override when the platform supports cleanup.
+ No-op by default.
+ """
diff --git a/dotflow/cloud/gcp/__init__.py b/dotflow/cloud/gcp/__init__.py
new file mode 100644
index 00000000..da4b1276
--- /dev/null
+++ b/dotflow/cloud/gcp/__init__.py
@@ -0,0 +1,5 @@
+"""GCP cloud module."""
+
+from dotflow.cloud.gcp.cloudrun_deployer import CloudRunDeployer
+
+__all__ = ["CloudRunDeployer"]
diff --git a/dotflow/cloud/gcp/cloudrun_deployer.py b/dotflow/cloud/gcp/cloudrun_deployer.py
new file mode 100644
index 00000000..93121001
--- /dev/null
+++ b/dotflow/cloud/gcp/cloudrun_deployer.py
@@ -0,0 +1,111 @@
+"""Cloud Run deployment."""
+
+from __future__ import annotations
+
+from rich import print # type: ignore
+
+from dotflow.cloud.core import Deployer
+from dotflow.cloud.gcp.constants import (
+ CREDENTIALS_NOT_FOUND,
+ GCP_SDK_NOT_FOUND,
+)
+from dotflow.cloud.gcp.services.apis import APIs
+from dotflow.cloud.gcp.services.artifact_registry import ArtifactRegistry
+from dotflow.settings import Settings as settings
+
+
+class CloudRunDeployer(Deployer):
+ """Deploy dotflow pipelines to Google Cloud Run."""
+
+ def __init__(self, region: str = "us-central1", project: str = None):
+ try:
+ from google.api_core import exceptions
+ from google.cloud import run_v2
+ except ImportError as err:
+ raise SystemExit(GCP_SDK_NOT_FOUND) from err
+
+ self._run_v2 = run_v2
+ self._exceptions = exceptions
+ self._region = region
+
+ if project:
+ self._project = project
+ else:
+ try:
+ import google.auth
+
+ _, self._project = google.auth.default()
+ except Exception as err:
+ raise SystemExit(CREDENTIALS_NOT_FOUND) from err
+
+ if not self._project:
+ raise SystemExit(CREDENTIALS_NOT_FOUND)
+
+ self._registry = ArtifactRegistry(
+ project=self._project, region=self._region
+ )
+
+ def setup(self, name: str) -> None:
+ """Enable required GCP APIs and ensure Cloud Build service account."""
+ APIs(self._project).enable()
+ self._ensure_cloudbuild_sa()
+
+ def _ensure_cloudbuild_sa(self):
+ """Ensure Cloud Build service identity exists."""
+ try:
+ from google.cloud import service_usage_v1
+
+ print(" Ensuring Cloud Build service account...")
+ client = service_usage_v1.ServiceUsageClient()
+ client.generate_service_identity(
+ request={
+ "parent": f"projects/{self._project}/services/cloudbuild.googleapis.com"
+ }
+ )
+ except Exception:
+ pass
+
+ def ensure_roles(self, name: str) -> str:
+ """No-op — Cloud Run manages service accounts automatically."""
+ return ""
+
+ def ensure_logs(self, name: str) -> None:
+ """No-op — Cloud Logging is automatic."""
+
+ def deploy(self, name: str, **kwargs) -> None:
+ """Deploy to Cloud Run via Cloud Build + Run SDK."""
+ print(settings.INFO_ALERT, f"Deploying Cloud Run '{name}'...")
+
+ self.setup(name)
+ image = self._registry.push(name)
+ self._deploy_service(name, image)
+
+ print(settings.INFO_ALERT, "Done.")
+
+ def _deploy_service(self, name: str, image: str):
+ """Create or update Cloud Run service."""
+ print(" Deploying service...")
+
+ client = self._run_v2.ServicesClient()
+ parent = f"projects/{self._project}/locations/{self._region}"
+
+ service = self._run_v2.Service(
+ template=self._run_v2.RevisionTemplate(
+ containers=[
+ self._run_v2.Container(image=image),
+ ],
+ ),
+ )
+
+ try:
+ request = self._run_v2.CreateServiceRequest(
+ parent=parent, service=service, service_id=name
+ )
+ operation = client.create_service(request=request)
+ operation.result()
+ except self._exceptions.AlreadyExists:
+ print(" Updating existing service...")
+ service.name = f"{parent}/services/{name}"
+ request = self._run_v2.UpdateServiceRequest(service=service)
+ operation = client.update_service(request=request)
+ operation.result()
diff --git a/dotflow/cloud/gcp/constants.py b/dotflow/cloud/gcp/constants.py
new file mode 100644
index 00000000..22c5f348
--- /dev/null
+++ b/dotflow/cloud/gcp/constants.py
@@ -0,0 +1,11 @@
+"""GCP constants."""
+
+GCP_SDK_NOT_FOUND = (
+ "google-cloud-run is required: pip install dotflow[deploy-gcp]"
+)
+
+CREDENTIALS_NOT_FOUND = (
+ "GCP credentials not found. "
+ "Run 'gcloud auth application-default login' "
+ "or set GOOGLE_APPLICATION_CREDENTIALS environment variable."
+)
diff --git a/dotflow/cloud/gcp/services/__init__.py b/dotflow/cloud/gcp/services/__init__.py
new file mode 100644
index 00000000..753b2638
--- /dev/null
+++ b/dotflow/cloud/gcp/services/__init__.py
@@ -0,0 +1 @@
+"""GCP services module."""
diff --git a/dotflow/cloud/gcp/services/apis.py b/dotflow/cloud/gcp/services/apis.py
new file mode 100644
index 00000000..f9f46111
--- /dev/null
+++ b/dotflow/cloud/gcp/services/apis.py
@@ -0,0 +1,39 @@
+"""GCP API enablement."""
+
+from __future__ import annotations
+
+from dotflow.core.exception import ModuleNotFound
+
+
+class APIs:
+ """Enable GCP APIs via Service Usage SDK."""
+
+ REQUIRED = [
+ "cloudbuild.googleapis.com",
+ "artifactregistry.googleapis.com",
+ "run.googleapis.com",
+ ]
+
+ def __init__(self, project: str):
+ try:
+ from google.cloud import service_usage_v1
+ except ImportError:
+ raise ModuleNotFound(
+ module="google-cloud-service-usage",
+ library="dotflow[deploy-gcp]",
+ ) from None
+
+ self._client = service_usage_v1.ServiceUsageClient()
+ self._project = project
+
+ def enable(self):
+ """Enable all required APIs."""
+ print(" Enabling APIs...")
+ for api in self.REQUIRED:
+ name = f"projects/{self._project}/services/{api}"
+ try:
+ self._client.enable_service(name=name).result()
+ except Exception as err:
+ if "already enabled" in str(err).lower():
+ continue
+ print(f" Warning: Failed to enable {api}: {err}")
diff --git a/dotflow/cloud/gcp/services/artifact_registry.py b/dotflow/cloud/gcp/services/artifact_registry.py
new file mode 100644
index 00000000..891162e6
--- /dev/null
+++ b/dotflow/cloud/gcp/services/artifact_registry.py
@@ -0,0 +1,127 @@
+"""Artifact Registry operations — build and push Docker images."""
+
+from __future__ import annotations
+
+import io
+import os
+import tarfile
+
+from dotflow.cloud.core import Registry
+from dotflow.core.exception import ModuleNotFound
+
+
+class ArtifactRegistry(Registry):
+ """Google Artifact Registry container registry."""
+
+ def __init__(self, project: str, region: str):
+ try:
+ from google.cloud import artifactregistry_v1
+ from google.cloud import storage as gcs
+ from google.cloud.devtools import cloudbuild_v1
+ except ImportError:
+ raise ModuleNotFound(
+ module="google-cloud-build",
+ library="dotflow[deploy-gcp]",
+ ) from None
+
+ self._cloudbuild = cloudbuild_v1
+ self._ar = artifactregistry_v1
+ self._gcs = gcs
+ self._project = project
+ self._region = region
+ self._bucket_name = f"{project}_cloudbuild"
+ self._image_base = (
+ f"{region}-docker.pkg.dev/{project}/cloud-run-source-deploy"
+ )
+
+ def login(self) -> None:
+ """No-op — Cloud Build handles authentication."""
+
+ def push(self, name: str) -> str:
+ """Build and push Docker image via Cloud Build. Returns image URI."""
+ image = f"{self._image_base}/{name}:latest"
+ object_name = f"source/{name}.tar.gz"
+
+ self._ensure_repository(name)
+ self._upload_source(object_name)
+ self._build(image, object_name)
+
+ return image
+
+ def _ensure_repository(self, name: str):
+ """Create Artifact Registry Docker repository if it doesn't exist."""
+ print(" Ensuring Artifact Registry repository...")
+
+ client = self._ar.ArtifactRegistryClient()
+ parent = f"projects/{self._project}/locations/{self._region}"
+ repo_name = "cloud-run-source-deploy"
+
+ try:
+ client.get_repository(name=f"{parent}/repositories/{repo_name}")
+ except Exception:
+ client.create_repository(
+ parent=parent,
+ repository=self._ar.Repository(
+ format_=self._ar.Repository.Format.DOCKER,
+ description=f"Docker images for {name}",
+ ),
+ repository_id=repo_name,
+ ).result()
+
+ def _upload_source(self, object_name: str):
+ """Tar the current directory and upload to GCS."""
+ print(" Uploading source...")
+
+ buf = io.BytesIO()
+ with tarfile.open(fileobj=buf, mode="w:gz") as tar:
+ for entry in os.listdir("."):
+ if entry.startswith("."):
+ continue
+ tar.add(entry)
+ buf.seek(0)
+
+ client = self._gcs.Client(project=self._project)
+ bucket = client.bucket(self._bucket_name)
+
+ try:
+ bucket.reload()
+ except Exception:
+ bucket.create(location=self._region)
+
+ blob = bucket.blob(object_name)
+ blob.upload_from_file(buf, content_type="application/gzip")
+
+ def _build(self, image: str, object_name: str):
+ """Build Docker image via Cloud Build."""
+ print(" Building and pushing image via Cloud Build...")
+
+ client = self._cloudbuild.CloudBuildClient()
+
+ build = self._cloudbuild.Build(
+ source=self._cloudbuild.Source(
+ storage_source=self._cloudbuild.StorageSource(
+ bucket=self._bucket_name,
+ object_=object_name,
+ )
+ ),
+ steps=[
+ self._cloudbuild.BuildStep(
+ name="gcr.io/cloud-builders/docker",
+ args=["build", "-t", image, "."],
+ ),
+ ],
+ images=[image],
+ )
+
+ try:
+ operation = client.create_build(
+ project_id=self._project,
+ build=build,
+ )
+ print(f" Build started: {operation.metadata.build.id}")
+ result = operation.result(timeout=600)
+ print(f" Build status: {result.status.name}")
+ except Exception as err:
+ raise SystemExit(
+ f"Cloud Build failed: {type(err).__name__}: {err}"
+ ) from err
diff --git a/dotflow/cloud/gcp/services/gcs.py b/dotflow/cloud/gcp/services/gcs.py
new file mode 100644
index 00000000..855449a1
--- /dev/null
+++ b/dotflow/cloud/gcp/services/gcs.py
@@ -0,0 +1,45 @@
+"""GCS operations — read and write objects."""
+
+from __future__ import annotations
+
+from json import dumps, loads
+
+from dotflow.cloud.core import ObjectStorage
+from dotflow.core.exception import ModuleNotFound
+
+
+class GCS(ObjectStorage):
+ """Google Cloud Storage object storage."""
+
+ def __init__(self, bucket: str, prefix: str = "", project: str = None):
+ try:
+ from google.api_core.exceptions import NotFound
+ from google.cloud import storage as gcs
+ except ImportError:
+ raise ModuleNotFound(
+ module="google-cloud-storage",
+ library="dotflow[gcp]",
+ ) from None
+
+ self._not_found = NotFound
+ self._client = gcs.Client(project=project)
+ self._bucket = self._client.bucket(bucket)
+ self._bucket.reload()
+ self.prefix = prefix
+
+ def read(self, key: str) -> list:
+ """Read a JSON list from GCS. Returns empty list if key not found."""
+ blob = self._bucket.blob(f"{self.prefix}{key}")
+ try:
+ data = blob.download_as_text()
+ return loads(data)
+ except self._not_found:
+ return []
+
+ def write(self, key: str, data: list) -> None:
+ """Write a JSON list to GCS."""
+ blob = self._bucket.blob(f"{self.prefix}{key}")
+ blob.upload_from_string(
+ dumps(data),
+ content_type="application/json",
+ )
diff --git a/dotflow/cloud/github/__init__.py b/dotflow/cloud/github/__init__.py
new file mode 100644
index 00000000..661313da
--- /dev/null
+++ b/dotflow/cloud/github/__init__.py
@@ -0,0 +1,5 @@
+"""GitHub cloud module."""
+
+from dotflow.cloud.github.actions_deployer import ActionsDeployer
+
+__all__ = ["ActionsDeployer"]
diff --git a/dotflow/cloud/github/actions_deployer.py b/dotflow/cloud/github/actions_deployer.py
new file mode 100644
index 00000000..1059bab9
--- /dev/null
+++ b/dotflow/cloud/github/actions_deployer.py
@@ -0,0 +1,116 @@
+"""GitHub Actions deployment."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+from rich import print # type: ignore
+
+from dotflow.cloud.core import Deployer
+from dotflow.cloud.github.constants import PYGITHUB_NOT_FOUND, TOKEN_NOT_FOUND
+from dotflow.settings import Settings as settings
+
+
+class ActionsDeployer(Deployer):
+ """Deploy dotflow pipeline to GitHub Actions."""
+
+ def __init__(self, token: str = None):
+ try:
+ from github import Github, GithubException
+ except ImportError as err:
+ raise SystemExit(PYGITHUB_NOT_FOUND) from err
+
+ self._token = token or os.environ.get("GITHUB_TOKEN")
+ if not self._token:
+ raise SystemExit(TOKEN_NOT_FOUND)
+
+ self._github = Github(self._token)
+ self._GithubException = GithubException
+ self._user = self._github.get_user()
+
+ def setup(self, name: str) -> None:
+ """No-op — GitHub manages everything."""
+
+ def ensure_roles(self, name: str) -> str:
+ """No-op."""
+ return ""
+
+ def ensure_logs(self, name: str) -> None:
+ """No-op."""
+
+ def deploy(self, name: str, **kwargs) -> None:
+ """Create repo and push all project files."""
+ print(settings.INFO_ALERT, f"Deploying to GitHub Actions '{name}'...")
+
+ repo = self._create_or_get_repo(name)
+ self._push_files(repo)
+
+ print(f" Repository: {repo.html_url}")
+ print(f" Actions: {repo.html_url}/actions")
+ print(settings.INFO_ALERT, "Done.")
+
+ def _create_or_get_repo(self, name: str):
+ """Create repo or get existing."""
+ try:
+ repo = self._user.get_repo(name)
+ print(f" Using existing repo '{name}'")
+ return repo
+ except self._GithubException:
+ pass
+
+ print(f" Creating repo '{name}'...")
+ return self._user.create_repo(
+ name=name,
+ private=True,
+ auto_init=False,
+ )
+
+ def _push_files(self, repo):
+ """Push all project files to the repo."""
+ print(" Pushing files...")
+
+ project_dir = Path.cwd()
+ tracked = self._get_tracked_files(project_dir)
+ count = 0
+
+ for relative in tracked:
+ filepath = project_dir / relative
+
+ try:
+ content = filepath.read_text()
+ except UnicodeDecodeError:
+ continue
+
+ self._create_or_update_file(repo, relative, content)
+ count += 1
+
+ print(f" Pushed {count} files")
+
+ def _get_tracked_files(self, project_dir: Path) -> list[str]:
+ """Get files that git would track, respecting .gitignore."""
+ from git import Repo
+
+ repo = Repo(project_dir)
+ tracked = set(repo.git.ls_files("--cached").splitlines())
+ untracked = set(
+ repo.git.ls_files("--others", "--exclude-standard").splitlines()
+ )
+ return sorted(tracked | untracked)
+
+ def _create_or_update_file(self, repo, path: str, content: str):
+ """Create or update a single file in the repo."""
+ try:
+ existing = repo.get_contents(path)
+ repo.update_file(
+ path=path,
+ message=f"update {path}",
+ content=content,
+ sha=existing.sha,
+ )
+ except self._GithubException:
+ repo.create_file(
+ path=path,
+ message=f"add {path}",
+ content=content,
+ )
diff --git a/dotflow/cloud/github/constants.py b/dotflow/cloud/github/constants.py
new file mode 100644
index 00000000..104fe965
--- /dev/null
+++ b/dotflow/cloud/github/constants.py
@@ -0,0 +1,8 @@
+"""GitHub constants."""
+
+PYGITHUB_NOT_FOUND = "PyGithub is required: pip install dotflow[deploy-github]"
+
+TOKEN_NOT_FOUND = (
+ "GitHub token not found. "
+ "Set GITHUB_TOKEN environment variable or pass --token."
+)
diff --git a/dotflow/logging.py b/dotflow/logging.py
index ffb5d7ee..6d3b42ef 100644
--- a/dotflow/logging.py
+++ b/dotflow/logging.py
@@ -1,27 +1,14 @@
"""Log"""
import logging
-import logging.config
from dotflow.settings import Settings as settings
-settings.START_PATH.mkdir(parents=True, exist_ok=True)
-
-logging.basicConfig(
- filename=settings.LOG_PATH,
- format=settings.LOG_FORMAT,
- level=logging.INFO,
- filemode="a",
-)
-
logger = logging.getLogger(settings.LOG_PROFILE)
logger.setLevel(logging.INFO)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
-
-formatter = logging.Formatter(settings.LOG_FORMAT)
-
-ch.setFormatter(formatter)
+ch.setFormatter(logging.Formatter(settings.LOG_FORMAT))
logger.addHandler(ch)
diff --git a/dotflow/providers/storage_gcs.py b/dotflow/providers/storage_gcs.py
index e39b1bf6..269c604b 100644
--- a/dotflow/providers/storage_gcs.py
+++ b/dotflow/providers/storage_gcs.py
@@ -5,8 +5,8 @@
from typing import Any
from dotflow.abc.storage import Storage
+from dotflow.cloud.gcp.services.gcs import GCS
from dotflow.core.context import Context
-from dotflow.core.exception import ModuleNotFound
class StorageGCS(Storage):
@@ -46,20 +46,7 @@ def __init__(
project: str = None,
**kwargs,
):
- try:
- from google.api_core.exceptions import NotFound
- from google.cloud import storage as gcs
- except ImportError:
- raise ModuleNotFound(
- module="google-cloud-storage",
- library="dotflow[gcp]",
- ) from None
-
- self._not_found = NotFound
- self.client = gcs.Client(project=project)
- self.bucket_obj = self.client.bucket(bucket)
- self.bucket_obj.reload()
- self.prefix = prefix
+ self._gcs = GCS(bucket=bucket, prefix=prefix, project=project)
def post(self, key: str, context: Context) -> None:
task_context = []
@@ -71,10 +58,10 @@ def post(self, key: str, context: Context) -> None:
else:
task_context.append(self._dumps(storage=context.storage))
- self._write(key=key, data=task_context)
+ self._gcs.write(key=key, data=task_context)
def get(self, key: str) -> Context:
- task_context = self._read(key)
+ task_context = self._gcs.read(key)
if len(task_context) == 0:
return Context()
@@ -91,21 +78,6 @@ def get(self, key: str) -> Context:
def key(self, task: Callable):
return f"{task.workflow_id}-{task.task_id}"
- def _read(self, key: str) -> list:
- blob = self.bucket_obj.blob(f"{self.prefix}{key}")
- try:
- data = blob.download_as_text()
- return loads(data)
- except self._not_found:
- return []
-
- def _write(self, key: str, data: list) -> None:
- blob = self.bucket_obj.blob(f"{self.prefix}{key}")
- blob.upload_from_string(
- dumps(data),
- content_type="application/json",
- )
-
def _loads(self, storage: Any) -> Context:
try:
return Context(storage=loads(storage))
diff --git a/dotflow/providers/storage_s3.py b/dotflow/providers/storage_s3.py
index c9733017..4589a27c 100644
--- a/dotflow/providers/storage_s3.py
+++ b/dotflow/providers/storage_s3.py
@@ -5,8 +5,8 @@
from typing import Any
from dotflow.abc.storage import Storage
+from dotflow.cloud.aws.services.s3 import S3
from dotflow.core.context import Context
-from dotflow.core.exception import ModuleNotFound
class StorageS3(Storage):
@@ -46,18 +46,7 @@ def __init__(
region: str = None,
**kwargs,
):
- try:
- import boto3
- except ImportError:
- raise ModuleNotFound(
- module="boto3", library="dotflow[aws]"
- ) from None
-
- self.s3 = boto3.client("s3", region_name=region)
- self.bucket = bucket
- self.prefix = prefix
-
- self.s3.head_bucket(Bucket=self.bucket)
+ self._s3 = S3(bucket=bucket, prefix=prefix, region=region)
def post(self, key: str, context: Context) -> None:
task_context = []
@@ -69,10 +58,10 @@ def post(self, key: str, context: Context) -> None:
else:
task_context.append(self._dumps(storage=context.storage))
- self._write(key=key, data=task_context)
+ self._s3.write(key=key, data=task_context)
def get(self, key: str) -> Context:
- task_context = self._read(key)
+ task_context = self._s3.read(key)
if len(task_context) == 0:
return Context()
@@ -89,25 +78,6 @@ def get(self, key: str) -> Context:
def key(self, task: Callable):
return f"{task.workflow_id}-{task.task_id}"
- def _read(self, key: str) -> list:
- try:
- response = self.s3.get_object(
- Bucket=self.bucket,
- Key=f"{self.prefix}{key}",
- )
- data = response["Body"].read().decode("utf-8")
- return loads(data)
- except self.s3.exceptions.NoSuchKey:
- return []
-
- def _write(self, key: str, data: list) -> None:
- self.s3.put_object(
- Bucket=self.bucket,
- Key=f"{self.prefix}{key}",
- Body=dumps(data),
- ContentType="application/json",
- )
-
def _loads(self, storage: Any) -> Context:
try:
return Context(storage=loads(storage))
diff --git a/dotflow/settings.py b/dotflow/settings.py
index f75df4be..00e53347 100644
--- a/dotflow/settings.py
+++ b/dotflow/settings.py
@@ -1,19 +1,26 @@
"""Settings"""
+import os
from pathlib import Path
+_OUTPUT = Path(os.environ.get("DOTFLOW_OUTPUT_PATH", ".output"))
+
class Settings:
"""Settings DotFlow"""
- START_PATH = Path(".output")
+ START_PATH = _OUTPUT
GITIGNORE = Path(".gitignore")
LOG_PROFILE = "dotflow"
LOG_FILE_NAME = Path("flow.log")
- LOG_PATH = Path(".output/flow.log")
+ LOG_PATH = _OUTPUT / "flow.log"
LOG_FORMAT = "%(asctime)s - %(levelname)s [%(name)s]: %(message)s"
+ TEMPLATE_REPO = "https://github.com/dotflow-io/template.git"
+ TEMPLATE_BRANCH = "master"
+ TEMPLATE_CLOUD_DIR = "cloud"
+
ICON = ":game_die:"
ERROR_ALERT = f"{ICON} [bold red]Error:[/bold red]"
INFO_ALERT = f"{ICON} [bold blue]Info:[/bold blue]"
diff --git a/mkdocs.yml b/mkdocs.yml
index 58f749ca..c1c0166a 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -222,30 +222,6 @@ nav:
- nav/concepts/process-mode-parallel-group.md
- How-to guides:
- Overview: nav/how-to/index.md
- - Providers:
- - Storage:
- - nav/tutorial/storage-default.md
- - nav/tutorial/storage-file.md
- - nav/tutorial/storage-s3.md
- - nav/tutorial/storage-gcs.md
- - Notify:
- - nav/tutorial/notify-default.md
- - nav/tutorial/notify-telegram.md
- - nav/tutorial/notify-discord.md
- - Log:
- - nav/tutorial/log-default.md
- - nav/tutorial/log-opentelemetry.md
- - nav/tutorial/log-sentry.md
- - Scheduler:
- - nav/tutorial/scheduler-default.md
- - nav/tutorial/scheduler-cron.md
- - Tracer:
- - nav/tutorial/tracer-default.md
- - nav/tutorial/tracer-opentelemetry.md
- - nav/tutorial/tracer-sentry.md
- - Metrics:
- - nav/tutorial/metrics-default.md
- - nav/tutorial/metrics-opentelemetry.md
- Workflows:
- nav/tutorial/install.md
- nav/tutorial/first-steps.md
@@ -302,6 +278,25 @@ nav:
- nav/tutorial/log-default.md
- nav/tutorial/scheduler-default.md
- nav/tutorial/scheduler-cron.md
+ - Cloud:
+ - Overview: nav/cloud/index.md
+ - Docker:
+ - nav/cloud/docker.md
+ - AWS:
+ - nav/cloud/lambda.md
+ - nav/cloud/lambda-scheduled.md
+ - nav/cloud/lambda-s3-trigger.md
+ - nav/cloud/lambda-sqs-trigger.md
+ - nav/cloud/lambda-api-trigger.md
+ - nav/cloud/ecs.md
+ - nav/cloud/ecs-scheduled.md
+ - GCP:
+ - nav/cloud/cloud-run.md
+ - nav/cloud/cloud-run-scheduled.md
+ - Kubernetes:
+ - nav/cloud/kubernetes.md
+ - CI/CD:
+ - nav/cloud/github-actions.md
- Advanced:
- nav/advanced/scheduled-pipeline-with-resume.md
- nav/advanced/parallel-with-groups.md
diff --git a/poetry.lock b/poetry.lock
index 3cdc17a5..f451a3f8 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -308,7 +308,7 @@ files = [
{file = "cffi-2.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:b882b3df248017dba09d6b16defe9b5c407fe32fc7c65a9c69798e6175601be9"},
{file = "cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529"},
]
-markers = {main = "platform_python_implementation != \"PyPy\" and extra == \"gcp\"", dev = "platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\""}
+markers = {main = "platform_python_implementation != \"PyPy\" and (extra == \"gcp\" or extra == \"deploy-gcp\" or extra == \"deploy-github\")", dev = "platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\""}
[package.dependencies]
pycparser = {version = "*", markers = "implementation_name != \"PyPy\""}
@@ -852,7 +852,7 @@ files = [
{file = "cryptography-43.0.3-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:2ce6fae5bdad59577b44e4dfed356944fbf1d925269114c28be377692643b4ff"},
{file = "cryptography-43.0.3.tar.gz", hash = "sha256:315b9001266a492a6ff443b61238f956b214dbec9910a081ba5b6646a055a805"},
]
-markers = {main = "python_version < \"3.10\" and extra == \"gcp\"", dev = "python_version < \"3.10\""}
+markers = {main = "python_version < \"3.10\" and (extra == \"gcp\" or extra == \"deploy-gcp\" or extra == \"deploy-github\")", dev = "python_version < \"3.10\""}
[package.dependencies]
cffi = {version = ">=1.12", markers = "platform_python_implementation != \"PyPy\""}
@@ -925,7 +925,7 @@ files = [
{file = "cryptography-46.0.6-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6728c49e3b2c180ef26f8e9f0a883a2c585638db64cf265b49c9ba10652d430e"},
{file = "cryptography-46.0.6.tar.gz", hash = "sha256:27550628a518c5c6c903d84f637fbecf287f6cb9ced3804838a1295dc1fd0759"},
]
-markers = {main = "python_version >= \"3.10\" and extra == \"gcp\"", dev = "python_version >= \"3.10\""}
+markers = {main = "python_version >= \"3.10\" and (extra == \"gcp\" or extra == \"deploy-gcp\" or extra == \"deploy-github\")", dev = "python_version >= \"3.10\""}
[package.dependencies]
cffi = {version = ">=2.0.0", markers = "python_full_version >= \"3.9.0\" and platform_python_implementation != \"PyPy\""}
@@ -1068,7 +1068,7 @@ description = "Google API client core library"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "google_api_core-2.30.2-py3-none-any.whl", hash = "sha256:a4c226766d6af2580577db1f1a51bf53cd262f722b49731ce7414c43068a9594"},
{file = "google_api_core-2.30.2.tar.gz", hash = "sha256:9a8113e1a88bdc09a7ff629707f2214d98d61c7f6ceb0ea38c42a095d02dc0f9"},
@@ -1077,9 +1077,19 @@ files = [
[package.dependencies]
google-auth = ">=2.14.1,<3.0.0"
googleapis-common-protos = ">=1.63.2,<2.0.0"
+grpcio = [
+ {version = ">=1.75.1,<2.0.0", optional = true, markers = "python_version >= \"3.14\" and extra == \"grpc\""},
+ {version = ">=1.49.1,<2.0.0", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\""},
+ {version = ">=1.33.2,<2.0.0", optional = true, markers = "python_version < \"3.11\" and extra == \"grpc\""},
+]
+grpcio-status = [
+ {version = ">=1.75.1,<2.0.0", optional = true, markers = "python_version >= \"3.14\" and extra == \"grpc\""},
+ {version = ">=1.49.1,<2.0.0", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\""},
+ {version = ">=1.33.2,<2.0.0", optional = true, markers = "extra == \"grpc\""},
+]
proto-plus = [
{version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
- {version = ">=1.22.3,<2.0.0", markers = "python_version < \"3.13\""},
+ {version = ">=1.22.3,<2.0.0"},
]
protobuf = ">=4.25.8,<8.0.0"
requests = ">=2.20.0,<3.0.0"
@@ -1095,7 +1105,7 @@ description = "Google Authentication Library"
optional = true
python-versions = ">=3.8"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "google_auth-2.49.1-py3-none-any.whl", hash = "sha256:195ebe3dca18eddd1b3db5edc5189b76c13e96f29e73043b923ebcf3f1a860f7"},
{file = "google_auth-2.49.1.tar.gz", hash = "sha256:16d40da1c3c5a0533f57d268fe72e0ebb0ae1cc3b567024122651c045d879b64"},
@@ -1117,6 +1127,60 @@ rsa = ["rsa (>=3.1.4,<5)"]
testing = ["aiohttp (<3.10.0)", "aiohttp (>=3.6.2,<4.0.0)", "aioresponses", "flask", "freezegun", "grpcio", "packaging", "pyjwt (>=2.0)", "pyopenssl (<24.3.0)", "pyopenssl (>=20.0.0)", "pytest", "pytest-asyncio", "pytest-cov", "pytest-localserver", "pyu2f (>=0.1.5)", "requests (>=2.20.0,<3.0.0)", "responses", "urllib3"]
urllib3 = ["packaging", "urllib3"]
+[[package]]
+name = "google-cloud-artifact-registry"
+version = "1.21.0"
+description = "Google Cloud Artifact Registry API client library"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "google_cloud_artifact_registry-1.21.0-py3-none-any.whl", hash = "sha256:a07079035438fd0f2e7264d4318b388650495f011db575405c18c9881449025c"},
+ {file = "google_cloud_artifact_registry-1.21.0.tar.gz", hash = "sha256:546e51eb5d463a6e5c668be6727d14f8ec82bc798031398006b2213d703e184c"},
+]
+
+[package.dependencies]
+google-api-core = {version = ">=2.11.0,<3.0.0", extras = ["grpc"]}
+google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
+grpc-google-iam-v1 = ">=0.14.0,<1.0.0"
+grpcio = [
+ {version = ">=1.75.1,<2.0.0", markers = "python_version >= \"3.14\""},
+ {version = ">=1.33.2,<2.0.0", markers = "python_version < \"3.14\""},
+]
+proto-plus = [
+ {version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
+ {version = ">=1.22.3,<2.0.0", markers = "python_version < \"3.13\""},
+]
+protobuf = ">=4.25.8,<8.0.0"
+
+[[package]]
+name = "google-cloud-build"
+version = "3.36.0"
+description = "Google Cloud Build API client library"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "google_cloud_build-3.36.0-py3-none-any.whl", hash = "sha256:4a350eb90c384a8ce08649a3384c3813bf4708590e02353f953cc57b1aa2e15f"},
+ {file = "google_cloud_build-3.36.0.tar.gz", hash = "sha256:f70de0882ffd0f48ff39b7fcb8b543c2eaeda06c2f067b8aa61125b9427d3ffc"},
+]
+
+[package.dependencies]
+google-api-core = {version = ">=2.11.0,<3.0.0", extras = ["grpc"]}
+google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
+grpc-google-iam-v1 = ">=0.12.4,<1.0.0"
+grpcio = [
+ {version = ">=1.75.1,<2.0.0", markers = "python_version >= \"3.14\""},
+ {version = ">=1.33.2,<2.0.0"},
+]
+proto-plus = [
+ {version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
+ {version = ">=1.22.3,<2.0.0"},
+]
+protobuf = ">=4.25.8,<8.0.0"
+
[[package]]
name = "google-cloud-core"
version = "2.5.1"
@@ -1124,7 +1188,7 @@ description = "Google Cloud API client core library"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "google_cloud_core-2.5.1-py3-none-any.whl", hash = "sha256:ea62cdf502c20e3e14be8a32c05ed02113d7bef454e40ff3fab6fe1ec9f1f4e7"},
{file = "google_cloud_core-2.5.1.tar.gz", hash = "sha256:3dc94bdec9d05a31d9f355045ed0f369fbc0d8c665076c734f065d729800f811"},
@@ -1137,6 +1201,59 @@ google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
[package.extras]
grpc = ["grpcio (>=1.38.0,<2.0.0) ; python_version < \"3.14\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "grpcio-status (>=1.38.0,<2.0.0)"]
+[[package]]
+name = "google-cloud-run"
+version = "0.16.0"
+description = "Google Cloud Run API client library"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "google_cloud_run-0.16.0-py3-none-any.whl", hash = "sha256:d7d2dd7307130fde2a0ce27e96d580dd23b7b2d973b6484b94d902e6b2618860"},
+ {file = "google_cloud_run-0.16.0.tar.gz", hash = "sha256:d52cf4e6ad3702ae48caccf6abcab543afee6f61c2a6ec753cc62a31e5b629f1"},
+]
+
+[package.dependencies]
+google-api-core = {version = ">=2.11.0,<3.0.0", extras = ["grpc"]}
+google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
+grpc-google-iam-v1 = ">=0.14.0,<1.0.0"
+grpcio = [
+ {version = ">=1.75.1,<2.0.0", markers = "python_version >= \"3.14\""},
+ {version = ">=1.33.2,<2.0.0"},
+]
+proto-plus = [
+ {version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
+ {version = ">=1.22.3,<2.0.0"},
+]
+protobuf = ">=4.25.8,<8.0.0"
+
+[[package]]
+name = "google-cloud-service-usage"
+version = "1.16.0"
+description = "Google Cloud Service Usage API client library"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "google_cloud_service_usage-1.16.0-py3-none-any.whl", hash = "sha256:592ba0f799801e8bb1e9f77c451cd3bd8f4e875336951eb5ca4944189f241ec2"},
+ {file = "google_cloud_service_usage-1.16.0.tar.gz", hash = "sha256:2c4d16f2d1cdc029c65350ebaa81791da010cd59960204f37fb72c713b3f9658"},
+]
+
+[package.dependencies]
+google-api-core = {version = ">=2.11.0,<3.0.0", extras = ["grpc"]}
+google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0"
+grpcio = [
+ {version = ">=1.75.1,<2.0.0", markers = "python_version >= \"3.14\""},
+ {version = ">=1.33.2,<2.0.0"},
+]
+proto-plus = [
+ {version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""},
+ {version = ">=1.22.3,<2.0.0"},
+]
+protobuf = ">=4.25.8,<8.0.0"
+
[[package]]
name = "google-cloud-storage"
version = "3.9.0"
@@ -1144,7 +1261,7 @@ description = "Google Cloud Storage API client library"
optional = true
python-versions = ">=3.7"
groups = ["main"]
-markers = "python_version < \"3.10\" and extra == \"gcp\""
+markers = "python_version < \"3.10\" and (extra == \"gcp\" or extra == \"deploy-gcp\")"
files = [
{file = "google_cloud_storage-3.9.0-py3-none-any.whl", hash = "sha256:2dce75a9e8b3387078cbbdad44757d410ecdb916101f8ba308abf202b6968066"},
{file = "google_cloud_storage-3.9.0.tar.gz", hash = "sha256:f2d8ca7db2f652be757e92573b2196e10fbc09649b5c016f8b422ad593c641cc"},
@@ -1171,7 +1288,7 @@ description = "Google Cloud Storage API client library"
optional = true
python-versions = ">=3.10"
groups = ["main"]
-markers = "python_version >= \"3.10\" and extra == \"gcp\""
+markers = "python_version >= \"3.10\" and (extra == \"gcp\" or extra == \"deploy-gcp\")"
files = [
{file = "google_cloud_storage-3.10.1-py3-none-any.whl", hash = "sha256:a72f656759b7b99bda700f901adcb3425a828d4a29f911bc26b3ea79c5b1217f"},
{file = "google_cloud_storage-3.10.1.tar.gz", hash = "sha256:97db9aa4460727982040edd2bd13ff3d5e2260b5331ad22895802da1fc2a5286"},
@@ -1198,7 +1315,7 @@ description = "A python wrapper of the C library 'Google CRC32C'"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "google_crc32c-1.8.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:0470b8c3d73b5f4e3300165498e4cf25221c7eb37f1159e221d1825b6df8a7ff"},
{file = "google_crc32c-1.8.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:119fcd90c57c89f30040b47c211acee231b25a45d225e3225294386f5d258288"},
@@ -1242,7 +1359,7 @@ description = "Utilities for Google Media Downloads and Resumable Uploads"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "google_resumable_media-2.8.2-py3-none-any.whl", hash = "sha256:82b6d8ccd11765268cdd2a2123f417ec806b8eef3000a9a38dfe3033da5fb220"},
{file = "google_resumable_media-2.8.2.tar.gz", hash = "sha256:f3354a182ebd193ae3f42e3ef95e6c9b10f128320de23ac7637236713b1acd70"},
@@ -1262,13 +1379,14 @@ description = "Common protobufs used in Google APIs"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "python_version >= \"3.11\" and (extra == \"deploy-gcp\" or extra == \"gcp\") or extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "googleapis_common_protos-1.74.0-py3-none-any.whl", hash = "sha256:702216f78610bb510e3f12ac3cafd281b7ac45cc5d86e90ad87e4d301a3426b5"},
{file = "googleapis_common_protos-1.74.0.tar.gz", hash = "sha256:57971e4eeeba6aad1163c1f0fc88543f965bb49129b8bb55b2b7b26ecab084f1"},
]
[package.dependencies]
+grpcio = {version = ">=1.44.0,<2.0.0", optional = true, markers = "extra == \"grpc\""}
protobuf = ">=4.25.8,<8.0.0"
[package.extras]
@@ -1305,6 +1423,120 @@ files = [
griffe = ">=0.49"
typing-extensions = ">=4.7"
+[[package]]
+name = "grpc-google-iam-v1"
+version = "0.14.4"
+description = "IAM API client library"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "grpc_google_iam_v1-0.14.4-py3-none-any.whl", hash = "sha256:412facc320fcbd94034b4df3d557662051d4d8adfa86e0ddb4dca70a3f739964"},
+ {file = "grpc_google_iam_v1-0.14.4.tar.gz", hash = "sha256:392b3796947ed6334e61171d9ab06bf7eb357f554e5fc7556ad7aab6d0e17038"},
+]
+
+[package.dependencies]
+googleapis-common-protos = {version = ">=1.63.2,<2.0.0", extras = ["grpc"]}
+grpcio = ">=1.44.0,<2.0.0"
+protobuf = ">=4.25.8,<8.0.0"
+
+[[package]]
+name = "grpcio"
+version = "1.80.0"
+description = "HTTP/2-based RPC framework"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "grpcio-1.80.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:886457a7768e408cdce226ad1ca67d2958917d306523a0e21e1a2fdaa75c9c9c"},
+ {file = "grpcio-1.80.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:7b641fc3f1dc647bfd80bd713addc68f6d145956f64677e56d9ebafc0bd72388"},
+ {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:33eb763f18f006dc7fee1e69831d38d23f5eccd15b2e0f92a13ee1d9242e5e02"},
+ {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:52d143637e3872633fc7dd7c3c6a1c84e396b359f3a72e215f8bf69fd82084fc"},
+ {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c51bf8ac4575af2e0678bccfb07e47321fc7acb5049b4482832c5c195e04e13a"},
+ {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:50a9871536d71c4fba24ee856abc03a87764570f0c457dd8db0b4018f379fed9"},
+ {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:a72d84ad0514db063e21887fbacd1fd7acb4d494a564cae22227cd45c7fbf199"},
+ {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f7691a6788ad9196872f95716df5bc643ebba13c97140b7a5ee5c8e75d1dea81"},
+ {file = "grpcio-1.80.0-cp310-cp310-win32.whl", hash = "sha256:46c2390b59d67f84e882694d489f5b45707c657832d7934859ceb8c33f467069"},
+ {file = "grpcio-1.80.0-cp310-cp310-win_amd64.whl", hash = "sha256:dc053420fc75749c961e2a4c906398d7c15725d36ccc04ae6d16093167223b58"},
+ {file = "grpcio-1.80.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:dfab85db094068ff42e2a3563f60ab3dddcc9d6488a35abf0132daec13209c8a"},
+ {file = "grpcio-1.80.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:5c07e82e822e1161354e32da2662f741a4944ea955f9f580ec8fb409dd6f6060"},
+ {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ba0915d51fd4ced2db5ff719f84e270afe0e2d4c45a7bdb1e8d036e4502928c2"},
+ {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:3cb8130ba457d2aa09fa6b7c3ed6b6e4e6a2685fce63cb803d479576c4d80e21"},
+ {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:09e5e478b3d14afd23f12e49e8b44c8684ac3c5f08561c43a5b9691c54d136ab"},
+ {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:00168469238b022500e486c1c33916acf2f2a9b2c022202cf8a1885d2e3073c1"},
+ {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8502122a3cc1714038e39a0b071acb1207ca7844208d5ea0d091317555ee7106"},
+ {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ce1794f4ea6cc3ca29463f42d665c32ba1b964b48958a66497917fe9069f26e6"},
+ {file = "grpcio-1.80.0-cp311-cp311-win32.whl", hash = "sha256:51b4a7189b0bef2aa30adce3c78f09c83526cf3dddb24c6a96555e3b97340440"},
+ {file = "grpcio-1.80.0-cp311-cp311-win_amd64.whl", hash = "sha256:02e64bb0bb2da14d947a49e6f120a75e947250aebe65f9629b62bb1f5c14e6e9"},
+ {file = "grpcio-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:c624cc9f1008361014378c9d776de7182b11fe8b2e5a81bc69f23a295f2a1ad0"},
+ {file = "grpcio-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:f49eddcac43c3bf350c0385366a58f36bed8cc2c0ec35ef7b74b49e56552c0c2"},
+ {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d334591df610ab94714048e0d5b4f3dd5ad1bee74dfec11eee344220077a79de"},
+ {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0cb517eb1d0d0aaf1d87af7cc5b801d686557c1d88b2619f5e31fab3c2315921"},
+ {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4e78c4ac0d97dc2e569b2f4bcbbb447491167cb358d1a389fc4af71ab6f70411"},
+ {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2ed770b4c06984f3b47eb0517b1c69ad0b84ef3f40128f51448433be904634cd"},
+ {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:256507e2f524092f1473071a05e65a5b10d84b82e3ff24c5b571513cfaa61e2f"},
+ {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9a6284a5d907c37db53350645567c522be314bac859a64a7a5ca63b77bb7958f"},
+ {file = "grpcio-1.80.0-cp312-cp312-win32.whl", hash = "sha256:c71309cfce2f22be26aa4a847357c502db6c621f1a49825ae98aa0907595b193"},
+ {file = "grpcio-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:9fe648599c0e37594c4809d81a9e77bd138cc82eb8baa71b6a86af65426723ff"},
+ {file = "grpcio-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e9e408fc016dffd20661f0126c53d8a31c2821b5c13c5d67a0f5ed5de93319ad"},
+ {file = "grpcio-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:92d787312e613754d4d8b9ca6d3297e69994a7912a32fa38c4c4e01c272974b0"},
+ {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac393b58aa16991a2f1144ec578084d544038c12242da3a215966b512904d0f"},
+ {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:68e5851ac4b9afe07e7f84483803ad167852570d65326b34d54ca560bfa53fb6"},
+ {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:873ff5d17d68992ef6605330127425d2fc4e77e612fa3c3e0ed4e668685e3140"},
+ {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2bea16af2750fd0a899bf1abd9022244418b55d1f37da2202249ba4ba673838d"},
+ {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba0db34f7e1d803a878284cd70e4c63cb6ae2510ba51937bf8f45ba997cefcf7"},
+ {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8eb613f02d34721f1acf3626dfdb3545bd3c8505b0e52bf8b5710a28d02e8aa7"},
+ {file = "grpcio-1.80.0-cp313-cp313-win32.whl", hash = "sha256:93b6f823810720912fd131f561f91f5fed0fda372b6b7028a2681b8194d5d294"},
+ {file = "grpcio-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:e172cf795a3ba5246d3529e4d34c53db70e888fa582a8ffebd2e6e48bc0cba50"},
+ {file = "grpcio-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:3d4147a97c8344d065d01bbf8b6acec2cf86fb0400d40696c8bdad34a64ffc0e"},
+ {file = "grpcio-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d8e11f167935b3eb089ac9038e1a063e6d7dbe995c0bb4a661e614583352e76f"},
+ {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f14b618fc30de822681ee986cfdcc2d9327229dc4c98aed16896761cacd468b9"},
+ {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4ed39fbdcf9b87370f6e8df4e39ca7b38b3e5e9d1b0013c7b6be9639d6578d14"},
+ {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2dcc70e9f0ba987526e8e8603a610fb4f460e42899e74e7a518bf3c68fe1bf05"},
+ {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:448c884b668b868562b1bda833c5fce6272d26e1926ec46747cda05741d302c1"},
+ {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a1dc80fe55685b4a543555e6eef975303b36c8db1023b1599b094b92aa77965f"},
+ {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:31b9ac4ad1aa28ffee5503821fafd09e4da0a261ce1c1281c6c8da0423c83b6e"},
+ {file = "grpcio-1.80.0-cp314-cp314-win32.whl", hash = "sha256:367ce30ba67d05e0592470428f0ec1c31714cab9ef19b8f2e37be1f4c7d32fae"},
+ {file = "grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f"},
+ {file = "grpcio-1.80.0-cp39-cp39-linux_armv7l.whl", hash = "sha256:aacdfb4ed3eb919ca997504d27e03d5dba403c85130b8ed450308590a738f7a4"},
+ {file = "grpcio-1.80.0-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:a361c20ec1ccd3c3953d20fb6d7b4125093bdd10dff44c5e2bbb39e58917cedc"},
+ {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:43168871f170d1e4ed16ae03d10cd21efa29f190e710a624cee7e5ae07da6f4f"},
+ {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1b97cd29a8eda100b559b455331c487a80915b6ea6bd91cf3e89836c4ee8d957"},
+ {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bac1d573dfa84ce59a5547073e28fa7326d53352adda6912e362da0b917fcef4"},
+ {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:4560cf0e86514595dbbd330cd65b7afad4b5c4b8c4905c041cfffa138d45e6fd"},
+ {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:ec0a592e926071b4abad50c1495cd0d0d513324b3ff5e7267067c33ba27506e4"},
+ {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:deb10a1528473c11f72a0939eed36d83e847d7cbb63e8cc5611fb7a912d38614"},
+ {file = "grpcio-1.80.0-cp39-cp39-win32.whl", hash = "sha256:627fb7312171cdc52828bd6fac8d7028ff2a64b89f1957b6f3416caa2218d141"},
+ {file = "grpcio-1.80.0-cp39-cp39-win_amd64.whl", hash = "sha256:05d55e1798756282cddd52d56c896b3e7d673e3a8798c2f1cd05ba249a3bb4de"},
+ {file = "grpcio-1.80.0.tar.gz", hash = "sha256:29aca15edd0688c22ba01d7cc01cb000d72b2033f4a3c72a81a19b56fd143257"},
+]
+
+[package.dependencies]
+typing-extensions = ">=4.12,<5.0"
+
+[package.extras]
+protobuf = ["grpcio-tools (>=1.80.0)"]
+
+[[package]]
+name = "grpcio-status"
+version = "1.80.0"
+description = "Status proto mapping for gRPC"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-gcp\""
+files = [
+ {file = "grpcio_status-1.80.0-py3-none-any.whl", hash = "sha256:4b56990363af50dbf2c2ebb80f1967185c07d87aa25aa2bea45ddb75fc181dbe"},
+ {file = "grpcio_status-1.80.0.tar.gz", hash = "sha256:df73802a4c89a3ea88aa2aff971e886fccce162bc2e6511408b3d67a144381cd"},
+]
+
+[package.dependencies]
+googleapis-common-protos = ">=1.5.5"
+grpcio = ">=1.80.0"
+protobuf = ">=6.31.1,<7.0.0"
+
[[package]]
name = "idna"
version = "3.11"
@@ -1324,14 +1556,14 @@ all = ["flake8 (>=7.1.1)", "mypy (>=1.11.2)", "pytest (>=8.3.2)", "ruff (>=0.6.2
name = "importlib-metadata"
version = "8.7.1"
description = "Read metadata from Python packages"
-optional = false
+optional = true
python-versions = ">=3.9"
-groups = ["dev", "docs"]
-markers = "python_version < \"3.10\""
+groups = ["main", "dev", "docs"]
files = [
{file = "importlib_metadata-8.7.1-py3-none-any.whl", hash = "sha256:5a1f80bf1daa489495071efbb095d75a634cf28a8bc299581244063b53176151"},
{file = "importlib_metadata-8.7.1.tar.gz", hash = "sha256:49fef1ae6440c182052f407c8d34a68f72efc36db9ca90dc0113398f2fdde8bb"},
]
+markers = {main = "extra == \"otel\"", dev = "python_full_version < \"3.10.2\"", docs = "python_version < \"3.10\""}
[package.dependencies]
zipp = ">=3.20"
@@ -1345,31 +1577,6 @@ perf = ["ipython"]
test = ["flufl.flake8", "jaraco.test (>=5.4)", "packaging", "pyfakefs", "pytest (>=6,!=8.1.*)", "pytest-perf (>=0.9.2)"]
type = ["mypy (<1.19) ; platform_python_implementation == \"PyPy\"", "pytest-mypy (>=1.0.1)"]
-[[package]]
-name = "importlib-metadata"
-version = "9.0.0"
-description = "Read metadata from Python packages"
-optional = false
-python-versions = ">=3.10"
-groups = ["dev"]
-markers = "python_version == \"3.10\" and python_full_version < \"3.10.2\""
-files = [
- {file = "importlib_metadata-9.0.0-py3-none-any.whl", hash = "sha256:2d21d1cc5a017bd0559e36150c21c830ab1dc304dedd1b7ea85d20f45ef3edd7"},
- {file = "importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc"},
-]
-
-[package.dependencies]
-zipp = ">=3.20"
-
-[package.extras]
-check = ["pytest-checkdocs (>=2.14)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\""]
-cover = ["pytest-cov"]
-doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"]
-enabler = ["pytest-enabler (>=3.4)"]
-perf = ["ipython"]
-test = ["packaging", "pyfakefs", "pytest (>=6,!=8.1.*)", "pytest-perf (>=0.9.2)"]
-type = ["pytest-mypy (>=1.0.1) ; platform_python_implementation != \"PyPy\""]
-
[[package]]
name = "importlib-resources"
version = "6.5.2"
@@ -2262,6 +2469,58 @@ files = [
{file = "mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558"},
]
+[[package]]
+name = "opentelemetry-api"
+version = "1.40.0"
+description = "OpenTelemetry Python API"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"otel\""
+files = [
+ {file = "opentelemetry_api-1.40.0-py3-none-any.whl", hash = "sha256:82dd69331ae74b06f6a874704be0cfaa49a1650e1537d4a813b86ecef7d0ecf9"},
+ {file = "opentelemetry_api-1.40.0.tar.gz", hash = "sha256:159be641c0b04d11e9ecd576906462773eb97ae1b657730f0ecf64d32071569f"},
+]
+
+[package.dependencies]
+importlib-metadata = ">=6.0,<8.8.0"
+typing-extensions = ">=4.5.0"
+
+[[package]]
+name = "opentelemetry-sdk"
+version = "1.40.0"
+description = "OpenTelemetry Python SDK"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"otel\""
+files = [
+ {file = "opentelemetry_sdk-1.40.0-py3-none-any.whl", hash = "sha256:787d2154a71f4b3d81f20524a8ce061b7db667d24e46753f32a7bc48f1c1f3f1"},
+ {file = "opentelemetry_sdk-1.40.0.tar.gz", hash = "sha256:18e9f5ec20d859d268c7cb3c5198c8d105d073714db3de50b593b8c1345a48f2"},
+]
+
+[package.dependencies]
+opentelemetry-api = "1.40.0"
+opentelemetry-semantic-conventions = "0.61b0"
+typing-extensions = ">=4.5.0"
+
+[[package]]
+name = "opentelemetry-semantic-conventions"
+version = "0.61b0"
+description = "OpenTelemetry Semantic Conventions"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"otel\""
+files = [
+ {file = "opentelemetry_semantic_conventions-0.61b0-py3-none-any.whl", hash = "sha256:fa530a96be229795f8cef353739b618148b0fe2b4b3f005e60e262926c4d38e2"},
+ {file = "opentelemetry_semantic_conventions-0.61b0.tar.gz", hash = "sha256:072f65473c5d7c6dc0355b27d6c9d1a679d63b6d4b4b16a9773062cb7e31192a"},
+]
+
+[package.dependencies]
+opentelemetry-api = "1.40.0"
+typing-extensions = ">=4.5.0"
+
[[package]]
name = "packaging"
version = "26.0"
@@ -2392,7 +2651,7 @@ description = "Beautiful, Pythonic protocol buffers"
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "proto_plus-1.27.2-py3-none-any.whl", hash = "sha256:6432f75893d3b9e70b9c412f1d2f03f65b11fb164b793d14ae2ca01821d22718"},
{file = "proto_plus-1.27.2.tar.gz", hash = "sha256:b2adde53adadf75737c44d3dcb0104fde65250dfc83ad59168b4aa3e574b6a24"},
@@ -2411,7 +2670,7 @@ description = ""
optional = true
python-versions = ">=3.9"
groups = ["main"]
-markers = "python_version < \"3.10\" and extra == \"gcp\""
+markers = "python_version >= \"3.11\" and (extra == \"deploy-gcp\" or extra == \"gcp\") or extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3"},
{file = "protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326"},
@@ -2425,25 +2684,6 @@ files = [
{file = "protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135"},
]
-[[package]]
-name = "protobuf"
-version = "7.34.1"
-description = ""
-optional = true
-python-versions = ">=3.10"
-groups = ["main"]
-markers = "python_version >= \"3.10\" and extra == \"gcp\""
-files = [
- {file = "protobuf-7.34.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:d8b2cc79c4d8f62b293ad9b11ec3aebce9af481fa73e64556969f7345ebf9fc7"},
- {file = "protobuf-7.34.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:5185e0e948d07abe94bb76ec9b8416b604cfe5da6f871d67aad30cbf24c3110b"},
- {file = "protobuf-7.34.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:403b093a6e28a960372b44e5eb081775c9b056e816a8029c61231743d63f881a"},
- {file = "protobuf-7.34.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:8ff40ce8cd688f7265326b38d5a1bed9bfdf5e6723d49961432f83e21d5713e4"},
- {file = "protobuf-7.34.1-cp310-abi3-win32.whl", hash = "sha256:34b84ce27680df7cca9f231043ada0daa55d0c44a2ddfaa58ec1d0d89d8bf60a"},
- {file = "protobuf-7.34.1-cp310-abi3-win_amd64.whl", hash = "sha256:e97b55646e6ce5cbb0954a8c28cd39a5869b59090dfaa7df4598a7fba869468c"},
- {file = "protobuf-7.34.1-py3-none-any.whl", hash = "sha256:bb3812cd53aefea2b028ef42bd780f5b96407247f20c6ef7c679807e9d188f11"},
- {file = "protobuf-7.34.1.tar.gz", hash = "sha256:9ce42245e704cc5027be797c1db1eb93184d44d1cdd71811fb2d9b25ad541280"},
-]
-
[[package]]
name = "pyasn1"
version = "0.6.3"
@@ -2451,7 +2691,7 @@ description = "Pure-Python implementation of ASN.1 types and DER/BER/CER codecs
optional = true
python-versions = ">=3.8"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "pyasn1-0.6.3-py3-none-any.whl", hash = "sha256:a80184d120f0864a52a073acc6fc642847d0be408e7c7252f31390c0f4eadcde"},
{file = "pyasn1-0.6.3.tar.gz", hash = "sha256:697a8ecd6d98891189184ca1fa05d1bb00e2f84b5977c481452050549c8a72cf"},
@@ -2464,7 +2704,7 @@ description = "A collection of ASN.1-based protocols modules"
optional = true
python-versions = ">=3.8"
groups = ["main"]
-markers = "extra == \"gcp\""
+markers = "extra == \"gcp\" or extra == \"deploy-gcp\""
files = [
{file = "pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a"},
{file = "pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6"},
@@ -2496,7 +2736,7 @@ files = [
{file = "pycparser-2.23-py3-none-any.whl", hash = "sha256:e5c6e8d3fbad53479cab09ac03729e0a9faf2bee3db8208a550daf5af81a5934"},
{file = "pycparser-2.23.tar.gz", hash = "sha256:78816d4f24add8f10a06d6f05b4d424ad9e96cfebf68a4ddc99c65c0720d00c2"},
]
-markers = {main = "extra == \"gcp\" and platform_python_implementation != \"PyPy\" and implementation_name != \"PyPy\" and python_version < \"3.10\"", dev = "(platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\") and python_version < \"3.10\" and implementation_name != \"PyPy\""}
+markers = {main = "(extra == \"gcp\" or extra == \"deploy-gcp\" or extra == \"deploy-github\") and platform_python_implementation != \"PyPy\" and implementation_name != \"PyPy\" and python_version < \"3.10\"", dev = "(platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\") and python_version < \"3.10\" and implementation_name != \"PyPy\""}
[[package]]
name = "pycparser"
@@ -2509,7 +2749,7 @@ files = [
{file = "pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992"},
{file = "pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29"},
]
-markers = {main = "python_version >= \"3.10\" and platform_python_implementation != \"PyPy\" and extra == \"gcp\" and implementation_name != \"PyPy\"", dev = "python_version >= \"3.10\" and (platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\") and implementation_name != \"PyPy\""}
+markers = {main = "python_version >= \"3.10\" and platform_python_implementation != \"PyPy\" and (extra == \"gcp\" or extra == \"deploy-gcp\" or extra == \"deploy-github\") and implementation_name != \"PyPy\"", dev = "python_version >= \"3.10\" and (platform_python_implementation != \"PyPy\" or implementation_name == \"pypy\") and implementation_name != \"PyPy\""}
[[package]]
name = "pydantic"
@@ -2679,6 +2919,26 @@ files = [
{file = "pyflakes-3.4.0.tar.gz", hash = "sha256:b24f96fafb7d2ab0ec5075b7350b3d2d2218eab42003821c06344973d3ea2f58"},
]
+[[package]]
+name = "pygithub"
+version = "2.9.0"
+description = "Use the full Github API v3"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-github\""
+files = [
+ {file = "pygithub-2.9.0-py3-none-any.whl", hash = "sha256:5e2b260ce327bffce9b00f447b65953ef7078ffe93e5a5425624a3075483927c"},
+ {file = "pygithub-2.9.0.tar.gz", hash = "sha256:a26abda1222febba31238682634cad11d8b966137ed6cc3c5e445b29a11cb0a4"},
+]
+
+[package.dependencies]
+pyjwt = {version = ">=2.4.0", extras = ["crypto"]}
+pynacl = ">=1.4.0"
+requests = ">=2.14.0"
+typing-extensions = ">=4.5.0"
+urllib3 = ">=1.26.0"
+
[[package]]
name = "pygments"
version = "2.20.0"
@@ -2694,6 +2954,29 @@ files = [
[package.extras]
windows-terminal = ["colorama (>=0.4.6)"]
+[[package]]
+name = "pyjwt"
+version = "2.12.1"
+description = "JSON Web Token implementation in Python"
+optional = true
+python-versions = ">=3.9"
+groups = ["main"]
+markers = "extra == \"deploy-github\""
+files = [
+ {file = "pyjwt-2.12.1-py3-none-any.whl", hash = "sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c"},
+ {file = "pyjwt-2.12.1.tar.gz", hash = "sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b"},
+]
+
+[package.dependencies]
+cryptography = {version = ">=3.4.0", optional = true, markers = "extra == \"crypto\""}
+typing_extensions = {version = ">=4.0", markers = "python_version < \"3.11\""}
+
+[package.extras]
+crypto = ["cryptography (>=3.4.0)"]
+dev = ["coverage[toml] (==7.10.7)", "cryptography (>=3.4.0)", "pre-commit", "pytest (>=8.4.2,<9.0.0)", "sphinx", "sphinx-rtd-theme", "zope.interface"]
+docs = ["sphinx", "sphinx-rtd-theme", "zope.interface"]
+tests = ["coverage[toml] (==7.10.7)", "pytest (>=8.4.2,<9.0.0)"]
+
[[package]]
name = "pylint"
version = "3.3.9"
@@ -2711,8 +2994,8 @@ astroid = ">=3.3.8,<=3.4.0.dev0"
colorama = {version = ">=0.4.5", markers = "sys_platform == \"win32\""}
dill = [
{version = ">=0.3.7", markers = "python_version >= \"3.12\""},
- {version = ">=0.2", markers = "python_version < \"3.11\""},
{version = ">=0.3.6", markers = "python_version == \"3.11\""},
+ {version = ">=0.2", markers = "python_version < \"3.11\""},
]
isort = ">=4.2.5,<5.13 || >5.13,<7"
mccabe = ">=0.6,<0.8"
@@ -2744,6 +3027,49 @@ pyyaml = "*"
[package.extras]
extra = ["pygments (>=2.19.1)"]
+[[package]]
+name = "pynacl"
+version = "1.6.2"
+description = "Python binding to the Networking and Cryptography (NaCl) library"
+optional = true
+python-versions = ">=3.8"
+groups = ["main"]
+markers = "extra == \"deploy-github\""
+files = [
+ {file = "pynacl-1.6.2-cp314-cp314t-macosx_10_10_universal2.whl", hash = "sha256:622d7b07cc5c02c666795792931b50c91f3ce3c2649762efb1ef0d5684c81594"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d071c6a9a4c94d79eb665db4ce5cedc537faf74f2355e4d502591d850d3913c0"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fe9847ca47d287af41e82be1dd5e23023d3c31a951da134121ab02e42ac218c9"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:04316d1fc625d860b6c162fff704eb8426b1a8bcd3abacea11142cbd99a6b574"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44081faff368d6c5553ccf55322ef2819abb40e25afaec7e740f159f74813634"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:a9f9932d8d2811ce1a8ffa79dcbdf3970e7355b5c8eb0c1a881a57e7f7d96e88"},
+ {file = "pynacl-1.6.2-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:bc4a36b28dd72fb4845e5d8f9760610588a96d5a51f01d84d8c6ff9849968c14"},
+ {file = "pynacl-1.6.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3bffb6d0f6becacb6526f8f42adfb5efb26337056ee0831fb9a7044d1a964444"},
+ {file = "pynacl-1.6.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2fef529ef3ee487ad8113d287a593fa26f48ee3620d92ecc6f1d09ea38e0709b"},
+ {file = "pynacl-1.6.2-cp314-cp314t-win32.whl", hash = "sha256:a84bf1c20339d06dc0c85d9aea9637a24f718f375d861b2668b2f9f96fa51145"},
+ {file = "pynacl-1.6.2-cp314-cp314t-win_amd64.whl", hash = "sha256:320ef68a41c87547c91a8b58903c9caa641ab01e8512ce291085b5fe2fcb7590"},
+ {file = "pynacl-1.6.2-cp314-cp314t-win_arm64.whl", hash = "sha256:d29bfe37e20e015a7d8b23cfc8bd6aa7909c92a1b8f41ee416bbb3e79ef182b2"},
+ {file = "pynacl-1.6.2-cp38-abi3-macosx_10_10_universal2.whl", hash = "sha256:c949ea47e4206af7c8f604b8278093b674f7c79ed0d4719cc836902bf4517465"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8845c0631c0be43abdd865511c41eab235e0be69c81dc66a50911594198679b0"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:22de65bb9010a725b0dac248f353bb072969c94fa8d6b1f34b87d7953cf7bbe4"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:46065496ab748469cdd999246d17e301b2c24ae2fdf739132e580a0e94c94a87"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8a66d6fb6ae7661c58995f9c6435bda2b1e68b54b598a6a10247bfcdadac996c"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:26bfcd00dcf2cf160f122186af731ae30ab120c18e8375684ec2670dccd28130"},
+ {file = "pynacl-1.6.2-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:c8a231e36ec2cab018c4ad4358c386e36eede0319a0c41fed24f840b1dac59f6"},
+ {file = "pynacl-1.6.2-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:68be3a09455743ff9505491220b64440ced8973fe930f270c8e07ccfa25b1f9e"},
+ {file = "pynacl-1.6.2-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8b097553b380236d51ed11356c953bf8ce36a29a3e596e934ecabe76c985a577"},
+ {file = "pynacl-1.6.2-cp38-abi3-win32.whl", hash = "sha256:5811c72b473b2f38f7e2a3dc4f8642e3a3e9b5e7317266e4ced1fba85cae41aa"},
+ {file = "pynacl-1.6.2-cp38-abi3-win_amd64.whl", hash = "sha256:62985f233210dee6548c223301b6c25440852e13d59a8b81490203c3227c5ba0"},
+ {file = "pynacl-1.6.2-cp38-abi3-win_arm64.whl", hash = "sha256:834a43af110f743a754448463e8fd61259cd4ab5bbedcf70f9dabad1d28a394c"},
+ {file = "pynacl-1.6.2.tar.gz", hash = "sha256:018494d6d696ae03c7e656e5e74cdfd8ea1326962cc401bcf018f1ed8436811c"},
+]
+
+[package.dependencies]
+cffi = {version = ">=2.0.0", markers = "platform_python_implementation != \"PyPy\" and python_version >= \"3.9\""}
+
+[package.extras]
+docs = ["sphinx (<7)", "sphinx_rtd_theme"]
+tests = ["hypothesis (>=3.27.0)", "pytest (>=7.4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"]
+
[[package]]
name = "pyparsing"
version = "3.3.2"
@@ -3614,14 +3940,14 @@ test = ["pytest", "pytest-cov"]
name = "zipp"
version = "3.23.0"
description = "Backport of pathlib-compatible object wrapper for zip files"
-optional = false
+optional = true
python-versions = ">=3.9"
-groups = ["dev", "docs"]
+groups = ["main", "dev", "docs"]
files = [
{file = "zipp-3.23.0-py3-none-any.whl", hash = "sha256:071652d6115ed432f5ce1d34c336c0adfd6a884660d1e9712a256d3d3bd4b14e"},
{file = "zipp-3.23.0.tar.gz", hash = "sha256:a07157588a12518c9d4034df3fbbee09c814741a33ff63c05fa29d26a2404166"},
]
-markers = {dev = "python_full_version < \"3.10.2\"", docs = "python_version < \"3.10\""}
+markers = {main = "extra == \"otel\"", dev = "python_full_version < \"3.10.2\"", docs = "python_version < \"3.10\""}
[package.extras]
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\""]
@@ -3633,10 +3959,14 @@ type = ["pytest-mypy"]
[extras]
aws = ["boto3"]
+deploy-aws = ["boto3"]
+deploy-gcp = ["google-auth", "google-cloud-artifact-registry", "google-cloud-build", "google-cloud-run", "google-cloud-service-usage", "google-cloud-storage"]
+deploy-github = ["PyGithub"]
gcp = ["google-cloud-storage"]
+otel = ["opentelemetry-api", "opentelemetry-sdk"]
scheduler = ["croniter"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.9.0"
-content-hash = "8bc802a04084146429175caab4cf9431867a4253da0c0d117727a3a969164ce0"
+content-hash = "87f99b43e3f745344716d16147410d299834c2a6c1ed4b8fbb8a27ce6d60953c"
diff --git a/pyproject.toml b/pyproject.toml
index ee0c2cc0..f56d7c60 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "dotflow"
-version = "0.14.1.dev2"
+version = "0.15.0.dev2"
authors = [
{ name="Fernando Celmer", email="email@fernandocelmer.com" },
]
@@ -56,13 +56,16 @@ Changelog = "https://dotflow-io.github.io/dotflow/nav/development/release-notes/
[project.optional-dependencies]
aws = ["boto3"]
gcp = ["google-cloud-storage"]
+deploy-aws = ["boto3"]
+deploy-gcp = ["google-cloud-run", "google-cloud-build", "google-cloud-artifact-registry", "google-cloud-service-usage", "google-cloud-storage", "google-auth"]
+deploy-github = ["PyGithub", "GitPython"]
scheduler = ["croniter"]
otel = ["opentelemetry-api", "opentelemetry-sdk"]
sentry = ["sentry-sdk"]
[tool.poetry]
name = "dotflow"
-version = "0.14.1.dev2"
+version = "0.15.0.dev2"
description = "🎲 Dotflow turns an idea into flow! Lightweight Python library for execution pipelines with retry, parallel, cron and async support."
authors = ["Fernando Celmer "]
readme = "README.md"
diff --git a/tests/cloud/__init__.py b/tests/cloud/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/aws/__init__.py b/tests/cloud/aws/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/aws/services/__init__.py b/tests/cloud/aws/services/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/aws/services/test_s3.py b/tests/cloud/aws/services/test_s3.py
new file mode 100644
index 00000000..0bf21697
--- /dev/null
+++ b/tests/cloud/aws/services/test_s3.py
@@ -0,0 +1,38 @@
+"""Test S3 service."""
+
+import unittest
+
+import boto3
+from moto import mock_aws
+
+from dotflow.cloud.aws.services.s3 import S3
+from dotflow.cloud.core import ObjectStorage
+
+BUCKET = "test-bucket"
+PREFIX = "dotflow/"
+REGION = "us-east-1"
+
+
+class TestS3(unittest.TestCase):
+ def setUp(self):
+ self.mock = mock_aws()
+ self.mock.start()
+ boto3.client("s3", region_name=REGION).create_bucket(Bucket=BUCKET)
+
+ def tearDown(self):
+ self.mock.stop()
+
+ def test_instance(self):
+ s3 = S3(bucket=BUCKET, prefix=PREFIX, region=REGION)
+ self.assertIsInstance(s3, ObjectStorage)
+
+ def test_write_and_read(self):
+ s3 = S3(bucket=BUCKET, prefix=PREFIX, region=REGION)
+ s3.write("key1", [{"a": 1}])
+ result = s3.read("key1")
+ self.assertEqual(result, [{"a": 1}])
+
+ def test_read_nonexistent(self):
+ s3 = S3(bucket=BUCKET, prefix=PREFIX, region=REGION)
+ result = s3.read("missing")
+ self.assertEqual(result, [])
diff --git a/tests/cloud/aws/test_ecs_deployer.py b/tests/cloud/aws/test_ecs_deployer.py
new file mode 100644
index 00000000..ca066bf5
--- /dev/null
+++ b/tests/cloud/aws/test_ecs_deployer.py
@@ -0,0 +1,73 @@
+"""Test ECSDeployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.aws.ecs_deployer import ECSDeployer
+from dotflow.cloud.core import Deployer
+
+
+class TestECSDeployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(ECSDeployer, "__init__", return_value=None):
+ deployer = ECSDeployer()
+ deployer._region = "us-east-1"
+ deployer._account_id = "123456789012"
+ deployer._ecs = MagicMock()
+ deployer._ecr = MagicMock()
+ deployer._iam = MagicMock()
+ deployer._logs = MagicMock()
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_setup_creates_role_logs_cluster(self):
+ deployer = self._make_deployer()
+ deployer._ecs.describe_clusters.return_value = {"clusters": []}
+
+ deployer.setup("test")
+
+ deployer._iam.ensure_ecs_execution_role.assert_called_once()
+ deployer._logs.ensure_log_group.assert_called_once_with("/ecs/test")
+ deployer._ecs.create_cluster.assert_called_once()
+
+ def test_deploy_with_task_definition_file(self):
+ deployer = self._make_deployer()
+ deployer._ecs.describe_clusters.return_value = {"clusters": []}
+
+ with patch("dotflow.cloud.aws.ecs_deployer.Path") as mock_path:
+ mock_path.cwd.return_value.__truediv__ = MagicMock(
+ return_value=MagicMock(
+ exists=MagicMock(return_value=True),
+ read_text=MagicMock(return_value='{"family": "test"}'),
+ )
+ )
+ deployer.deploy("test")
+
+ deployer._ecr.push.assert_called_once_with("test")
+ deployer._ecs.register_task_definition.assert_called_once()
+
+ def test_deploy_creates_task_definition(self):
+ deployer = self._make_deployer()
+ deployer._ecs.describe_clusters.return_value = {"clusters": []}
+ deployer._iam.ensure_ecs_execution_role.return_value = "arn:role"
+
+ with patch("dotflow.cloud.aws.ecs_deployer.Path") as mock_path:
+ mock_path.cwd.return_value.__truediv__ = MagicMock(
+ return_value=MagicMock(exists=MagicMock(return_value=False))
+ )
+ deployer.deploy("test")
+
+ deployer._ecs.register_task_definition.assert_called_once()
+
+ def test_ensure_cluster_skips_if_active(self):
+ deployer = self._make_deployer()
+ deployer._ecs.describe_clusters.return_value = {
+ "clusters": [{"status": "ACTIVE"}]
+ }
+
+ deployer._ensure_cluster("test")
+
+ deployer._ecs.create_cluster.assert_not_called()
diff --git a/tests/cloud/aws/test_lambda_api_deployer.py b/tests/cloud/aws/test_lambda_api_deployer.py
new file mode 100644
index 00000000..a885ca41
--- /dev/null
+++ b/tests/cloud/aws/test_lambda_api_deployer.py
@@ -0,0 +1,64 @@
+"""Test LambdaApiDeployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.aws.lambda_api_deployer import LambdaApiDeployer
+from dotflow.cloud.core import Deployer
+
+
+class _ResourceConflict(Exception):
+ pass
+
+
+class TestLambdaApiDeployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(LambdaApiDeployer, "__init__", return_value=None):
+ deployer = LambdaApiDeployer()
+ deployer._region = "us-east-1"
+ deployer._account_id = "123456789012"
+ deployer._name = "test"
+ deployer._boto3 = MagicMock()
+ deployer._lambda = MagicMock()
+ deployer._lambda.exceptions.ResourceConflictException = (
+ _ResourceConflict
+ )
+ deployer._ecr = MagicMock()
+ deployer._iam = MagicMock()
+ deployer._logs = MagicMock()
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_deploy_creates_api(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ mock_apigw = MagicMock()
+ mock_apigw.get_apis.return_value = {"Items": []}
+ mock_apigw.create_api.return_value = {"ApiId": "abc123"}
+ mock_apigw.create_integration.return_value = {"IntegrationId": "int1"}
+ deployer._boto3.client.return_value = mock_apigw
+
+ deployer.deploy("test")
+
+ mock_apigw.create_api.assert_called_once()
+ mock_apigw.create_route.assert_called_once()
+
+ def test_deploy_skips_existing_api(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ mock_apigw = MagicMock()
+ mock_apigw.get_apis.return_value = {
+ "Items": [{"Name": "test-api", "ApiId": "existing"}]
+ }
+ deployer._boto3.client.return_value = mock_apigw
+
+ deployer.deploy("test")
+
+ mock_apigw.create_api.assert_not_called()
diff --git a/tests/cloud/aws/test_lambda_deployer.py b/tests/cloud/aws/test_lambda_deployer.py
new file mode 100644
index 00000000..92b444b6
--- /dev/null
+++ b/tests/cloud/aws/test_lambda_deployer.py
@@ -0,0 +1,79 @@
+"""Test LambdaDeployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.aws.lambda_deployer import LambdaDeployer
+from dotflow.cloud.core import Deployer
+
+
+class _ResourceNotFound(Exception):
+ pass
+
+
+class _ResourceConflict(Exception):
+ pass
+
+
+class TestLambdaDeployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(LambdaDeployer, "__init__", return_value=None):
+ deployer = LambdaDeployer()
+ deployer._region = "us-east-1"
+ deployer._account_id = "123456789012"
+ deployer._name = "test"
+ deployer._boto3 = MagicMock()
+ deployer._lambda = MagicMock()
+ deployer._lambda.exceptions.ResourceNotFoundException = (
+ _ResourceNotFound
+ )
+ deployer._lambda.exceptions.ResourceConflictException = (
+ _ResourceConflict
+ )
+ deployer._ecr = MagicMock()
+ deployer._iam = MagicMock()
+ deployer._logs = MagicMock()
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_deploy_creates_function(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+ deployer._lambda.get_function.side_effect = _ResourceNotFound()
+
+ deployer.deploy("test")
+
+ deployer._lambda.create_function.assert_called_once()
+
+ def test_deploy_updates_existing(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ deployer.deploy("test")
+
+ deployer._lambda.update_function_code.assert_called_once()
+
+ def test_deploy_with_schedule(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+ mock_events = MagicMock()
+ deployer._boto3.client.return_value = mock_events
+
+ deployer.deploy("test", schedule="rate(6 hours)")
+
+ mock_events.put_rule.assert_called_once()
+
+ def test_deploy_without_schedule(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ deployer.deploy("test")
+
+ deployer._boto3.client.assert_not_called()
diff --git a/tests/cloud/aws/test_lambda_s3_deployer.py b/tests/cloud/aws/test_lambda_s3_deployer.py
new file mode 100644
index 00000000..25b4d567
--- /dev/null
+++ b/tests/cloud/aws/test_lambda_s3_deployer.py
@@ -0,0 +1,51 @@
+"""Test LambdaS3Deployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.aws.lambda_s3_deployer import LambdaS3Deployer
+from dotflow.cloud.core import Deployer
+
+
+class _ResourceConflict(Exception):
+ pass
+
+
+class _BucketExists(Exception):
+ pass
+
+
+class TestLambdaS3Deployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(LambdaS3Deployer, "__init__", return_value=None):
+ deployer = LambdaS3Deployer()
+ deployer._region = "us-east-1"
+ deployer._account_id = "123456789012"
+ deployer._name = "test"
+ deployer._boto3 = MagicMock()
+ deployer._lambda = MagicMock()
+ deployer._lambda.exceptions.ResourceConflictException = (
+ _ResourceConflict
+ )
+ deployer._ecr = MagicMock()
+ deployer._iam = MagicMock()
+ deployer._logs = MagicMock()
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_deploy_creates_bucket_and_notification(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ mock_s3 = MagicMock()
+ mock_s3.exceptions.BucketAlreadyOwnedByYou = _BucketExists
+ deployer._boto3.client.return_value = mock_s3
+
+ deployer.deploy("test")
+
+ mock_s3.create_bucket.assert_called_once()
+ mock_s3.put_bucket_notification_configuration.assert_called_once()
diff --git a/tests/cloud/aws/test_lambda_sqs_deployer.py b/tests/cloud/aws/test_lambda_sqs_deployer.py
new file mode 100644
index 00000000..649c85e5
--- /dev/null
+++ b/tests/cloud/aws/test_lambda_sqs_deployer.py
@@ -0,0 +1,73 @@
+"""Test LambdaSQSDeployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.aws.lambda_sqs_deployer import LambdaSQSDeployer
+from dotflow.cloud.core import Deployer
+
+
+class _QueueExists(Exception):
+ pass
+
+
+class TestLambdaSQSDeployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(LambdaSQSDeployer, "__init__", return_value=None):
+ deployer = LambdaSQSDeployer()
+ deployer._region = "us-east-1"
+ deployer._account_id = "123456789012"
+ deployer._name = "test"
+ deployer._boto3 = MagicMock()
+ deployer._lambda = MagicMock()
+ deployer._ecr = MagicMock()
+ deployer._iam = MagicMock()
+ deployer._logs = MagicMock()
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_deploy_creates_queue_and_mapping(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ mock_sqs = MagicMock()
+ mock_sqs.create_queue.return_value = {
+ "QueueUrl": "https://sqs/test-queue"
+ }
+ mock_sqs.get_queue_attributes.return_value = {
+ "Attributes": {"QueueArn": "arn:aws:sqs:us-east-1:123:test-queue"}
+ }
+ deployer._boto3.client.return_value = mock_sqs
+ deployer._lambda.list_event_source_mappings.return_value = {
+ "EventSourceMappings": []
+ }
+
+ deployer.deploy("test")
+
+ mock_sqs.create_queue.assert_called_once()
+ deployer._lambda.create_event_source_mapping.assert_called_once()
+
+ def test_deploy_skips_existing_mapping(self):
+ deployer = self._make_deployer()
+ deployer._ecr.push.return_value = "uri"
+ deployer._iam.ensure_lambda_role.return_value = "arn"
+
+ mock_sqs = MagicMock()
+ mock_sqs.create_queue.return_value = {
+ "QueueUrl": "https://sqs/test-queue"
+ }
+ mock_sqs.get_queue_attributes.return_value = {
+ "Attributes": {"QueueArn": "arn:aws:sqs:us-east-1:123:test-queue"}
+ }
+ deployer._boto3.client.return_value = mock_sqs
+ deployer._lambda.list_event_source_mappings.return_value = {
+ "EventSourceMappings": [{"UUID": "existing"}]
+ }
+
+ deployer.deploy("test")
+
+ deployer._lambda.create_event_source_mapping.assert_not_called()
diff --git a/tests/cloud/gcp/__init__.py b/tests/cloud/gcp/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/gcp/services/__init__.py b/tests/cloud/gcp/services/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/gcp/services/test_gcs.py b/tests/cloud/gcp/services/test_gcs.py
new file mode 100644
index 00000000..9252578f
--- /dev/null
+++ b/tests/cloud/gcp/services/test_gcs.py
@@ -0,0 +1,56 @@
+"""Test GCS service."""
+
+import sys
+import unittest
+from json import dumps
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.core import ObjectStorage
+
+mock_gcloud = MagicMock()
+sys.modules["google"] = MagicMock()
+sys.modules["google.cloud"] = MagicMock()
+sys.modules["google.cloud.storage"] = mock_gcloud
+sys.modules["google.api_core"] = MagicMock()
+sys.modules["google.api_core.exceptions"] = MagicMock()
+
+from dotflow.cloud.gcp.services.gcs import GCS # noqa: E402, I001
+
+
+class TestGCS(unittest.TestCase):
+ def setUp(self):
+ with patch.object(GCS, "__init__", return_value=None):
+ self.gcs = GCS()
+ self.mock_bucket = MagicMock()
+ self.gcs._bucket = self.mock_bucket
+ self.gcs._not_found = Exception
+ self.gcs.prefix = "dotflow/"
+
+ def test_instance(self):
+ self.assertIsInstance(self.gcs, ObjectStorage)
+
+ def test_write(self):
+ blob = MagicMock()
+ self.mock_bucket.blob.return_value = blob
+
+ self.gcs.write("key1", [{"a": 1}])
+
+ blob.upload_from_string.assert_called_once()
+
+ def test_read(self):
+ blob = MagicMock()
+ blob.download_as_text.return_value = dumps([{"a": 1}])
+ self.mock_bucket.blob.return_value = blob
+
+ result = self.gcs.read("key1")
+
+ self.assertEqual(result, [{"a": 1}])
+
+ def test_read_nonexistent(self):
+ blob = MagicMock()
+ blob.download_as_text.side_effect = Exception("not found")
+ self.mock_bucket.blob.return_value = blob
+
+ result = self.gcs.read("missing")
+
+ self.assertEqual(result, [])
diff --git a/tests/cloud/github/__init__.py b/tests/cloud/github/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/cloud/github/test_actions_deployer.py b/tests/cloud/github/test_actions_deployer.py
new file mode 100644
index 00000000..21de3284
--- /dev/null
+++ b/tests/cloud/github/test_actions_deployer.py
@@ -0,0 +1,79 @@
+"""Test ActionsDeployer."""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from dotflow.cloud.core import Deployer
+from dotflow.cloud.github.actions_deployer import ActionsDeployer
+
+
+class _GithubException(Exception):
+ pass
+
+
+class TestActionsDeployer(unittest.TestCase):
+ def _make_deployer(self):
+ with patch.object(ActionsDeployer, "__init__", return_value=None):
+ deployer = ActionsDeployer()
+ deployer._github = MagicMock()
+ deployer._GithubException = _GithubException
+ deployer._user = MagicMock()
+ deployer._token = "fake-token"
+ return deployer
+
+ def test_instance(self):
+ deployer = self._make_deployer()
+ self.assertIsInstance(deployer, Deployer)
+
+ def test_create_new_repo(self):
+ deployer = self._make_deployer()
+ deployer._user.get_repo.side_effect = _GithubException()
+ mock_repo = MagicMock()
+ deployer._user.create_repo.return_value = mock_repo
+
+ repo = deployer._create_or_get_repo("test")
+
+ deployer._user.create_repo.assert_called_once()
+ self.assertEqual(repo, mock_repo)
+
+ def test_use_existing_repo(self):
+ deployer = self._make_deployer()
+ mock_repo = MagicMock()
+ deployer._user.get_repo.return_value = mock_repo
+
+ repo = deployer._create_or_get_repo("test")
+
+ deployer._user.create_repo.assert_not_called()
+ self.assertEqual(repo, mock_repo)
+
+ def test_create_file(self):
+ deployer = self._make_deployer()
+ mock_repo = MagicMock()
+ mock_repo.get_contents.side_effect = _GithubException()
+
+ deployer._create_or_update_file(mock_repo, "test.py", "print('hi')")
+
+ mock_repo.create_file.assert_called_once()
+
+ def test_update_existing_file(self):
+ deployer = self._make_deployer()
+ mock_repo = MagicMock()
+ mock_existing = MagicMock()
+ mock_existing.sha = "abc123"
+ mock_repo.get_contents.return_value = mock_existing
+
+ deployer._create_or_update_file(mock_repo, "test.py", "print('hi')")
+
+ mock_repo.update_file.assert_called_once()
+
+ def test_deploy_creates_repo_and_pushes(self):
+ deployer = self._make_deployer()
+ mock_repo = MagicMock()
+ mock_repo.html_url = "https://github.com/user/test"
+ deployer._user.get_repo.return_value = mock_repo
+ mock_repo.get_contents.side_effect = _GithubException()
+
+ with patch.object(deployer, "_push_files") as mock_push:
+ deployer.deploy("test")
+
+ mock_push.assert_called_once_with(mock_repo)
diff --git a/tests/providers/test_storage_gcs.py b/tests/providers/test_storage_gcs.py
index 1cb899db..f273b0dc 100644
--- a/tests/providers/test_storage_gcs.py
+++ b/tests/providers/test_storage_gcs.py
@@ -30,13 +30,34 @@ def setUp(self):
from dotflow.providers.storage_gcs import StorageGCS
self.storage = StorageGCS()
- self.storage.client = self.mock_client
- self.storage.bucket_obj = self.mock_bucket
- self.storage.prefix = PREFIX
- self.storage._not_found = Exception
+
+ mock_gcs = MagicMock()
+ mock_gcs._bucket = self.mock_bucket
+ mock_gcs._not_found = Exception
+ mock_gcs.prefix = PREFIX
+ mock_gcs.read = self._mock_read
+ mock_gcs.write = self._mock_write
+ self.storage._gcs = mock_gcs
+ self._written = {}
+
+ def _mock_read(self, key):
+ blob = self.mock_bucket.blob(f"{PREFIX}{key}")
+ try:
+ data = blob.download_as_text()
+ from json import loads
+
+ return loads(data)
+ except Exception:
+ return []
+
+ def _mock_write(self, key, data):
+ blob = self.mock_bucket.blob(f"{PREFIX}{key}")
+ from json import dumps
+
+ blob.upload_from_string(dumps(data), content_type="application/json")
def test_storage_gcs_instance(self):
- self.assertEqual(self.storage.prefix, PREFIX)
+ self.assertEqual(self.storage._gcs.prefix, PREFIX)
def test_post(self):
expected_value = {"foo": "bar"}
diff --git a/tests/providers/test_storage_s3.py b/tests/providers/test_storage_s3.py
index e29c5e4e..a11eb6e6 100644
--- a/tests/providers/test_storage_s3.py
+++ b/tests/providers/test_storage_s3.py
@@ -30,8 +30,8 @@ def tearDown(self):
def test_storage_s3_instance(self):
storage = StorageS3(bucket=BUCKET, prefix=PREFIX, region=REGION)
- self.assertEqual(storage.bucket, BUCKET)
- self.assertEqual(storage.prefix, PREFIX)
+ self.assertEqual(storage._s3.bucket, BUCKET)
+ self.assertEqual(storage._s3.prefix, PREFIX)
def test_post(self):
expected_value = {"foo": "bar"}
**Docker** | `docker compose up` |
+| 
**AWS Lambda** | `dotflow deploy` |
+| 
**AWS ECS Fargate** | `dotflow deploy` |
+| 
**Google Cloud Run** | `gcloud run deploy` |
+| 
**Kubernetes** | `kubectl apply` |
+| 
**GitHub Actions** | `dotflow deploy` |
+
+> [See all 34+ platforms →](https://dotflow-io.github.io/dotflow/nav/cloud/)
+
+## Optional extras
+
+```bash
+pip install dotflow[aws] # S3 storage
+pip install dotflow[gcp] # Google Cloud Storage
+pip install dotflow[scheduler] # Cron scheduler
+pip install dotflow[otel] # OpenTelemetry
+pip install dotflow[sentry] # Sentry error tracking
+pip install dotflow[deploy-aws] # AWS deploy (Lambda, ECS)
+pip install dotflow[deploy-gcp] # GCP deploy (Cloud Run)
+pip install dotflow[deploy-github] # GitHub Actions deploy
+```
+
+> **[Read the full documentation →](https://dotflow-io.github.io/dotflow/)**
+
+## Documentation
+
+| Section | Description |
+|---------|-------------|
+| [Concepts](https://dotflow-io.github.io/dotflow/nav/concepts/concept-workflow-and-tasks/) | Workflows, tasks, context, providers, process modes |
+| [How-to Guides](https://dotflow-io.github.io/dotflow/nav/how-to/) | Step-by-step tutorials for workflows, tasks, and CLI |
+| [Cloud Deployment](https://dotflow-io.github.io/dotflow/nav/cloud/) | Deploy to AWS, GCP, Kubernetes, Docker, GitHub Actions |
+| [Integrations](https://dotflow-io.github.io/dotflow/nav/integrations/) | OpenTelemetry, Sentry, Telegram, Discord, S3, GCS |
+| [Examples](https://dotflow-io.github.io/dotflow/nav/examples/) | Real-world pipelines: ETL, health checks, async, scheduler |
+| [Reference](https://dotflow-io.github.io/dotflow/nav/reference/dotflow/) | API reference for all classes and providers |
+| [Custom Providers](https://dotflow-io.github.io/dotflow/nav/development/custom-providers/) | Build your own storage, notify, log, tracer, or metrics provider |
+
## Features
-### Execution Modes
+