|
| 1 | +# Design: Replace aioboto3 with aiobotocore |
| 2 | + |
| 3 | +**Date:** 2026-06-22 |
| 4 | +**Status:** Approved (design) |
| 5 | +**Topic:** Drop the unmaintained `aioboto3` dependency in favor of calling `aiobotocore` directly. |
| 6 | + |
| 7 | +## Problem & Rationale |
| 8 | + |
| 9 | +`aioboto3` is effectively unmaintained. It is a thin wrapper over `aiobotocore` |
| 10 | +(the actively-maintained async layer that wraps `botocore`); its main value-add |
| 11 | +over `aiobotocore` is the high-level **resource** API (`session.resource(...)`, |
| 12 | +`Table` objects with automatic type marshalling). |
| 13 | + |
| 14 | +The zae-limiter async data path uses **only the low-level client API** |
| 15 | +(`session.client("dynamodb")` with explicitly-typed attribute values such as |
| 16 | +`{"S": ...}` / `{"N": ...}`). It never uses the resource API. Therefore dropping |
| 17 | +`aioboto3` and creating clients directly from an `aiobotocore` session costs us |
| 18 | +nothing functionally, removes an unmaintained dependency, and brings us one layer |
| 19 | +closer to `botocore`. |
| 20 | + |
| 21 | +## Scope |
| 22 | + |
| 23 | +### In scope — the only modules importing `aioboto3` |
| 24 | + |
| 25 | +| File | AWS clients created | |
| 26 | +|------|---------------------| |
| 27 | +| `src/zae_limiter/repository.py` | `dynamodb`, `sts` | |
| 28 | +| `src/zae_limiter/infra/discovery.py` | `cloudformation`, `resourcegroupstaggingapi` | |
| 29 | +| `src/zae_limiter/infra/stack_manager.py` | `cloudformation`, `lambda` | |
| 30 | +| `scripts/generate_sync.py` | AST transformer (async → sync codegen) | |
| 31 | +| `pyproject.toml` | runtime deps + mypy overrides | |
| 32 | +| Tests patching `aioboto3.Session` | `tests/unit/test_limiter.py`, `tests/unit/test_stack_manager.py`, and generated `tests/unit/test_sync_*.py` (~40 patch sites) | |
| 33 | + |
| 34 | +### Out of scope — already pure `boto3` (sync) |
| 35 | + |
| 36 | +- `src/zae_limiter_aggregator/` (Lambda aggregator) |
| 37 | +- `src/zae_limiter_provisioner/` (Lambda provisioner) |
| 38 | +- `src/zae_limiter/loadtest/` |
| 39 | +- `src/zae_limiter/limits_cli.py` |
| 40 | +- All generated `sync_*.py` files (regenerated from async source, never hand-edited) |
| 41 | + |
| 42 | +The aggregator's `boto3.resource("dynamodb")` usage is sync-only and out of scope. |
| 43 | + |
| 44 | +## Approach Decision |
| 45 | + |
| 46 | +**Chosen: direct call-site replacement** (not an internal factory/abstraction |
| 47 | +module). The code already uses the low-level client uniformly, so the change is |
| 48 | +faithful to "use aiobotocore directly" and keeps the diff minimal. The cost — |
| 49 | +adapting the sync-gen transformer and the ~40 test patch targets to the new call |
| 50 | +shape — is accepted. |
| 51 | + |
| 52 | +**Process decision:** open a tracking GitHub issue (via `/issue`); **no ADR** |
| 53 | +(treated as a maintenance/dependency change). Implement on branch |
| 54 | +`refactor/aiobotocore-migration`; open a draft PR (via `/pr`) referencing the issue. |
| 55 | + |
| 56 | +## Detailed Design |
| 57 | + |
| 58 | +### 1. Call-site change (async source) |
| 59 | + |
| 60 | +Two substitutions; the `await`, `.__aenter__()`, `async with`, and `__aexit__` |
| 61 | +lifecycle are unchanged because `aioboto3` delegates the client path straight to |
| 62 | +`aiobotocore`: |
| 63 | + |
| 64 | +```python |
| 65 | +# before |
| 66 | +import aioboto3 |
| 67 | +self._session = aioboto3.Session() |
| 68 | +self._client = await self._session.client( |
| 69 | + "dynamodb", region_name=..., endpoint_url=... |
| 70 | +).__aenter__() |
| 71 | + |
| 72 | +# after |
| 73 | +from aiobotocore.session import get_session |
| 74 | +self._session = get_session() |
| 75 | +self._client = await self._session.create_client( |
| 76 | + "dynamodb", region_name=..., endpoint_url=... |
| 77 | +).__aenter__() |
| 78 | +``` |
| 79 | + |
| 80 | +- `close()` (`await self._client.__aexit__(None, None, None)`) needs **no change**: |
| 81 | + the object returned by `__aenter__()` supports `__aexit__` in both libraries. |
| 82 | +- `async with self._session.client(...) as c:` becomes |
| 83 | + `async with self._session.create_client(...) as c:`. |
| 84 | +- Credential/region/endpoint resolution is identical (same `botocore` underneath). |
| 85 | + `get_session()` takes no args; all config flows through `create_client(...)` |
| 86 | + exactly as it flowed through `aioboto3`'s `.client(...)` today. |
| 87 | + |
| 88 | +### 2. Sync-generator transformer (`scripts/generate_sync.py`) — highest risk |
| 89 | + |
| 90 | +The transformer currently rewrites `aioboto3 → boto3` and unwraps |
| 91 | +`async with session.client(...)`. Direct replacement requires three new/updated |
| 92 | +rules so generated sync remains idiomatic `boto3`: |
| 93 | + |
| 94 | +| Async (aiobotocore) | Generated sync (boto3) | Transformer rule | |
| 95 | +|---|---|---| |
| 96 | +| `from aiobotocore.session import get_session` | `import boto3` | import rewrite (handle the `from … import` form) | |
| 97 | +| `get_session()` | `boto3.Session()` | call rewrite (Name `get_session()` → `boto3.Session()`) | |
| 98 | +| `.create_client(...)` | `.client(...)` | attribute rename `create_client → client`, applied **before** the existing `.client` context-manager unwrap so that logic continues to fire unchanged | |
| 99 | + |
| 100 | +The `IMPORT_MODULE_REWRITES = {"aioboto3": "boto3"}` entry is replaced by the |
| 101 | +above rules. |
| 102 | + |
| 103 | +**Verification of the transformer:** regenerate the full sync tree and diff |
| 104 | +against the current generated files. The generated `sync_repository.py`, |
| 105 | +`sync_discovery.py`, `sync_stack_manager.py`, etc. must come out **functionally |
| 106 | +identical to today** — only the async source changed, so the sync output should |
| 107 | +be unchanged modulo the session-construction lines (`boto3.Session()` / |
| 108 | +`.client(...)`), which must match the current output exactly. This is the primary |
| 109 | +checkpoint. |
| 110 | + |
| 111 | +### 3. Dependencies & typing (`pyproject.toml`) |
| 112 | + |
| 113 | +- Runtime: replace `aioboto3>=12.0.0` with `aiobotocore` at a floor whose |
| 114 | + pinned `botocore` range is compatible with the existing `boto3>=1.34.0` |
| 115 | + (e.g. `aiobotocore>=2.13.0`); the exact floor is confirmed at implementation |
| 116 | + time by resolving the env and checking the installed `botocore` matches. |
| 117 | + Keep `boto3>=1.34.0` (used by sync code, aggregator, provisioner, loadtest). |
| 118 | +- Typing: `types-aiobotocore[dynamodb,s3]` is **already** a dev dependency — no |
| 119 | + change needed there. |
| 120 | +- mypy overrides: replace the `aioboto3` / `aioboto3.*` `ignore_missing_imports` |
| 121 | + entries. First attempt: remove them entirely (since `types-aiobotocore` is |
| 122 | + installed and ships types). If the base `aiobotocore` / `aiobotocore.session` |
| 123 | + import still errors under mypy, add `aiobotocore` / `aiobotocore.*` overrides |
| 124 | + instead. |
| 125 | + |
| 126 | +### 4. Tests |
| 127 | + |
| 128 | +- Replace `patch("….aioboto3.Session")` with a patch of the new seam in each |
| 129 | + module — e.g. `patch("zae_limiter.repository.get_session")`, |
| 130 | + `patch("zae_limiter.infra.stack_manager.get_session")` — returning a mock |
| 131 | + session whose `.create_client(...)` returns the mock client (mirroring how the |
| 132 | + current mocks wire `.client(...)`). |
| 133 | +- The Lambda-packaging tests that assert `aioboto3/` is absent from the built zip |
| 134 | + (`tests/integration/test_lambda_builder.py`, `tests/unit/test_lambda_builder.py`) |
| 135 | + update to assert `aiobotocore/` is absent (the aggregator zip should contain |
| 136 | + neither). |
| 137 | +- Generated `tests/unit/test_sync_*.py` are **regenerated**, not hand-edited; the |
| 138 | + async test sources (`test_limiter.py`, `test_stack_manager.py`) are the source |
| 139 | + of truth and must be updated so their sync counterparts regenerate correctly. |
| 140 | + |
| 141 | +## Verification Gates (run in order) |
| 142 | + |
| 143 | +1. `hatch run generate-sync` — clean regeneration, **no drift** vs committed sync files (beyond intended session-line changes). |
| 144 | +2. `pre-commit run --all-files` — includes the "generated code up-to-date" hook, ruff, and cfn-lint. |
| 145 | +3. `uv run mypy src/zae_limiter` — type check passes. |
| 146 | +4. `uv run pytest tests/unit/` — unit suite (incl. regenerated `test_sync_*`). |
| 147 | +5. Integration / e2e against LocalStack (`zae-limiter local up`; `pytest -m integration`, `-m e2e`). |
| 148 | + |
| 149 | +## Risks & Mitigations |
| 150 | + |
| 151 | +| Risk | Mitigation | |
| 152 | +|------|------------| |
| 153 | +| Transformer produces non-idiomatic or broken sync code | Diff regenerated sync against current output; treat any non-session-line diff as a bug. Checkpoint before proceeding. | |
| 154 | +| Hidden resource-API usage | Confirmed via grep: only `session.client(...)` is used in async code; no `session.resource(...)` / `Table(...)`. | |
| 155 | +| Subtle credential/endpoint behavior change | Same `botocore` resolution path; LocalStack e2e exercises `endpoint_url`, real-AWS e2e exercises default credential chain. | |
| 156 | +| mypy regressions from stub differences | `types-aiobotocore` already installed; gate #3 catches issues. | |
| 157 | + |
| 158 | +## Out of Scope / Non-Goals |
| 159 | + |
| 160 | +- No migration to the resource API or any change to the DynamoDB |
| 161 | + serialization/marshalling code. |
| 162 | +- No changes to sync `boto3` usage in the aggregator, provisioner, loadtest, or CLI. |
| 163 | +- No ADR (per process decision above). |
0 commit comments