diff --git a/README.md b/README.md index 772b469ddd..0660360bd1 100644 --- a/README.md +++ b/README.md @@ -362,7 +362,7 @@ docker run -d --name floci \ | `FLOCI_SERVICES_RDS_DEFAULT_MYSQL_IMAGE` | `mysql:8.0` | | `FLOCI_SERVICES_RDS_DEFAULT_MARIADB_IMAGE` | `mariadb:11` | | `FLOCI_SERVICES_MSK_DEFAULT_IMAGE` | `redpandadata/redpanda:latest` | -| `FLOCI_SERVICES_OPENSEARCH_DEFAULT_IMAGE` | `opensearchproject/opensearch:2` | +| `FLOCI_SERVICES_OPENSEARCH_DEFAULT_IMAGE` | *(unset — images resolve per requested `EngineVersion`)* | | `FLOCI_SERVICES_KINESIS_ANALYTICS_DEFAULT_IMAGE` | _(unset; chosen per RuntimeEnvironment)_ | | `FLOCI_SERVICES_NEPTUNE_DEFAULT_IMAGE` | `tinkerpop/gremlin-server:3.7.3` | | `FLOCI_SERVICES_NEPTUNE_DEFAULT_NEO4J_IMAGE` | `neo4j:5-community` | diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md index da64eddbe8..4c781063da 100644 --- a/docs/configuration/environment-variables.md +++ b/docs/configuration/environment-variables.md @@ -360,7 +360,7 @@ These services spawn Docker containers. They require access to the Docker socket |---|---|---| | `FLOCI_SERVICES_OPENSEARCH_ENABLED` | `true` | Enable the OpenSearch service | | `FLOCI_SERVICES_OPENSEARCH_MOCK` | `false` | When `true`, domains are created instantly without a real container (API only) | -| `FLOCI_SERVICES_OPENSEARCH_DEFAULT_IMAGE` | `opensearchproject/opensearch:2` | Docker image for OpenSearch domains | +| `FLOCI_SERVICES_OPENSEARCH_DEFAULT_IMAGE` | *(unset)* | Optional fixed Docker image for every OpenSearch domain; when unset, images resolve per requested `EngineVersion` | | `FLOCI_SERVICES_OPENSEARCH_PROXY_BASE_PORT` | `9400` | First port in the OpenSearch proxy range | | `FLOCI_SERVICES_OPENSEARCH_PROXY_MAX_PORT` | `9499` | Last port in the OpenSearch proxy range | | `FLOCI_SERVICES_OPENSEARCH_KEEP_RUNNING_ON_SHUTDOWN` | `false` | Keep OpenSearch containers running when Floci stops | diff --git a/docs/configuration/ports.md b/docs/configuration/ports.md index 3f94a0e2d3..5a07a8a61d 100644 --- a/docs/configuration/ports.md +++ b/docs/configuration/ports.md @@ -7,8 +7,11 @@ | `4566` | HTTP | All AWS API calls (every service) | Yes | | `5100–5199` | HTTP | ECR Registry sidecar — bound directly by the `registry:2` container | **No** (see note) | | `6379–6399` | TCP | ElastiCache Redis proxy (inside Floci) | Yes | +| `6400–6419` | TCP | MemoryDB proxy (inside Floci) | Yes | | `6500–6599` | HTTPS | EKS k3s API server — bound directly by each k3s container | **No** | | `7001–7099` | TCP | RDS proxy (inside Floci) | Yes | +| `8182–8282` | TCP | Neptune Gremlin proxy (inside Floci) | Yes | +| `8700–8799` | HTTP | MWAA Airflow webserver proxy (inside Floci) | Yes | | `9400–9499` | HTTP | OpenSearch data-plane — bound directly by each OpenSearch container | **No** | | `12000–12499` | HTTP | Lambda Runtime API (internal, Docker-network only) | **No** | @@ -16,9 +19,9 @@ There are two distinct patterns Floci uses to expose container ports: -### Proxy-in-Floci (ElastiCache, RDS) +### Proxy-in-Floci (ElastiCache, MemoryDB, RDS, Neptune, MWAA) -Floci runs a **TCP proxy process inside its own container**. The proxy listens on the host port and forwards traffic to the backend container. +Floci runs a **proxy process inside its own container**. The proxy listens on the host port and forwards traffic to the backend container. ``` host:6379 → [docker-compose ports mapping] → Floci container:6379 → Redis container:6379 @@ -147,7 +150,7 @@ host:5100 ←── floci-ecr-registry (registry:2 container, started by Floci ## Exposing Ports in Docker Compose -Only the proxy-based services (ElastiCache and RDS) need port mappings in `docker-compose.yml`. Direct-binding services (ECR, EKS, OpenSearch) bind their ports on the host automatically via Docker: +Only the proxy-based services (ElastiCache, MemoryDB, RDS, Neptune, MWAA) need port mappings in `docker-compose.yml`. Direct-binding services (ECR, EKS, OpenSearch) bind their ports on the host automatically via Docker: ```yaml services: @@ -156,7 +159,10 @@ services: ports: - "4566:4566" # All AWS API calls - "6379-6399:6379-6399" # ElastiCache / Redis proxy (proxy in Floci) + - "6400-6419:6400-6419" # MemoryDB proxy (proxy in Floci) - "7001-7099:7001-7099" # RDS proxy (proxy in Floci) + - "8182-8282:8182-8282" # Neptune Gremlin proxy (proxy in Floci) + - "8700-8799:8700-8799" # MWAA webserver proxy (proxy in Floci) volumes: - /var/run/docker.sock:/var/run/docker.sock ``` diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md index aabdff1c37..a487f12809 100644 --- a/docs/getting-started/quick-start.md +++ b/docs/getting-started/quick-start.md @@ -165,21 +165,21 @@ Floci emulates ECR with a real OCI registry behind it, so the stock `docker` cli ```bash # Create the repository (lazy-starts the backing registry container) -aws ecr create-repository --repository-name floci-it/app --endpoint-url $AWS_ENDPOINT +aws ecr create-repository --repository-name floci-it/app --endpoint-url $AWS_ENDPOINT_URL # Authenticate -aws ecr get-login-password --endpoint-url $AWS_ENDPOINT \ +aws ecr get-login-password --endpoint-url $AWS_ENDPOINT_URL \ | docker login --username AWS --password-stdin \ - 000000000000.dkr.ecr.us-east-1.localhost:5000 + 000000000000.dkr.ecr.us-east-1.localhost:5100 # Push docker pull alpine:3.19 -docker tag alpine:3.19 000000000000.dkr.ecr.us-east-1.localhost:5000/floci-it/app:v1 -docker push 000000000000.dkr.ecr.us-east-1.localhost:5000/floci-it/app:v1 +docker tag alpine:3.19 000000000000.dkr.ecr.us-east-1.localhost:5100/floci-it/app:v1 +docker push 000000000000.dkr.ecr.us-east-1.localhost:5100/floci-it/app:v1 # Pull from a clean local image store -docker rmi 000000000000.dkr.ecr.us-east-1.localhost:5000/floci-it/app:v1 -docker pull 000000000000.dkr.ecr.us-east-1.localhost:5000/floci-it/app:v1 +docker rmi 000000000000.dkr.ecr.us-east-1.localhost:5100/floci-it/app:v1 +docker pull 000000000000.dkr.ecr.us-east-1.localhost:5100/floci-it/app:v1 ``` See the [ECR service docs](../services/ecr.md) for the full action surface, image-backed Lambda integration, and CDK `DockerImageFunction` support. @@ -198,7 +198,7 @@ If you want to scope it tighter to just the Lambda Runtime API and the ECR regis ```bash sudo ufw allow in on docker0 to any port 12000:12499 proto tcp comment 'floci lambda runtime api' -sudo ufw allow in on docker0 to any port 5000:5099 proto tcp comment 'floci ecr registry' +sudo ufw allow in on docker0 to any port 5100:5199 proto tcp comment 'floci ecr registry' ``` **Docker Desktop** (macOS / Windows / Linux) does not need this — it routes container → host through the Docker VM, which Floci's `DockerHostResolver` detects automatically. diff --git a/docs/service-api-parity-todos.md b/docs/service-api-parity-todos.md new file mode 100644 index 0000000000..5dbe44f4cb --- /dev/null +++ b/docs/service-api-parity-todos.md @@ -0,0 +1,84 @@ +# Floci service/API parity TODO inventory + +This is the consolidated backlog from the service/API investigations performed during the +current Floci compatibility work. It turns the existing service guides, parity epics, issue +write-ups, and completed branch work into one actionable queue. + +This is a planning document. It does not claim that an AWS operation is unsupported merely +because it is not listed here: the service guide and SDK compatibility tests remain the source +of truth for an individual operation. Every item below names the evidence that caused it to be +included and the next verification or implementation step. + +## Priority and status + +| Priority | Meaning | +| --- | --- | +| P0 | Blocks a demonstrated LZA path, can cause false-green behavior, or affects account/region correctness. | +| P1 | Important public API parity or lifecycle gap with a clear compatibility consumer. | +| P2 | Deliberate capability limit or lower-frequency API surface; schedule after P0/P1. | + +`Open` means work remains. `Implemented — verify` means the code is on a feature branch but +needs rollup, compatibility evidence, or release documentation before it can be called done. + +## P0: correctness and product decisions + +| ID | Service / area | Status | Evidence / gap | Next step | +| --- | --- | --- | --- | --- | +| PAR-001 | Account and region scoping across services | Open | A prior sweep of the CloudFormation provisioning path (provisioners, IAM, EC2, SQS, StackSets, Lambda launch/store) found and fixed 13 instances of code reaching for an ambient account/region (`RegionResolver.getAccountId()`/`getDefaultRegion()`, an unscoped `RequestContext`) instead of an already-resolved value passed to it. That sweep did not cover the rest of the service tree, nor the region-ambient variant of the same bug shape. A grep-based candidate filter currently flags 13 files: Amazon MQ, AppSync, Backup, CloudMap, CloudTrail, CodeDeploy, EKS, Floci UI, Kinesis Analytics, MSK, MWAA, OpenSearch, and SQS. | Grep `src/main/java` for `regionResolver\.getAccountId\(\)\|requestContext\.getAccountId\(\)\|regionResolver\.getDefaultRegion\(\)`, cross-reference hits against files using `ExecutorService`, `CompletableFuture`, `Executors.`, `.submit(`, `@Scheduled`, `ScheduledExecutorService`, or `new Thread(` (ambient reads are structurally wrong once code crosses off the originating request thread), then perform bounded, evidence-backed service batches. Record each finding in a numbered issue and exclude only confirmed global-resource exceptions. | +| PAR-002 | LZA CloudFormation replay/idempotency | Implemented — verify | Status filtering, per-resource checkpoints, security-group/custom-resource idempotency, VPC update handling, and a governed-pipeline integration test are implemented. | Roll the work into integration and run the net-new LZA matrix (including the supported 1.14/1.15/1.16 compatibility targets). Preserve exact stack events and restart behavior as the acceptance record. | +| PAR-003 | CodeBuild execution backend | Open investigation | [CodeBuild local-agent epic](services/codebuild-local-agent-investigation-epic.md) separates the AWS-compatible control plane from the execution backend. The published local agent is not the same image as `aws/codebuild/standard:7.0`; image mapping, output translation, cancellation, secrets, artifacts, and restart behavior remain decisions. | Characterize the published agent with deterministic fixtures, pin image digests, define a versioned backend seam, and run differential native-vs-agent tests before selecting preferred, opt-in, oracle-only, or rejected adoption. | +| PAR-004 | CodePipeline V2 | Open follow-up | [CodePipeline V2 epic](services/codepipeline-v2-epic.md) documents that current support is partial and that LZA currently exercises V1. Trigger execution, validation, queued/superseded/parallel isolation, condition providers, artifact lineage, retry/rollback, events, and CloudFormation round-trips remain. | Implement in slices beginning with V2 validation and trigger fixtures; require AWS SDK integration tests and preserve the existing V1/LZA suite as a regression gate. | +| PAR-005 | Local VPC network data plane | Open investigation | [Network data-plane epic](services/network-data-plane-investigation-epic.md) defines the gap between control-plane records and observable local traffic. Route, security-group, Network Firewall, DNS, endpoint, and logging behavior are not implied by the current API models. | Build a bounded privileged prototype behind a reconciler interface, then decide whether to adopt Podman, an appliance backend, an opt-in experiment, or control-plane-only behavior. Do not make normal API use require host privileges before the decision is proven. | + +## P1: public API and lifecycle gaps + +| ID | Service / area | Status | Evidence / gap | Next step | +| --- | --- | --- | --- | --- | +| PAR-101 | CloudFormation | Open | `docs/services/cloudformation.md` marks `ValidateTemplate`, stack-policy operations, some intrinsic resolution, and update/delete behaviors as stubs or unimplemented. Unsupported resource types intentionally receive stub physical IDs, which can hide missing service provisioners. | Add AWS SDK contract tests for each advertised stub/error; make unsupported resource handling explicit in the service guide and add exact `Ref`/`Fn::GetAtt` assertions for every newly wired provisioner. | +| PAR-102 | Lambda | Open | `docs/services/lambda.md` marks `ListLayers` and `ListLayerVersions` as empty stubs and notes that SQS event-source `MaximumConcurrency` is tracked but not enforced. | Add layer storage and SDK tests, then enforce event-source concurrency with restart-safe state and throttling/error semantics. | +| PAR-103 | RAM | Open | `docs/services/ram.md` states that resource-share APIs such as `CreateResourceShare` and `GetResourceShares` are not implemented; the persistence branch only addresses resource-share state retention. | Define the supported RAM resource/share model, implement the management API through `StorageFactory`, and verify account/region visibility plus persistence. | +| PAR-104 | AWS Batch | Open | `docs/services/batch.md` states that `process` mode, array-child fan-out, `CancelJob`, `TerminateJob`, and full Batch-specific input transformers are not implemented. Capacity, VCPU, and VPC behavior are metadata-only. | Choose a bounded local scheduler contract, implement cancellation/termination first, and add SDK tests that distinguish accepted metadata from executable behavior. | +| PAR-105 | API Gateway | Open | `docs/services/api-gateway.md` contains explicit “Not Implemented” sections for management/data-plane operations. | Convert each listed operation into an AWS SDK compatibility test, then prioritize operations required by LZA and common IaC providers. | +| PAR-106 | EKS | Open | `docs/services/eks.md` lists Phase 1 features as not implemented; current support does not imply a local Kubernetes control/data plane. | Keep unsupported operations AWS-shaped, document the supported IRSA/issuer boundary, and scope any future cluster behavior as a separate product decision. | +| PAR-107 | RUM | Open | `docs/services/rum.md` says event/data-plane, tag, resource-policy, metric-definition, and metric-destination APIs are not implemented; `CwLogEnabled` does not emit logs. | Implement the management subset only if a consumer requires it; otherwise add negative SDK tests and make the control-plane-only boundary explicit. | +| PAR-108 | Config | Open | `docs/services/config.md` notes that external evaluation does not record resource configurations and that rule evaluation is invocation bookkeeping rather than real evaluation. | Define the minimum resource recorder/configuration model, then add deterministic rule evaluation fixtures and AWS-shaped failure semantics. | +| PAR-109 | CloudWatch Logs Insights | Open | `docs/services/cloudwatch.md` documents a supported subset where unsupported commands are skipped with warnings rather than rejected, and data-protection policy behavior is incomplete. | Decide whether compatibility requires strict rejection or documented degradation; add query corpus tests for `stats`, `parse`, field projection, pagination, and data-protection APIs. | +| PAR-110 | Cost Explorer | Open | `docs/services/ce.md` lists reservation/Savings Plans coverage/utilization, cost categories, and anomaly management as zeroed/empty stubs or out of scope. | Keep stubs clearly marked, then implement only from a concrete consumer requirement with AWS SDK response-shape tests. | +| PAR-111 | RDS Data API | Open | `docs/services/rds-data.md` marks `BatchExecuteStatement`, parameter binding, JSON formatting/result options, and generated fields as unsupported. | Add parameter binding and batch execution against the local JDBC boundary, with tests for malformed requests and engine-specific errors. | +| PAR-112 | IoT Core | Open | `docs/services/iot.md` identifies missing TLS/mTLS, dynamic thing groups, fleet indexing, job rollouts/cancellation, S3 documents, and advanced scheduling. | Treat TLS/mTLS and job lifecycle as separate slices; avoid claiming data-plane parity while the embedded broker remains plaintext-only. | + +## P2: deliberate capability boundaries worth tracking + +| ID | Service / area | Status | Evidence / gap | Next step | +| --- | --- | --- | --- | --- | +| PAR-201 | DocumentDB | Open | `docs/services/docdb.md` leaves snapshot creation/restore out of scope and returns an empty snapshot result. | Add snapshot persistence only when a real compatibility consumer needs it; otherwise add an explicit negative test. | +| PAR-202 | CUR / BCM Data Exports | Open | `docs/services/cur.md` and `docs/services/bcm-data-exports.md` limit output to Parquet and reject CSV/text and compression variants. | Add format/compression support behind shared export fixtures, or keep the limitation explicit and tested. | +| PAR-203 | MWAA | Open | `docs/services/mwaa.md` notes metadata-only updates for several environment fields, stubbed web-login tokens, and no automatic Docker reconnection after restart. | Prioritize restart-safe reconnection and update semantics if LZA or a compatibility suite exercises them; leave hosted Airflow authentication explicit until then. | +| PAR-204 | OpenSearch | Open | `docs/services/opensearch.md` lists cross-cluster connections, VPC endpoints, packages, applications, and data sources as unsupported. | Add only the resources required by local IaC scenarios; preserve `UnsupportedOperationException`/AWS-shaped errors for the remainder. | +| PAR-205 | Transfer Family | Open | `docs/services/transfer.md` emulates management state but explicitly excludes actual SFTP/FTP protocol handling. | Keep the control/data-plane distinction visible; investigate a local protocol backend separately if a consumer requires end-to-end file transfer. | +| PAR-206 | Textract / Transcribe | Open | `docs/services/textract.md` and `docs/services/transcribe.md` are synthetic/stub control-plane implementations rather than OCR/transcription engines. | Document as deterministic test doubles and add negative/fixture tests; do not describe synthetic output as model parity. | + +## Completed work that still needs rollup evidence + +These are not new implementation TODOs, but they must not be lost during branch consolidation: + +| Area | Required close-out | +| --- | --- | +| KMS grant metadata | Roll up and run KMS integration plus persistence/reload assertions. | +| SNS / Control Tower prerequisite | Roll up and run the Control Tower/SNS prerequisite slice. | +| CodeBuild image mapping | Verify the selected ARM image digest and keep the curated-vs-public image distinction in docs. | +| CodePipeline retry artifacts | Run retry/rollback and persistence tests before V2 work changes lineage. | +| DynamoDB index persistence | Run hybrid-storage reload coverage. | +| RAM persistence | Pair with the open RAM API work above. | +| Network Firewall and Service Catalog | Roll up service and CloudFormation provisioner tests, then update the service count once both are present. | + +## Working rules for turning rows into issues + +1. One row becomes an issue only after its source guide or compatibility test is named in the + issue body. +2. A TODO is not closed by accepting an AWS-shaped request: the observable response, lifecycle, + persistence, error, and account/region behavior need an SDK-backed test where applicable. +3. Keep intentional stubs explicit. Never silently return success for an unsupported provider, + trigger, resource type, or data-plane action. +4. Re-run the integration rollup and branch-orphan audit after each batch; parity consolidation + comes after branch hygiene, not before. diff --git a/docs/services/codebuild-local-agent-investigation-epic.md b/docs/services/codebuild-local-agent-investigation-epic.md new file mode 100644 index 0000000000..cb396e4d9e --- /dev/null +++ b/docs/services/codebuild-local-agent-investigation-epic.md @@ -0,0 +1,261 @@ +# Investigation epic: AWS CodeBuild local agent backend + +## Outcome + +Determine whether Floci should use AWS's official CodeBuild local agent as its preferred build +execution backend. + +The investigation must produce a tested architectural decision, not merely prove that the agent +container starts. A successful result demonstrates that the agent improves Floci's observable +CodeBuild compatibility while Floci continues to provide the AWS API, stored state, service +integrations, and lifecycle around each build. + +## Decision question + +Should Floci delegate buildspec interpretation and build-phase execution to +`public.ecr.aws/codebuild/local-builds`, while retaining the existing native runner as a fallback? + +The possible decisions are: + +1. adopt the local agent as the preferred backend; +2. offer it as an opt-in backend for selected workloads; +3. use it only as a compatibility oracle in tests; or +4. reject it and continue improving the native runner. + +### Build-environment decision + +For the ARM prototype and LZA validation, use +`public.ecr.aws/codebuild/amazonlinux2-aarch64-standard:4.0`. This is a product decision rather +than an investigation variable. Record its digest for each validation run so a mutable registry +tag cannot make results irreproducible. Treat it as Floci's pullable local substitute, not as a +claim that the same identifier is a currently supported hosted-CodeBuild curated image. + +## Why investigate this + +Floci currently owns both halves of local CodeBuild: + +- the AWS-compatible control plane, including projects, builds, persistence, status, logs, + artifacts, cancellation, retry, and CodePipeline integration; and +- the execution engine that parses buildspecs, prepares containers, runs phases, and interprets + phase results. + +AWS publishes a local CodeBuild agent for x86_64 and ARM. Its documented launcher gives that +agent a build-environment image, source directory, artifact directory, buildspec, environment +variables, and access to the container-runtime socket. The agent then launches the separate build +environment and simulates CodeBuild phase execution locally. + +Delegating the execution half could improve compatibility and reduce the amount of CodeBuild +behavior Floci must reproduce. It does **not** replace Floci's CodeBuild service and does **not** +provide a pullable version of every AWS-managed build image. + +## Product hypothesis + +An AWS-agent backend will be a better Floci product if it: + +- produces results closer to observable AWS CodeBuild behavior; +- reduces custom buildspec and phase-orchestration code; +- integrates without weakening the existing AWS API or persistence contract; +- supports the workloads already demonstrated by Floci, especially LZA; and +- can be versioned, tested, diagnosed, and upgraded predictably. + +This is a hypothesis. AWS describes the agent as a way to simulate and troubleshoot builds +locally, not as a stable embedding API or a complete replacement for the hosted service. + +## Architectural boundary + +The investigation should introduce a backend seam rather than coupling `CodeBuildService` +directly to the agent: + +```text +AWS SDK / CLI / CodePipeline + | + CodeBuildService + | + CodeBuildExecutionBackend + / \ +NativeFlociBackend AwsLocalAgentBackend +``` + +Floci remains responsible for: + +- AWS JSON 1.1 request and response compatibility; +- project, build, and report-group persistence; +- account and region isolation; +- IAM, SSM, Secrets Manager, and environment-variable resolution; +- source and secondary-source acquisition; +- build status transitions, phase metadata, timeout, stop, and retry; +- CloudWatch Logs and EventBridge integration; +- artifact publication to S3; +- CodePipeline action callbacks; and +- behavior across a Floci restart. + +The candidate agent backend is responsible only for the locally observable execution contract: + +- buildspec parsing; +- phase ordering and command execution; +- runtime setup performed by the selected build image; +- command exit and phase-result semantics; and +- local report and artifact production exposed by the agent. + +## Important image distinction + +The local agent and the build environment are separate images: + +```text +public.ecr.aws/codebuild/local-builds:aarch64 # orchestration agent +public.ecr.aws/codebuild/amazonlinux2-aarch64-standard:4.0 # build environment +``` + +The agent is therefore not a substitute for `aws/codebuild/standard:7.0`. Floci must still map, +build, or otherwise provide a compatible environment image. The backend must record both the +requested image and the resolved executable image so this distinction is visible to users. + +## SWOT + +### Strengths + +- Uses an execution engine distributed and maintained by AWS. +- Supports both x86_64 and ARM local execution. +- May track CodeBuild buildspec and phase behavior more closely than Floci's custom runner. +- Could retire or simplify custom parsing, shell orchestration, report handling, and phase-state + translation. +- Gives Floci a credible compatibility story grounded in an official AWS tool. + +### Weaknesses + +- The container is opaque and its environment-variable interface is not a documented stable API. +- It simulates CodeBuild; it is not the hosted CodeBuild service and cannot define AWS behavior by + itself. +- It still requires a separate, architecture-compatible build-environment image. +- Floci must translate agent-local output back into AWS build, phase, log, report, and artifact + models. +- Failures inside the agent may be harder to diagnose or patch than failures in Floci-owned code. + +### Opportunities + +- Make the agent the preferred backend after compatibility is demonstrated. +- Offer selectable `native` and `aws-local-agent` backends during migration. +- Use differential tests to discover and close native-runner compatibility gaps. +- Use the agent as a permanent conformance oracle even if it is not adopted for production use. +- Present a differentiated product capability: a complete local AWS control plane around AWS's + official local CodeBuild execution agent. + +### Threats + +- AWS may change tags, digests, inputs, output layout, or behavior without an embedding contract. +- A mutable agent tag could silently change Floci behavior between releases. +- License and redistribution terms may prevent bundling or impose documentation obligations. +- Agent limitations may force two subtly different feature sets across execution backends. +- Users may assume the AWS agent guarantees perfect hosted-CodeBuild parity; documentation must + state the tested boundary precisely. + +## Investigation workstreams + +### 1. Distribution, lifecycle, and support contract + +- Record supported architectures, tags, published digests, update notifications, and image size. +- Review the image and repository licenses for execution, redistribution, and documentation + requirements. +- Determine whether Floci should pull by release tag, pin by digest, or require a user-provided + image. +- Establish how an agent upgrade is tested and intentionally promoted. +- Confirm behavior when the image is absent or cannot be pulled. + +### 2. Execution interface + +- Document every launcher input used by Floci: image, source, secondary sources, artifacts, + reports, buildspec override, environment file, credentials, profile, source mounting, and + privileged mode. +- Capture the agent's output files, exit codes, logs, phase markers, and failure messages. +- Determine which inputs can be passed without shell interpolation or temporary plaintext secret + files. +- Verify cancellation and timeout behavior at every lifecycle point. +- Define a versioned adapter inside Floci rather than scattering agent-specific variables through + `CodeBuildRunner`. + +### 3. AWS API translation + +- Map agent execution states to CodeBuild `BuildStatus`, `BuildPhase`, phase contexts, timestamps, + and `buildComplete`. +- Preserve current synchronous `StartBuild` response and asynchronous execution behavior. +- Stream or import logs into the existing CloudWatch Logs model with deterministic ordering. +- Translate local artifacts and reports into the existing S3 and report-group implementations. +- Preserve `StopBuild`, `RetryBuild`, batch lookup, list ordering, and CodePipeline polling. +- Ensure an unsupported agent capability fails explicitly rather than producing a false-green + build. + +### 4. Source, environment, and endpoint fidelity + +- Test primary and secondary sources, `NO_SOURCE`, S3, CodePipeline artifacts, source overrides, + and buildspec overrides. +- Test plaintext, Parameter Store, and Secrets Manager environment variables without leaking + resolved values. +- Verify Floci's account, region, endpoint, DNS-spoofing, and TLS trust configuration inside the + agent and build environment. +- Preserve Unix modes, symlinks, archive boundaries, and artifact path rules. +- Exercise custom images, curated-image mappings, entrypoints, privileged mode, and both host + architectures. + +### 5. Differential compatibility suite + +Create fixtures that run through both `NativeFlociBackend` and `AwsLocalAgentBackend` and compare +observable results: + +| Area | Required cases | +|---|---| +| Buildspec | missing file, YAML errors, phase ordering, finally blocks, command failure | +| Shell | quoting, multiline commands, working directory, exported variables, exit codes | +| Environment | project variables, overrides, Parameter Store, Secrets Manager, precedence | +| Sources | primary, secondary, S3, CodePipeline, overrides, executable files, symlinks | +| Artifacts | include/exclude patterns, base directory, names, failure, empty result | +| Reports | discovery, supported formats, malformed reports, report-group publication | +| Lifecycle | start, success, failure, timeout, stop, retry, concurrent builds, restart | +| Integration | CloudWatch Logs, EventBridge, S3, CodePipeline, LZA installer and core pipeline | + +Differences must be classified as: + +- the agent is closer to documented or observed AWS behavior; +- the native backend is closer; +- both are compatible despite an internal difference; +- AWS behavior is unknown and needs an external compatibility probe; or +- the local agent intentionally does not emulate the hosted feature. + +### 6. LZA vertical slice + +Use a bounded LZA slice before attempting a complete installation: + +1. run the installer project's build through the agent backend; +2. run one Toolkit project action with the same configuration archive; +3. verify CloudFormation, S3 artifacts, logs, and CodePipeline action state; +4. run the complete installer and core pipeline; +5. restart Floci and verify persisted CodeBuild and CodePipeline state; and +6. repeat against the supported LZA compatibility versions. + +The LZA run is an integration gate, not the only compatibility evidence. + +## Prototype plan + +### Phase 0: black-box characterization + +Run the published launcher manually with small deterministic buildspec fixtures. Record the exact +container topology, inputs, outputs, logs, artifacts, reports, exit behavior, and architecture +selection. Do not change Floci yet. + +### Phase 1: backend seam + +Extract the existing execution behavior behind `CodeBuildExecutionBackend` without changing its +observable behavior. Keep the native backend as the default and run the existing CodeBuild suite +against it. + +### Phase 2: minimum agent adapter + +Implement one `NO_SOURCE` build with inline commands, environment variables, logs, success, and +failure. Pin the agent image by digest. Do not advertise general support yet. + +### Phase 3: service integration + +Add sources, artifacts, reports, cancellation, timeout, retry, endpoint/TLS integration, and +restart behavior. Run the differential matrix after each capability is added. + +### Phase 4: LZA validation + diff --git a/docs/services/codepipeline-v2-epic.md b/docs/services/codepipeline-v2-epic.md new file mode 100644 index 0000000000..81631fdea9 --- /dev/null +++ b/docs/services/codepipeline-v2-epic.md @@ -0,0 +1,118 @@ +# Follow-up epic: complete CodePipeline V2 + +## Outcome + +Promote Floci's partial CodePipeline V2 support into an AWS-compatible local execution +environment. A V2 pipeline created through the AWS CLI, an AWS SDK, or CloudFormation must +behave like the corresponding AWS pipeline for triggers, execution modes, stage conditions, +variables, retry, rollback, history, and events. + +This is follow-up work. The LZA scenario currently exercises a **V1** pipeline and does not +depend on completion of this epic. + +## Current baseline + +Floci already accepts `pipelineType: V2` and implements several V2 capabilities: + +- `SUPERSEDED`, `QUEUED`, and `PARALLEL` execution modes +- pipeline execution variables and `#{variables.name}` resolution +- `beforeEntry`, `onSuccess`, and `onFailure` stage conditions +- real `LambdaInvoke` and `VariableCheck` rule evaluation +- condition override, failed-stage retry, and stage rollback +- rule-execution history and CodePipeline state-change events + +The remaining work is to make those capabilities complete and to remove permissive or +metadata-only behavior that can produce a false-green local execution. + +## Scope + +### 1. V2 definition and validation parity + +- Validate the V2-only fields, required combinations, limits, names, and enum values used by + `CreatePipeline` and `UpdatePipeline`. +- Reject V2 fields on a V1 pipeline with AWS-shaped validation errors. +- Preserve `pipelineType`, `executionMode`, variables, triggers, conditions, and rule + declarations through create, get, update, list, CloudFormation, and persistent reload. +- Cover single-region `artifactStore` and cross-region `artifactStores` without silently + selecting the wrong store. + +### 2. Native V2 triggers + +- Implement trigger declarations and filter matching for supported source providers, + including branch, file-path, tag, and pull-request filters where AWS exposes them. +- Start executions from actual local source events rather than recording trigger metadata + only. +- Populate `trigger`, `sourceRevisions`, and source action output variables from the event that + started the execution. +- Deduplicate redelivery and apply each pipeline's execution mode before work is dispatched. +- Keep provider support explicit: an unsupported trigger/provider combination must fail + validation or execution, never pass silently. + +### 3. Execution-mode fidelity + +- `SUPERSEDED`: supersede only executions and stages AWS would supersede, and expose the same + terminal state and reason. +- `QUEUED`: maintain deterministic FIFO admission across restarts and release the next run only + after the active run reaches a terminal state. +- `PARALLEL`: isolate artifacts, variables, action tokens, approvals, retry state, and rollback + targets per execution. +- Enforce AWS restrictions on retry, rollback, and stage-condition operations for each mode. + +### 4. Complete stage-condition rules + +- Replace the current permissive `Commands` and `DeployWindow` behavior with real evaluation. +- Match AWS rule lifecycle, timeout, retry, failure, result, and override semantics. +- Implement rule input/output variable expansion and expose AWS-compatible rule execution + summaries. +- Reject unknown rule providers rather than treating them as successful. +- Verify `beforeEntry`, `onSuccess`, and `onFailure` combinations, including `FAIL`, `SKIP`, + `ROLLBACK`, and condition overrides. + +### 5. Retry, rollback, and artifact lineage + +- Persist immutable per-execution artifact and source-revision lineage. +- Retry `FAILED_ACTIONS` and `ALL_ACTIONS` with AWS-compatible attempt numbering and history. +- Roll back using the selected successful execution's artifacts instead of rerunning source + merely to reconstruct them. +- Preserve lineage across Floci restart and reject missing or expired rollback targets with an + AWS-shaped error. + +### 6. API, event, and CloudFormation parity + +- Make pipeline, stage, action, rule, retry, rollback, and trigger fields agree across + `GetPipeline`, `GetPipelineState`, `GetPipelineExecution`, and the list APIs. +- Publish accurate EventBridge events for queued, superseded, parallel, condition, retry, and + rollback transitions. +- Round-trip every supported V2 property through `AWS::CodePipeline::Pipeline`, including + create, update in place, replacement rules, `Ref`, `Fn::GetAtt`, and deletion. +- Keep API responses reflection-safe for native-image builds. + +## Acceptance criteria + +The epic is complete only when all of the following are demonstrated: + +1. AWS SDK integration tests create and update both V1 and V2 pipelines and verify that invalid + cross-version fields return AWS-compatible errors. +2. Event-driven tests prove each supported trigger filter starts exactly the intended execution + and records the real trigger and source revision. +3. Deterministic tests run overlapping executions in `SUPERSEDED`, `QUEUED`, and `PARALLEL` + modes and verify status, isolation, ordering, artifacts, variables, and events. +4. Every advertised condition rule provider has positive, negative, timeout, retry, and + override coverage; no provider succeeds through a permissive fallback. +5. Retry and rollback tests prove artifact lineage is reused correctly before and after a Floci + restart. +6. CloudFormation integration tests create and update a V2 pipeline and assert its exact + triggers, variables, conditions, execution mode, physical ID, and attributes. +7. The CodePipeline compatibility suite passes using AWS SDK clients, and the V1 suite remains + green to prove the V2 work did not regress LZA's pipeline. +8. This service guide is updated so every advertised V2 feature corresponds to an executable + test rather than accepted-but-inert configuration. + +## Out of scope + +- Calling real AWS accounts or hosted third-party CI/CD providers +- Undocumented AWS internals that are not observable through public APIs, events, or SDK + behavior +- Replacing the existing in-process orchestration engine + + diff --git a/docs/services/network-data-plane-investigation-epic.md b/docs/services/network-data-plane-investigation-epic.md new file mode 100644 index 0000000000..87b2a283d5 --- /dev/null +++ b/docs/services/network-data-plane-investigation-epic.md @@ -0,0 +1,414 @@ +# Investigation epic: local VPC network data plane + +## Outcome + +Determine whether Floci should provide a bounded Linux network data plane that makes emulated +EC2 networking resources affect real traffic between local workloads. + +The investigation must produce a tested architectural decision, not merely demonstrate that a +container can run Suricata or that Linux namespaces can exchange packets. A successful result +shows that AWS SDK and CloudFormation control-plane changes are reconciled into observable, +deterministic routing, filtering, DNS, endpoint, and logging behavior. + +## Decision question + +Should Floci translate its AWS-compatible VPC, route-table, security-group, network ACL, and +Network Firewall state into a managed Linux networking topology? + +The possible decisions are: + +1. adopt an embedded Podman-backed data plane for supported local environments; +2. adopt a separate privileged network-appliance backend controlled by Floci; +3. provide both backends behind one versioned reconciliation interface; +4. keep the data plane as an opt-in experimental capability; or +5. reject the approach and retain control-plane-only network emulation. + +## Product hypothesis + +A local data plane will improve Floci if an AWS API change can cause the corresponding observable +network effect without weakening protocol compatibility or making normal control-plane use depend +on privileged host access. + +For example, after a route is changed to target a Network Firewall endpoint, traffic from an +attached workload should actually traverse the configured inspection path. An allow, drop, reject, +or alert decision should be observable in the workload and in the configured Floci logging service. + +This is behavioral emulation. It is not an attempt to reproduce AWS's internal network fabric, +capacity, availability, latency, or proprietary implementation. + +## Architectural boundary + +Floci remains the AWS-compatible control plane: + +```text +AWS SDK / CLI / CloudFormation + | + Floci service models + | + NetworkDataPlaneReconciler + | + NetworkDataPlaneBackend + / \ +PodmanNetworkBackend ApplianceBackend + | + Linux namespaces, routes, nftables/eBPF, Suricata, DNS +``` + +Floci remains responsible for: + +- AWS request, response, validation, and error compatibility; +- account, region, VPC, subnet, and resource ownership; +- persistent desired state and lifecycle transitions; +- EC2, Route 53, Network Firewall, CloudWatch Logs, S3, and Firehose integration; +- dependency and in-use validation; +- stable identifiers and AWS-shaped status responses; +- reconciliation scheduling, retries, health, and recovery; and +- reporting a capability as ready only after the backend confirms it. + +The backend is responsible for: + +- isolated network domains and interfaces; +- address assignment, forwarding, and route programming; +- stateless and stateful policy enforcement; +- packet inspection and deterministic verdicts; +- DNS forwarding and private-zone attachment where selected; +- flow, alert, and health event production; and +- safe cleanup of backend resources owned by Floci. + +The backend must not introduce a public convenience API. Users continue to configure networking +through the published AWS APIs and CloudFormation resource types. + +## Candidate behavioral slice + +The smallest credible slice is two workload containers in separate emulated subnets with a route +through an emulated Network Firewall endpoint: + +```text +workload A + | +subnet A namespace + | +EC2 route-table decision + | +Network Firewall endpoint namespace + | +nftables stateless rules + Suricata stateful rules + | +subnet B namespace + | +workload B +``` + +The slice must demonstrate: + +1. workload attachment through existing AWS resource state; +2. deterministic subnet addressing and routing; +3. route-table selection of a firewall endpoint; +4. allow, drop, reject, and alert outcomes; +5. policy updates without rebuilding unrelated workloads; +6. observable flow and alert records; and +7. restart reconciliation from Floci's persisted desired state. + +Simply connecting containers to the same Podman network does not satisfy this slice because it +bypasses the AWS route and policy models being investigated. + +## Candidate implementation mechanisms + +### Network isolation and connectivity + +- Linux network namespaces for VPCs, subnets, endpoints, or routing domains. +- Bridges and virtual Ethernet pairs for workload attachment. +- Linux routes and policy-routing tables compiled from EC2 route tables. +- Network address translation for explicitly supported internet and egress paths. +- Deterministic address allocation derived from VPC and subnet CIDRs. + +The investigation must determine the minimum namespace granularity that preserves isolation while +remaining diagnosable and inexpensive. A namespace per subnet may be simpler to reason about than +a namespace per VPC, but it creates more interfaces and reconciliation work. + +### Policy enforcement + +- `nftables` as the baseline mechanism for stateless rules, security groups, network ACLs, NAT, + counters, and packet marking. +- Suricata for stateful Network Firewall rules and AWS-compatible Suricata rule strings. +- eBPF only if a concrete experiment demonstrates a material capability or performance benefit; + it must not be a prerequisite for the first useful slice. + +Security groups are stateful and network ACLs are stateless. Compiling both into one undifferentiated +firewall chain would produce incorrect behavior and is not an acceptable prototype shortcut. + +### DNS + +- Connect the data plane to Floci's Route 53 private hosted-zone and Resolver state. +- Provide per-VPC or per-subnet DNS endpoints only where their lifecycle is represented by AWS + resources. +- Preserve normal host DNS as an explicit forwarding path rather than silently mixing host and + emulated private-zone answers. + +### Observability + +- Translate packet counters and Suricata events into stable internal flow and alert records. +- Deliver records through the configured Floci CloudWatch Logs, S3, or Firehose implementation. +- Correlate events with account, region, VPC, subnet, endpoint, firewall, policy, and rule IDs. +- Expose reconciliation health in existing AWS status fields and normal Floci diagnostics. + +## Deployment models to compare + +### Embedded Podman topology + +Floci creates and owns networks, namespaces, helper containers, and inspection containers through +the configured container runtime. + +Potential advantages: + +- fits Floci's existing container-runtime integration; +- provides a single-product installation path; +- can reuse container lifecycle, image, retry, and cleanup infrastructure; and +- makes workload attachment straightforward for other container-backed Floci services. + +Questions to answer: + +- Which behaviors work with rootless Podman? +- Which require capabilities unavailable through the Docker-compatible API? +- Can topology changes be reconciled without restarting the Podman machine? +- How are leaked namespaces and interfaces detected and cleaned safely? +- Does Docker provide a meaningfully equivalent implementation path? + +### Privileged network appliance + +A dedicated Linux VM or privileged container owns namespaces, routing, firewall rules, and +Suricata. Floci sends desired state through a private, versioned reconciliation interface. + +Potential advantages: + +- isolates privileges from the Java process; +- provides direct access to Linux networking primitives; +- creates one consistent backend on macOS, Linux, and Windows hosts; and +- can expose precise health and reconciliation results. + +Questions to answer: + +- Is the appliance lifecycle manageable without becoming a second product? +- How are API compatibility, upgrades, and state recovery versioned? +- Can the control channel be authenticated and kept inaccessible to workloads? +- Can multiple Floci instances or accounts safely share one appliance? + +### Host-platform implications + +The actual packet machinery requires Linux. On macOS or Windows, it would run inside the Podman +machine or another Linux VM. The investigation must not assume that host-side network interfaces +or namespaces are directly visible from the Floci process. + +Rootless operation is preferred, but the investigation must report capability boundaries rather +than fabricating success. If a behavior requires privileged networking, it must be isolated, +opt-in, and documented. + +## Reconciliation and state model + +The data plane must be derived from persisted AWS desired state. Linux objects are disposable +materializations, not the authoritative database. + +Each reconciliation unit needs: + +- an account and region scope; +- an owning AWS resource identifier; +- a desired-state generation or revision; +- observed backend state and health; +- a deterministic backend object name; +- idempotent create, update, and delete behavior; +- retry classification and bounded backoff; and +- orphan detection that cannot delete objects not owned by Floci. + +Updates must be safe under concurrent AWS operations and after process interruption. A failed +reconciliation must leave the AWS resource in an accurate pending or failed state rather than +returning an unconditional `READY` status. + +## AWS capability mapping to investigate + +| AWS capability | Candidate local behavior | +|---|---| +| VPC and subnet | Isolated routing domain, CIDR, gateway, and attachment boundary | +| Elastic network interface | Stable virtual interface and addresses attached to a workload | +| Route table | Linux route or policy-routing decision compiled from AWS routes | +| Security group | Stateful ingress and egress policy tied to interfaces or identities | +| Network ACL | Ordered stateless subnet-boundary rules | +| Internet gateway | Explicit controlled forwarding to the host or appliance uplink | +| NAT gateway | Source NAT with connection tracking and observable counters | +| VPC peering | Routed connection between isolated VPC domains with AWS validation | +| Transit Gateway | Shared routing domain and attachment/propagation model | +| VPC endpoint | Local service destination and DNS behavior without public egress | +| Network Firewall endpoint | Mandatory inspection hop selected by EC2 route state | +| Stateless rule group | Ordered `nftables` rules and default actions | +| Stateful rule group | Suricata-compatible inspection and connection state | +| Firewall logging | Flow and alert delivery through configured Floci destinations | +| Route 53 private DNS | VPC-scoped resolution derived from hosted-zone associations | + +The table is a research inventory, not a claim that every row belongs in the first implementation. + +## Investigation workstreams + +### 1. Platform capability matrix + +- Characterize rootful and rootless Podman on Linux and through Podman Machine on macOS. +- Record required kernel features, capabilities, socket access, images, and minimum versions. +- Determine Docker compatibility and explicitly identify unsupported combinations. +- Measure startup time, steady-state memory, disk use, and cleanup behavior. +- Test operation when Floci itself runs inside a container. + +### 2. Topology prototype + +- Create two isolated subnets and attach deterministic test workloads. +- Compile an EC2 route into a real forwarding path. +- Prove that no unintended host or peer-network path bypasses the route. +- Change and delete the route while traffic is active. +- Restart Floci and reconstruct the topology without changing AWS resource IDs. + +### 3. Network Firewall prototype + +- Compile a stateless rule group into `nftables`. +- load an AWS-compatible Suricata rule string into Suricata; +- compose both through a firewall policy and endpoint; +- demonstrate allow, drop, reject, and alert cases; +- update rules using AWS update-token semantics; and +- prove that a route targeting the endpoint cannot bypass inspection. + +### 4. Security group and network ACL semantics + +- Demonstrate stateful return traffic for security groups. +- Demonstrate stateless, ordered, subnet-boundary behavior for network ACLs. +- Test IPv4, IPv6, protocol, port-range, and cross-reference cases selected for support. +- Determine how rule changes affect established connections. +- Compare observable results with documented AWS behavior. + +### 5. DNS and service endpoints + +- Resolve a Route 53 private-zone record only from its associated VPC. +- Forward an unrelated public query through an explicit resolver path. +- Model a gateway or interface endpoint for one Floci service. +- Verify endpoint-specific DNS and routing without leaking across accounts or VPCs. + +### 6. Logging and diagnostics + +- Deliver firewall flow and alert records to Floci CloudWatch Logs. +- Characterize ordering, timestamps, batching, retry, and destination failure. +- Provide an operator snapshot mapping AWS resources to backend objects. +- Capture packet and policy evidence sufficient to explain an unexpected verdict. +- Ensure secrets, payload content, and unrelated host traffic are not logged by default. + +### 7. Lifecycle, isolation, and failure recovery + +- Test concurrent create, update, and delete operations. +- Kill Floci and the backend during each reconciliation stage. +- Reconcile missing, stale, and partially created backend objects. +- Prove account, region, VPC, and subnet isolation with overlapping CIDRs. +- Detect and clean only positively identified Floci-owned orphans. +- Exercise disk pressure, backend unavailability, corrupt policy, and image-pull failure. + +### 8. Performance envelope + +- Measure reconciliation latency independently from packet throughput. +- Measure connection setup, steady-state throughput, CPU, memory, and packet loss. +- Test increasing workload, subnet, route, security-group rule, and firewall-rule counts. +- Establish explicit supported limits rather than copying AWS quotas that the backend cannot meet. +- Verify that enabling the data plane does not materially slow control-plane-only users. + +## Prototype phases + +### Phase 0: black-box platform characterization + +Build disposable Linux-network experiments outside Floci. Record exact commands, privileges, +kernel behavior, Podman/Docker differences, cleanup requirements, and failure modes. + +### Phase 1: backend seam and dry-run compiler + +Define `NetworkDataPlaneBackend` and a desired-state representation. Compile existing Floci +resources into a deterministic plan without changing the host. Snapshot-test that plan and prove +that account and region boundaries are preserved. + +### Phase 2: minimum routed topology + +Materialize two subnets, two workloads, and one EC2 route. Support create, update, delete, restart, +and orphan detection. Do not add Network Firewall until routing cannot be bypassed. + +### Phase 3: firewall vertical slice + +Add one endpoint, stateless policy, stateful Suricata policy, and CloudWatch Logs output. Drive the +entire flow through AWS SDK or CloudFormation calls and verify traffic externally from workloads. + +### Phase 4: policy boundaries + +Add selected security-group and network-ACL semantics, private DNS, and one VPC endpoint. Run +cross-account, cross-region, overlapping-CIDR, restart, and failure-recovery tests. + +### Phase 5: product decision + +Compare the measured result with the acceptance criteria and choose a deployment model and support +tier. If adopted, create implementation epics per AWS capability rather than expanding the +prototype directly into an unbounded network rewrite. + +## Acceptance criteria for the investigation + +The investigation is complete only when all of the following are demonstrated or explicitly shown +to be infeasible: + +1. AWS SDK or CloudFormation calls create a routed two-subnet topology without a custom public API. +2. An EC2 route-table update changes the path of real workload traffic. +3. A Network Firewall endpoint produces deterministic allow, drop, reject, and alert outcomes. +4. Stateless and stateful policy mechanisms remain distinct and match their selected AWS cases. +5. Flow and alert logs reach a configured Floci destination with traceable AWS resource IDs. +6. Floci reconstructs backend state after restart from persistent AWS desired state. +7. Two accounts and regions with overlapping CIDRs cannot observe or affect each other's traffic. +8. Failed or partial reconciliation produces accurate lifecycle status and can recover idempotently. +9. Cleanup removes only backend objects bearing verified Floci ownership metadata. +10. Rootless, privileged, macOS Podman Machine, Linux, and unsupported-host boundaries are recorded. +11. Resource use and throughput are measured against a declared local support envelope. +12. The final decision records whether to adopt, limit, or reject the data plane and why. + +## Decision gates + +Adoption requires all of these gates: + +- **Compatibility:** selected AWS control-plane operations cause the documented local effects. +- **Isolation:** account, region, VPC, and subnet boundaries survive adversarial tests. +- **Recoverability:** reconciliation is idempotent across restart and partial failure. +- **Operability:** users can diagnose topology, policy, and verdicts without inspecting Java internals. +- **Portability:** supported host/runtime combinations and privilege requirements are explicit. +- **Safety:** Floci cannot accidentally alter or delete unrelated host networking objects. +- **Value:** the behavioral fidelity gained justifies the runtime, privilege, and maintenance cost. + +Failure of the safety or isolation gate is a rejection criterion, not a deferred production-hardening +task. + +## Non-goals + +- Reproducing AWS's internal network architecture, scale, availability, or latency. +- Claiming equivalence to proprietary AWS threat intelligence or managed rule groups. +- Inspecting arbitrary host traffic outside explicitly attached Floci workloads. +- Making privileged networking mandatory for users who need only AWS control-plane emulation. +- Implementing every EC2 or Network Firewall operation in the prototype. +- Building packet forwarding or inspection directly in Java. +- Adding non-AWS public endpoints for topology management. + +## Expected investigation artifacts + +- architecture decision record with the selected deployment model; +- platform and privilege capability matrix; +- versioned desired-state and backend-interface proposal; +- reproducible topology and traffic fixtures; +- AWS SDK and CloudFormation vertical-slice tests; +- compatibility findings for each selected AWS behavior; +- performance and resource measurements; +- threat model and cleanup-safety review; +- documented limitations and support envelope; and +- sequenced implementation epics if the result is adoption. + +## Relationship to service parity work + +This epic does not replace the Network Firewall, EC2, Route 53, CloudWatch Logs, S3, Firehose, or +CloudFormation API parity epics. Those services define the AWS-compatible desired state. This epic +investigates whether Floci can materialize a useful subset of that state into a real local packet +path. + +An operation may be fully control-plane compatible before its data-plane effect is implemented. +Documentation and tests must distinguish those levels so Floci never reports a false-green +capability. diff --git a/mkdocs.yml b/mkdocs.yml index 594261d641..a8a2efa868 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -149,7 +149,6 @@ nav: - AppSync: services/appsync.md - Transfer Family: services/transfer.md - AWS Config: services/config.md - - CloudTrail: services/cloudtrail.md - EMR: services/emr.md - EMR Serverless: services/emr-serverless.md - WAF v2: services/wafv2.md