diff --git a/.claude/skills/document/SKILL.md b/.claude/skills/document/SKILL.md new file mode 100644 index 000000000..e32dbac49 --- /dev/null +++ b/.claude/skills/document/SKILL.md @@ -0,0 +1,81 @@ +--- +name: document +description: Generate or update project documentation. Use when asked to document code, create docs, or explain features. +--- + +# Documentation Skill — S3Mock + +Generate and maintain documentation for the S3Mock project. + +## When to Use + +- Creating or updating README.md +- Updating the CHANGELOG.md for new features/fixes +- Updating the S3 operations support table +- Documenting new configuration options +- Updating AGENTS.md files when architecture changes + +## Pre-Flight Checklist + +- [ ] Read existing `AGENTS.md` for project conventions and structure +- [ ] Review the documentation being updated for current style and tone +- [ ] Identify target audience (end users for README, agents for AGENTS.md, contributors for CONTRIBUTING.md) + +## S3Mock Documentation Structure + +| File | Audience | Purpose | +|------|----------|---------| +| `README.md` | End users & contributors | Usage, configuration, quick start | +| `CHANGELOG.md` | End users | Version history, breaking changes, migration notes | +| `AGENTS.md` (root) | AI agents & contributors | Architecture, code style, DO/DON'T guardrails | +| `server/AGENTS.md` | AI agents | Server module implementation details | +| `integration-tests/AGENTS.md` | AI agents | Integration test patterns and helpers | +| `testsupport/AGENTS.md` | AI agents | Test framework integration details | +| `.github/CONTRIBUTING.md` | Contributors | How to contribute, dev setup, code reviews | +| `.github/SECURITY.md` | Security researchers | Vulnerability reporting, supported versions | +| `.github/CODEOWNERS` | GitHub | Automatic PR review assignment | +| `.github/ISSUE_TEMPLATE/*.yml` | Users & contributors | Structured bug reports and feature requests | + +## Documentation Tasks + +### When a New S3 Operation is Implemented +1. Update the **operations table** in `README.md` — change `:x:` to `:white_check_mark:` for the operation +2. Add a CHANGELOG entry under the current version section in `CHANGELOG.md` +3. Update `server/AGENTS.md` if the implementation introduces new patterns + +### When Configuration Changes +1. Update the **Configuration table** in `README.md` +2. Update the **Configuration section** in `AGENTS.md` +3. Add a CHANGELOG entry + +### When Architecture Changes +1. Update the relevant module's `AGENTS.md` +2. Update the root `AGENTS.md` if the change affects the overall structure +3. Add a CHANGELOG entry for breaking changes + +### CHANGELOG Format +Follow the existing pattern in `CHANGELOG.md`: +- Group changes under the current version heading (e.g., `## 5.0.0`) +- Use clear, user-facing language +- Note breaking changes prominently +- Reference related GitHub issues or PRs + +## Writing Style + +- **Concise**: Short sentences, active voice +- **Code examples**: Include runnable examples where possible (Kotlin for API, shell for CLI) +- **Links**: Reference AWS S3 API docs for operations, link to source files for implementations +- **Consistent**: Match existing formatting — Markdown headings, table alignment, badge style + +## Post-Flight Checklist + +- [ ] Technical accuracy verified against source code +- [ ] Code examples compile/run correctly +- [ ] Links are valid (internal file paths, external URLs) +- [ ] Consistent style with surrounding documentation +- [ ] Markdown renders correctly (tables, code blocks, badges) +- [ ] No outdated version numbers or deprecated references + +## Output + +Provide documentation ready to integrate into the appropriate project files, matching existing conventions. diff --git a/.claude/skills/implement/SKILL.md b/.claude/skills/implement/SKILL.md new file mode 100644 index 000000000..295f67c89 --- /dev/null +++ b/.claude/skills/implement/SKILL.md @@ -0,0 +1,80 @@ +--- +name: implement +description: Implement features, fix bugs, or refactor code. Use when asked to add functionality, modify code, or improve structure. +--- + +# Implementation Skill — S3Mock + +Implement features and fix bugs in the S3Mock project (Kotlin 2.3, Spring Boot 4.0.x, Maven). + +## When to Use + +- Adding new S3 API operations +- Fixing bugs in existing operations +- Refactoring server, service, or store layers +- Updating DTOs or XML serialization + +## Pre-Flight Checklist + +- [ ] Read the root `AGENTS.md` — especially the DO/DON'T section +- [ ] Read the module-specific `AGENTS.md` (`server/AGENTS.md`, etc.) +- [ ] Check `CHANGELOG.md` for planned changes or deprecations +- [ ] Identify which S3 API operation is being implemented (check [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)) +- [ ] Review existing similar implementations in the codebase + +## Implementation Flow for New S3 Operations + +Follow the **DTO → Store → Service → Controller** layered architecture: + +### 1. DTO (`server/src/main/kotlin/com/adobe/testing/s3mock/dto/`) +- Create request/response data classes with Jackson XML annotations +- Use `@JacksonXmlRootElement(localName = "...")` matching the AWS API element name exactly +- Use `@JacksonXmlProperty(localName = "...")` for properties +- Use `@JacksonXmlElementWrapper(useWrapping = false)` for collections +- Verify naming against [AWS S3 API docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) + +### 2. Store (`server/src/main/kotlin/com/adobe/testing/s3mock/store/`) +- Add filesystem operations to `BucketStore` or `ObjectStore` +- Follow existing patterns for metadata JSON and binary data storage +- Handle file I/O with proper error handling + +### 3. Service (`server/src/main/kotlin/com/adobe/testing/s3mock/service/`) +- Add business logic, validation, and store coordination +- Throw S3 exceptions (`NoSuchBucketException`, `NoSuchKeyException`, etc.) +- Use constructor injection for dependencies + +### 4. Controller (`server/src/main/kotlin/com/adobe/testing/s3mock/controller/`) +- Add HTTP endpoint mapping (`@GetMapping`, `@PutMapping`, etc.) +- Controllers only map HTTP — delegate all logic to services +- Return proper HTTP status codes and headers (ETag, Content-Type, etc.) + +## Code Standards + +- **Language**: Kotlin 2.3, JVM target 17 +- **DI**: Constructor injection only — never `@Autowired` or field injection +- **DTOs**: Data classes with `val` properties +- **Null safety**: Use `?`, `?.`, `?:` — avoid `!!` +- **Functions**: Expression bodies for simple functions +- **Dependencies**: Prefer Kotlin stdlib over third-party libraries +- **Versions**: All dependency versions in root `pom.xml` only + +## Post-Flight Checklist + +- [ ] Run `./mvnw ktlint:format` to fix code style +- [ ] Run `./mvnw clean install` to verify build +- [ ] Verify no checkstyle violations +- [ ] Add/update unit tests (`*Test.kt`) for new service/store logic +- [ ] Add/update integration tests (`*IT.kt`) for new endpoints +- [ ] Update `CHANGELOG.md` under the current version section +- [ ] Update the operations table in `README.md` if a new S3 operation was added + +## Troubleshooting + +- **Build fails**: Check Java version (`java -version` — needs 25), run `./mvnw ktlint:format` +- **Checkstyle fails**: Review rules in `etc/checkstyle.xml` — common issues are import ordering and missing Javadoc +- **Tests fail after changes**: Ensure XML serialization matches AWS API exactly — compare element names against [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) +- **Docker build fails**: Try `./mvnw clean install -DskipDocker` first to isolate the issue + +## Output + +Provide clean, well-structured Kotlin code following the layered architecture and project conventions defined in AGENTS.md. diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md new file mode 100644 index 000000000..1cb20c841 --- /dev/null +++ b/.claude/skills/test/SKILL.md @@ -0,0 +1,133 @@ +--- +name: test +description: Write, update, or fix tests. Use when asked to test code, create test cases, or debug failing tests. +--- + +# Test Skill — S3Mock + +Create and maintain tests for S3Mock (JUnit 5, Mockito, AssertJ, AWS SDK v2). + +## When to Use + +- Writing new unit or integration tests +- Fixing failing tests +- Adding test coverage for S3 operations +- Creating test fixtures or helpers + +## Pre-Flight Checklist + +- [ ] Read `AGENTS.md` — especially DO/DON'T and Testing sections +- [ ] Identify test type needed (unit vs. integration — see below) +- [ ] Review existing test patterns in the target module +- [ ] For integration tests, review `S3TestBase` for available helpers + +## Test Types in S3Mock + +### Unit Tests (`*Test.kt`) +- **Location**: `server/src/test/kotlin/com/adobe/testing/s3mock/` +- **Framework**: JUnit 5 + `@SpringBootTest` with `@MockitoBean` for mocking +- **Assertions**: AssertJ (`assertThat(...)`) +- **Purpose**: Test services and stores in isolation with Spring context and mocked dependencies +- **Pattern**: +```kotlin +@SpringBootTest(classes = [ServiceConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) +@MockitoBean(types = [BucketService::class, MultipartService::class, MultipartStore::class]) +internal class ObjectServiceTest : ServiceTestBase() { + @Autowired + private lateinit var iut: ObjectService + + @Test + fun `should delete object`() { + val bucketName = "bucket" + val key = "key" + givenBucketWithContents(bucketName, "", listOf(givenS3Object(key))) + + iut.deleteObject(bucketName, key) + + assertThat(iut.getObject(bucketName, key)).isNull() + } +} +``` + +### Integration Tests (`*IT.kt`) +- **Location**: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/` +- **Base class**: Extend `S3TestBase` for access to pre-configured `s3Client` (AWS SDK v2) +- **Assertions**: AssertJ +- **Purpose**: Test S3Mock end-to-end with real AWS SDK clients against the Docker container +- **Pattern**: +```kotlin +internal class MyFeatureIT : S3TestBase() { + @Test + fun `should perform operation`(testInfo: TestInfo) { + // Arrange — always use unique bucket names + val bucketName = givenBucket(testInfo) + + // Act + s3Client.putObject( + PutObjectRequest.builder().bucket(bucketName).key("key").build(), + RequestBody.fromString("content") + ) + + // Assert + val response = s3Client.getObject( + GetObjectRequest.builder().bucket(bucketName).key("key").build() + ) + assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") + } +} +``` + +## Test Standards + +- **Naming**: Use backtick names with descriptive sentences: `` fun `should create bucket successfully`() `` + - Legacy `testSomething` camelCase naming exists in older tests — refactor to backtick style when touching those tests +- **Independence**: Each test creates its own resources — never share state between tests +- **Bucket names**: Use `givenBucket(testInfo)` from `S3TestBase` in integration tests for unique names +- **Arrange-Act-Assert**: Follow this pattern consistently +- **Both paths**: Test success cases AND error/exception cases +- **Error assertions**: Use `assertThatThrownBy { ... }.isInstanceOf(AwsServiceException::class.java)` (AssertJ), not `assertThrows` +- **SDK version**: Use AWS SDK v2 (`s3Client`) only — SDK v1 has been removed +- **No JUnit 4**: Use JUnit 5 exclusively (`@Test` from `org.junit.jupiter.api`) +- **Visibility**: Mark test classes as `internal` +- **No MockitoExtension**: Use `@SpringBootTest` with `@MockitoBean` for mocking — never `@ExtendWith(MockitoExtension::class)` +- **Mocking**: Use `@MockitoBean` (class-level `types` or field-level) instead of `@Mock` / `@InjectMocks` +- **Injection**: Use `@Autowired` for the class under test in Spring Boot tests + +## Running Tests + +```bash +# Unit tests (server module) +./mvnw test -pl server + +# All integration tests +./mvnw verify -pl integration-tests + +# Specific integration test class +./mvnw verify -pl integration-tests -Dit.test=BucketIT + +# Specific test method +./mvnw verify -pl integration-tests -Dit.test=BucketIT#shouldCreateBucket + +# Skip Docker (for unit tests only) +./mvnw test -pl server -DskipDocker +``` + +## Post-Flight Checklist + +- [ ] Tests pass locally (`./mvnw test` or `./mvnw verify`) +- [ ] Tests are independent (can run in any order) +- [ ] Both success and failure cases covered +- [ ] Assertions are specific (not just `isNotNull()`) +- [ ] No hardcoded bucket names (use UUID) +- [ ] Code style passes (`./mvnw ktlint:format`) + +## Troubleshooting Failing Tests + +- **Docker not running**: Integration tests require Docker — start Docker Desktop +- **Port conflict**: Ports 9090/9191 may be in use — check with `lsof -i :9090` +- **Flaky test**: Ensure test independence — check for shared state or ordering dependencies +- **Compilation error**: Run `./mvnw clean install -DskipDocker -DskipTests` first + +## Output + +Provide complete, runnable Kotlin tests following S3Mock conventions and the Arrange-Act-Assert pattern. diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 000000000..f06a7ed77 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,20 @@ +# Default owner for everything in the repo +* @afranken + +# Core server implementation +/server/ @afranken + +# Integration tests +/integration-tests/ @afranken + +# Test support modules +/testsupport/ @afranken + +# Docker configuration +/docker/ @afranken + +# CI/CD and GitHub configuration +/.github/ @afranken + +# Build configuration +/build-config/ @afranken diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 706fca46e..327ee7f34 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -12,11 +12,66 @@ This project adheres to the Adobe [code of conduct](../CODE_OF_CONDUCT.md). By p All third-party contributions to this project must be accompanied by a signed contributor license. This gives Adobe permission to redistribute your contributions as part of the project. Sign our [CLA](http://adobe.github.io/cla.html). You only need to submit an Adobe CLA one time, so if you have submitted one previously, you are good to go! +## Development Setup + +**Prerequisites:** +- Java 25 (compile version; targets JVM 17) +- Maven 3.9+ (use the included `./mvnw` wrapper) +- Docker (for Docker build and integration tests) + +**Build and verify:** +```shell +# Full build with Docker +./mvnw clean install + +# Skip Docker (faster, for unit tests only) +./mvnw clean install -DskipDocker + +# Run integration tests +./mvnw verify -pl integration-tests + +# Format Kotlin code +./mvnw ktlint:format +``` + +## Architecture + +S3Mock follows a **Controller - Service - Store** layered architecture. For detailed architecture documentation, code style guidelines, and project conventions, see the [AGENTS.md](../AGENTS.md) in the project root. + +Module-specific documentation: +- [Server Module](../server/AGENTS.md) - core implementation +- [Integration Tests](../integration-tests/AGENTS.md) - test patterns +- [Test Support](../testsupport/AGENTS.md) - framework integrations + +## Code Style + +- **Kotlin**: Enforced by ktlint - run `./mvnw ktlint:format` before submitting +- **XML/Java**: Enforced by Checkstyle - configuration in [`etc/checkstyle.xml`](../etc/checkstyle.xml) +- **Key conventions**: Constructor injection, data classes for DTOs, backtick test names, `val` over `var` +- See the DO / DON'T section in [AGENTS.md](../AGENTS.md) for the full list + ## Code Reviews All submissions should come in the form of pull requests and need to be reviewed by project committers. Read [GitHub's pull request documentation](https://help.github.com/articles/about-pull-requests/) for more information on sending pull requests. -All submissions must include unit tests for any new functionality or bug fixes. If you are adding a new feature, please include a test that demonstrates the feature. -S3Mock uses Unit tests for function coverage, Spring Boot tests for component coverage, and integration tests against the Docker container artefact for end-to-end coverage. Please ensure that your code is covered by at least one of these test types. +## Testing + +All submissions must include tests for any new functionality or bug fixes. + +S3Mock uses three test levels: +1. **Unit tests** (`*Test.kt`) - Spring Boot tests with `@MockitoBean` for mocking, in `server/src/test/` +2. **Spring Boot tests** - component-level coverage with Spring context +3. **Integration tests** (`*IT.kt`) - end-to-end tests against the Docker container using real AWS SDK v2 clients, in `integration-tests/src/test/` + +Please ensure that your code is covered by at least one of these test types. + +## Submitting Changes + +1. Ensure all CI gates pass (build, tests, ktlint, checkstyle, Docker) +2. Update [CHANGELOG.md](../CHANGELOG.md) under the current version section for user-facing changes +3. Update documentation if applicable (README.md, AGENTS.md) +4. Follow the [pull request template](PULL_REQUEST_TEMPLATE.md) when submitting + +## Security -Lastly, please follow the [pull request template](PULL_REQUEST_TEMPLATE.md) when submitting a pull request! +To report security vulnerabilities, see the [Security Policy](SECURITY.md). diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 000000000..f8b3b47af --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,87 @@ +name: Bug Report +description: Report a bug or unexpected behavior in S3Mock +title: "[Bug]: " +labels: ["bug"] +assignees: + - afranken +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to report a bug! Please fill out the information below to help us investigate. + - type: input + id: version + attributes: + label: S3Mock Version + description: Which version of S3Mock are you using? + placeholder: "e.g., 5.0.0" + validations: + required: true + - type: dropdown + id: usage-mode + attributes: + label: Usage Mode + description: How are you running S3Mock? + options: + - Docker + - Testcontainers + - JUnit 5 Extension + - TestNG Listener + - Other + validations: + required: true + - type: textarea + id: description + attributes: + label: Description + description: A clear description of the bug. + placeholder: Describe what happened and what you expected to happen. + validations: + required: true + - type: textarea + id: steps + attributes: + label: Steps to Reproduce + description: How can we reproduce this issue? + placeholder: | + 1. Start S3Mock with ... + 2. Call API ... + 3. Observe ... + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What did you expect to happen? + validations: + required: true + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened? + validations: + required: true + - type: dropdown + id: sdk + attributes: + label: AWS SDK + description: Which AWS SDK are you using? + options: + - AWS SDK v2 (Java/Kotlin) + - AWS SDK v2 (Python/boto3) + - AWS SDK v2 (Other language) + - AWS CLI + - Other / Not applicable + - type: textarea + id: logs + attributes: + label: Relevant Log Output + description: Paste any relevant log output or error messages. + render: shell + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other context, screenshots, or configuration details. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 000000000..e9486e756 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: true +contact_links: + - name: S3Mock Documentation + url: https://github.com/adobe/S3Mock#readme + about: Check the README for usage instructions, configuration, and troubleshooting. + - name: AWS S3 API Reference + url: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html + about: Reference for S3 API operations and expected behavior. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 000000000..0390f3095 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,62 @@ +name: Feature Request +description: Suggest a new feature or enhancement for S3Mock +title: "[Feature]: " +labels: ["enhancement"] +assignees: + - afranken +body: + - type: markdown + attributes: + value: | + Thanks for suggesting an improvement! Please describe the feature you'd like to see. + - type: dropdown + id: category + attributes: + label: Feature Category + description: What area of S3Mock does this relate to? + options: + - New S3 API operation + - Improve existing S3 API operation + - Configuration / Setup + - Testcontainers integration + - JUnit 5 Extension + - Docker image + - Documentation + - Other + validations: + required: true + - type: input + id: s3-operation + attributes: + label: S3 API Operation + description: If this is about an S3 API operation, which one? + placeholder: "e.g., GetBucketEncryption, PutBucketTagging" + - type: textarea + id: description + attributes: + label: Description + description: A clear description of the feature you'd like. + validations: + required: true + - type: textarea + id: use-case + attributes: + label: Use Case + description: Describe the use case or problem this feature would solve. + validations: + required: true + - type: textarea + id: proposed-solution + attributes: + label: Proposed Solution + description: If you have ideas on how this could be implemented, describe them here. + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Any alternative solutions or workarounds you've considered. + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other context, references, or screenshots. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1f3f65ade..ef574418d 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -9,3 +9,6 @@ - [ ] I have signed the [CLA](http://adobe.github.io/cla.html). - [ ] I have written tests and verified that they fail without my change. +- [ ] I have run `./mvnw ktlint:format` to fix code style. +- [ ] I have updated `CHANGELOG.md` (if applicable). +- [ ] I have updated documentation (if applicable). diff --git a/.github/SECURITY.md b/.github/SECURITY.md new file mode 100644 index 000000000..ced54cc9f --- /dev/null +++ b/.github/SECURITY.md @@ -0,0 +1,48 @@ +# Security Policy + +## Important Notice + +S3Mock is a **testing tool** designed for local integration testing. It is **not intended for production use** and lacks the security features required for production environments. Do not expose S3Mock to untrusted networks or use it to store sensitive data. + +## Supported Versions + +| Version | Supported | +|---------|--------------------| +| 5.x | :white_check_mark: | +| 4.x | :x: | +| 3.x | :x: | +| < 3.0 | :x: | + +## Reporting a Vulnerability + +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, please report vulnerabilities using one of the following methods: + +1. **GitHub Security Advisories** (preferred): Use the [private vulnerability reporting](https://github.com/adobe/S3Mock/security/advisories/new) feature to submit a report directly through GitHub. + +2. **Email**: Send a report to [Grp-opensourceoffice@adobe.com](mailto:Grp-opensourceoffice@adobe.com). + +### What to Include + +- Type of vulnerability +- Affected version(s) +- Steps to reproduce +- Impact assessment +- Suggested fix (if any) + +### Response Timeline + +- **Acknowledgment**: Within 5 business days +- **Initial assessment**: Within 10 business days +- **Resolution**: Depending on severity and complexity + +## Security Measures + +S3Mock uses the following automated security tools: + +- **CodeQL**: Static analysis for security vulnerabilities (via GitHub Actions) +- **SBOM**: Software Bill of Materials generation for dependency tracking +- **OpenSSF Scorecard**: Security health assessment +- **Dependabot**: Automated dependency updates for Maven, Docker, and GitHub Actions +- **Dependency Review**: Automated review of dependency changes in PRs diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 730288265..c60aceaae 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -17,6 +17,22 @@ updates: # Check for updates once per day, Github defaults to random time every day. schedule: interval: daily + # Group related dependencies to reduce PR noise + groups: + aws-sdk: + patterns: + - "software.amazon.awssdk*" + - "aws.sdk.kotlin*" + spring: + patterns: + - "org.springframework*" + maven-plugins: + patterns: + - "org.apache.maven.plugins*" + kotlin: + patterns: + - "org.jetbrains.kotlin*" + - "org.jetbrains.kotlinx*" # Enable version updates for Docker - package-ecosystem: docker # Look for `Dockerfile` in the `/docker/` directory, that's where all versions are managed @@ -41,3 +57,10 @@ updates: # Check for updates once per day, Github defaults to random time every day. schedule: interval: daily + # Group related actions to reduce PR noise + groups: + github-actions: + patterns: + - "actions/*" + - "github/*" + - "step-security/*" diff --git a/.junie/guidelines.md b/.junie/guidelines.md deleted file mode 100644 index 7d0d5129a..000000000 --- a/.junie/guidelines.md +++ /dev/null @@ -1,386 +0,0 @@ -# S3Mock Development Guidelines (Concise) - -Essential info for working on S3Mock. This top section is a quick, no‑frills guide. Details follow below. - -Quickstart TL;DR -- Build (fast): ./mvnw clean install -DskipDocker -- Build (full): ./mvnw clean install -- Server tests only: ./mvnw -pl server test -- All tests incl. ITs (requires Docker): ./mvnw verify -- One server test: ./mvnw -pl server test -Dtest=ObjectStoreTest -- One IT: ./mvnw -pl integration-tests -am verify -Dit.test=BucketIT - -Requirements -- JDK 17+ -- Docker (only for integration tests and Docker image build) - -Common config (env vars) -- COM_ADOBE_TESTING_S3MOCK_STORE_REGION (default: us-east-1) -- COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS -- COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT (default: false) -- debug / trace (Spring Boot flags) - -Tips -- IDE runs: use server tests; ITs need Docker via Maven lifecycle. -- Debug server on 9090/9191; trust self‑signed cert or use HTTP. -- Before declaring done: run a full Maven build successfully. - -Troubleshooting -- Connection refused: ensure S3Mock is running on expected ports. -- SSL errors: trust self‑signed cert or switch to HTTP. -- Docker errors: ensure Docker is running and you have permissions. - -## Junie Operations Playbook (Critical) -To ensure tests execute successfully in this environment, follow these strict rules: - -- Default test scope: server module only. Do NOT run full project builds by default. -- Use the test tool, not shell, to run tests: - - Preferred: run_test on specific test files, e.g., "server/src/test/kotlin/com/adobe/testing/s3mock/store/ObjectStoreTest.kt". - - One test by name: use run_test with full path and the test method name parameter. - - Note: Directory-wide runs via run_test may not be supported in this environment. If you need to run all server tests, use Maven with: ./mvnw -pl server -DskipDocker test. -- Avoid integration tests unless explicitly requested and Docker availability is confirmed. If requested, run via Maven lifecycle only. -- If a build is required, prefer fast builds: - - Use ./mvnw -pl server -am -DskipDocker clean test or rely on run_test which compiles as needed. - - Only run ./mvnw clean install (full) when the user explicitly asks for a full build or cross-module changes demand it. -- Never run mvnw verify without confirming Docker is available; if not available, add -DskipDocker. -- Java 17+ required; if build fails due to JDK, report and stop, do not retry with different commands. -- Decision tree: - 1) Need to validate changes in server module? -> run_test on one or more specific test files (fast path). If you truly need all server tests, use: ./mvnw -pl server -DskipDocker test. - 2) Need a specific server test? -> run_test on that file. - 3) Need ITs and Docker is confirmed? -> mvnw -pl integration-tests -am verify; otherwise skip. - 4) Need a build artifact quickly? -> mvnw clean install -DskipDocker. - -Note: Always summarize which scope you ran and why. - -— - -## Build and Configuration Instructions - -### Building the Project - -S3Mock uses Maven for building. The project includes a Maven wrapper (`mvnw`) so you don't need to install Maven separately. - -#### Basic Build Commands - -```bash -# Build the entire project -./mvnw clean install - -# Build without running Docker (useful for quick builds) -./mvnw clean install -DskipDocker - -# Build a specific module -./mvnw clean install -pl server -am -``` - -#### Build Requirements - -- JDK 17 or higher -- Docker (for building the Docker image and running integration tests) - -### Configuration Options - -S3Mock can be configured with the following environment variables: - -| Environment Variable | Legacy Name | Description | Default | -|-------------------------------------------------------|-----------------------------------|-------------------------------------------------------|---------------------| -| `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` | `validKmsKeys` | List of KMS Key-Refs that are treated as valid | none | -| `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` | `initialBuckets` | List of bucket names that will be available initially | none | -| `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` | `COM_ADOBE_TESTING_S3MOCK_REGION` | The region the S3Mock is supposed to mock | `us-east-1` | -| `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` | `root` | Base directory for temporary files | Java temp directory | -| `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` | `retainFilesOnExit` | Set to `true` to keep files after shutdown | `false` | -| `debug` | - | Enable Spring Boot's debug output | `false` | -| `trace` | - | Enable Spring Boot's trace output | `false` | - -## Testing Information - -### Test Structure - -The project uses JUnit 5 for testing. There are two main types of tests in the project: - -1. **Integration Tests**: Located in the `integration-tests` module and written in Kotlin. These tests verify the end-to-end functionality of S3Mock by starting a Docker container and making actual HTTP requests to it. - -2. **Server Module Tests**: Located in the `server` module and primarily written in Kotlin. These include: - - Unit tests for utility classes and DTOs - - Spring Boot tests for controllers, services, and stores - - XML serialization/deserialization tests - -#### Integration Tests - -The main test base class for integration tests is `S3TestBase` which provides utility methods for: -- Creating S3 clients -- Managing buckets and objects -- Handling SSL certificates -- Generating test data - -#### Server Module Tests - -The server module contains several types of tests: - -1. **Controller Tests**: Use `@SpringBootTest` with `WebEnvironment.RANDOM_PORT` and `TestRestTemplate` to test HTTP endpoints. These tests mock the service layer using `@MockitoBean`. - -2. **Store Tests**: Use `@SpringBootTest` with `WebEnvironment.NONE` to test the data storage layer. These tests often use `@Autowired` to inject the component under test. - -3. **Service Tests**: Test the service layer with mocked dependencies. - -4. **Unit Tests**: Simple tests for utility classes and DTOs without Spring context. - -Common base classes and utilities: -- `BaseControllerTest`: Sets up XML serialization/deserialization for controller tests -- `StoreTestBase`: Common setup for store tests -- `ServiceTestBase`: Common setup for service tests - -### Running Tests - -#### Running Server Module Tests - -The server module tests can be run without Docker: - -```bash -# Run all server module tests -./mvnw -pl server test - -# Run a specific test class -./mvnw -pl server test -Dtest=ObjectStoreTest - -# Run a specific test method -./mvnw -pl server test -Dtest=ObjectStoreTest#testStoreObject -``` - -The server module uses Spring Test Profiler to analyze test performance. Test execution times are reported in the `target/spring-test-profiler` directory. - -#### Running Integration Tests - -Integration tests require Docker to be running as they start an S3Mock Docker container. The integration tests can only be executed through Maven because they need the S3Mock to run. The process works as follows: - -1. The Docker Maven plugin (io.fabric8:docker-maven-plugin) starts the S3Mock Docker container in the pre-integration-test phase -2. The Maven Failsafe plugin runs the integration tests against the running container -3. The Docker Maven plugin stops the container in the post-integration-test phase - -```bash -# Run all integration tests -./mvnw verify - -# Run a specific integration test -./mvnw -pl integration-tests -am verify -Dit.test=BucketIT - -# Run tests without Docker (will skip integration tests) -./mvnw verify -DskipDocker -``` - -Note that attempting to run integration tests directly from your IDE without starting the S3Mock Docker container will result in connection errors, as the tests expect S3Mock to be running on specific ports. - -### Writing New Tests - -#### Writing Integration Tests - -To create a new integration test: - -1. Create a new Kotlin class in the `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its` directory -2. Extend the `S3TestBase` class to inherit utility methods -3. Name your test class with an `IT` suffix to be recognized as an integration test -4. Use the provided S3 client methods to interact with S3Mock - -Example integration test: - -```kotlin -// ExampleIT.kt -internal class ExampleIT : S3TestBase() { - private val s3Client = createS3Client() - - @Test - fun testPutAndGetObject(testInfo: TestInfo) { - // Create a bucket - val bucketName = givenBucket(testInfo) - - // Create test content - val content = "This is a test file content" - val contentBytes = content.toByteArray(StandardCharsets.UTF_8) - - // Put an object into the bucket - val key = "test-object.txt" - val putObjectResponse = s3Client.putObject( - PutObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build(), - RequestBody.fromBytes(contentBytes) - ) - - // Verify the object was uploaded successfully - assertThat(putObjectResponse.eTag()).isNotBlank() - - // Get the object back - val getObjectResponse = s3Client.getObject( - GetObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - // Read the content and verify it matches what we uploaded - val retrievedContent = getObjectResponse.readAllBytes() - assertThat(retrievedContent).isEqualTo(contentBytes) - - // Clean up - s3Client.deleteObject { it.bucket(bucketName).key(key) } - } -} -``` - -#### Writing Server Module Tests - -The server module uses different testing approaches depending on what's being tested: - -1. **Controller Tests**: - - Extend `BaseControllerTest` to inherit XML serialization setup - - Use `@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)` - - Use `@MockitoBean` to mock service dependencies - - Inject `TestRestTemplate` to make HTTP requests to the controller - -Example controller test: - -```kotlin -// BucketControllerTest.kt -@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) -@MockitoBean(classes = [BucketService::class, ObjectService::class, MultipartService::class]) -internal class BucketControllerTest : BaseControllerTest() { - @Autowired - private lateinit var restTemplate: TestRestTemplate - - @MockitoBean - private lateinit var bucketService: BucketService - - @Test - fun testListBuckets() { - // Mock service response - whenever(bucketService.listBuckets()).thenReturn(givenBuckets(2)) - - // Make HTTP request - val response = restTemplate.getForEntity("/", ListAllMyBucketsResult::class.java) - - // Verify response - assertThat(response.statusCode).isEqualTo(HttpStatus.OK) - assertThat(response.body.buckets.bucket).hasSize(2) - } -} -``` - -2. **Store Tests**: - - Extend `StoreTestBase` for common setup - - Use `@SpringBootTest(webEnvironment = WebEnvironment.NONE)` - - Use `@Autowired` to inject the component under test - -Example store test: - -```kotlin -// ObjectStoreTest.kt -@SpringBootTest(classes = [StoreConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) -@MockitoBean(classes = [KmsKeyStore::class, BucketStore::class]) -internal class ObjectStoreTest : StoreTestBase() { - @Autowired - private lateinit var objectStore: ObjectStore - - @Test - fun testStoreAndGetObject() { - // Test storing and retrieving an object - val sourceFile = File(TEST_FILE_PATH) - val id = UUID.randomUUID().toString() - val name = sourceFile.name - - objectStore.storeS3ObjectMetadata( - metadataFrom("test-bucket"), id, name, "text/plain", - emptyMap(), sourceFile.toPath(), - emptyMap(), emptyMap(), null, emptyList(), - null, null, Owner.DEFAULT_OWNER, StorageClass.STANDARD - ) - - val metadata = objectStore.getS3ObjectMetadata(metadataFrom("test-bucket"), id, null) - - assertThat(metadata.key).isEqualTo(name) - assertThat(metadata.contentType).isEqualTo("text/plain") - } -} -``` - -3. **Unit Tests**: - - Use standard JUnit 5 tests without Spring context - - Focus on testing a single class or method in isolation - -Example unit test: - -```kotlin -// DigestUtilTest.kt -internal class DigestUtilTest { - @Test - fun testHexDigest() { - val input = "test data".toByteArray() - val expected = DigestUtils.md5Hex(input) - - assertThat(DigestUtil.hexDigest(input)).isEqualTo(expected) - } -} -``` - -## Additional Development Information - -### Project Structure - -S3Mock is a multi-module Maven project: - -- `build-config`: Build configuration files -- `docker`: Docker image build module -- `integration-tests`: Integration tests -- `server`: Core S3Mock implementation -- `testsupport`: Test support modules for different testing frameworks - -### Code Style - -The project uses Checkstyle for Java code style checking. The configuration is in `build-config/checkstyle.xml`. - -### License Headers - -S3Mock uses the Apache License 2.0 and enforces proper license headers in all source files through the `license-maven-plugin`. Important rules: - -- All source files must include the proper license header -- When modifying a file, the copyright year range in the header must be updated to include the current year -- The format is: `Copyright 2017-YYYY Adobe.` where YYYY is the current year -- The license check runs automatically during the build process -- To fix license headers, run: `./mvnw license:format` -- To check license headers without modifying files: `./mvnw license:check` - -### Debugging - -To debug the S3Mock server: - -1. Run the S3MockApplication class in your IDE with debug mode -2. The server will start on ports 9090 (HTTP) and 9191 (HTTPS) -3. Configure your S3 client to connect to these ports - -Alternatively, you can run the Docker container with debug enabled: - -```bash -docker run -p 9090:9090 -p 9191:9191 -e debug=true -t adobe/s3mock -``` - -### Common Issues - -1. **Connection refused errors**: Ensure the S3Mock server is running and the ports are correctly configured. - -2. **SSL certificate errors**: S3Mock uses a self-signed certificate. Configure your client to trust all certificates or use HTTP instead. - -3. **Docker-related errors**: Make sure Docker is running, and you have permissions to create containers. - -### Recommended Development Workflow - -1. Make changes to the code -2. Validate changes with server module tests first (fast path) - - Use the run_test tool on "server/src/test" or on a specific test file/method. - - Prefer this over invoking Maven directly; run_test compiles as needed. -3. Only run a full Maven build when explicitly requested or when cross-module changes demand it - - If building in this environment, prefer fast builds: ./mvnw -pl server -am -DskipDocker clean test - - Do not run mvnw verify unless Docker is confirmed; otherwise add -DskipDocker -4. Run integration tests only when Docker availability is confirmed and when explicitly requested - - Execute via Maven lifecycle: ./mvnw -pl integration-tests -am verify (or add -DskipDocker to skip ITs) -5. Optionally build the Docker image to verify packaging when needed -6. Test with your application to verify real-world usage diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..15f5da4b7 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,194 @@ +# Agent Context for S3Mock + +Lightweight S3 API mock server for local integration testing. + +## Tech Stack +- **Kotlin 2.3** (JVM 17), Spring Boot 4.0.x, Maven 3.9+ +- **Testing**: JUnit 5, Mockito, AssertJ, Testcontainers +- **Container**: Docker/Alpine + +## Structure +``` +server/ # Core implementation (Controller→Service→Store) +integration-tests/ # AWS SDK integration tests +testsupport/ # JUnit 5, Testcontainers, TestNG integrations +build-config/ # Shared build configuration +docker/ # Docker image build +``` + +## Architecture + +**Layered**: Controller (REST) → Service (logic) → Store (filesystem) + +**Key packages**: `controller/`, `service/`, `store/`, `dto/` + +## DO / DON'T + +### DO +- Use **constructor injection** for all Spring beans (in production code) +- Use **data classes** for DTOs with Jackson XML annotations +- Use **Kotlin stdlib** and built-in language features over third-party utilities +- Use **AWS SDK v2** for all new integration tests +- Use **JUnit 5** for all new tests +- Use **`@SpringBootTest`** with **`@MockitoBean`** for unit tests — this is the project's standard mocking approach +- Use **expression bodies** for simple functions +- Use **null safety** (`?`, `?.`, `?:`) instead of null checks +- **Name the `it` parameter** in nested lambdas, loops, and scope functions to avoid shadowing: `.map { part -> ... }` instead of `.map { it.name }` +- Match **AWS S3 API naming exactly** in Jackson XML annotations (`localName = "..."`) +- Keep tests **independent** — each test creates its own resources (UUID bucket names) +- Use **backtick test names** with descriptive sentences: `` fun `should create bucket successfully`() `` +- Mark test classes as **`internal`**: `internal class ObjectServiceTest` +- **Refactor** legacy `testSomething` camelCase names to backtick style when touching existing tests +- Validate XML serialization against [AWS S3 API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) + +### DON'T +- DON'T use `@Autowired` or field injection in production code — always use constructor injection +- DON'T use `var` for public API properties — prefer `val` (immutability) +- DON'T use AWS SDK v1 — it has been removed in 5.x +- DON'T use JUnit 4 — it has been removed in 5.x +- DON'T use `@ExtendWith(MockitoExtension::class)` or `@Mock` / `@InjectMocks` — use `@SpringBootTest` with `@MockitoBean` instead +- DON'T add Apache Commons dependencies — use Kotlin stdlib equivalents +- DON'T put business logic in controllers — controllers only map HTTP, delegate to services +- DON'T return raw strings from controllers — use typed DTOs for XML/JSON responses +- DON'T declare dependency versions in sub-module POMs — all versions are managed in root `pom.xml` +- DON'T share mutable state between tests — each test must be self-contained +- DON'T hardcode bucket names in tests — use `UUID.randomUUID()` for uniqueness +- DON'T use legacy `testSomething` camelCase naming for new tests — use backtick names instead +- DON'T add new functionality to deprecated modules (`junit4/`) + +## Code Style + +**Kotlin idioms**: Data classes for DTOs, null safety, expression bodies, constructor injection + +**Spring**: `@RestController`, `@Service`, `@Component`, constructor injection over field injection + +**Example**: +```kotlin +@RestController +class ObjectController(private val objectService: ObjectService) { + @GetMapping("/{bucketName:.+}/{*key}") + fun getObject(@PathVariable bucketName: String, @PathVariable key: String) = + objectService.getObject(bucketName, key).let { + ResponseEntity.ok().header("ETag", it.etag).body(it.data) + } +} +``` + +## XML Serialization + +Jackson XML with AWS-compatible structure. Key annotations: +- `@JacksonXmlRootElement(localName = "...")` +- `@JacksonXmlProperty(localName = "...")` +- `@JacksonXmlElementWrapper(useWrapping = false)` for collections + +**Important**: XML element and attribute names must match the AWS S3 API specification exactly. +Verify against [AWS API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) +and existing integration tests. A compile-check is not sufficient — always verify that serialized +XML output matches the expected AWS response format by running integration tests. + +## Storage + +Filesystem layout: +``` +//bucketMetadata.json +///binaryData + objectMetadata.json +///-binaryData # versioning +//multiparts//.part +``` + +## Configuration + +Environment variables (prefix: `COM_ADOBE_TESTING_S3MOCK_STORE_`): +- `ROOT` - storage directory +- `RETAIN_FILES_ON_EXIT` - keep files after shutdown +- `REGION` - AWS region (default: us-east-1) +- `INITIAL_BUCKETS` - comma-separated bucket names +- `VALID_KMS_KEYS` - valid KMS ARNs + +## Error Handling + +`S3Exception` constants: `NO_SUCH_BUCKET`, `NO_SUCH_KEY`, `BUCKET_ALREADY_OWNED_BY_YOU`, `INVALID_BUCKET_NAME`, etc. + +HTTP codes: 200, 204, 404, 409, 500 + +## Testing + +- Unit tests: `@SpringBootTest` with `@MockitoBean`, suffix `Test` +- Integration tests: Real AWS SDK v2 against Docker container, suffix `IT` +- Test independence: Each test self-contained +- Name the class under test **`iut`** (implementation under test): `private lateinit var iut: ObjectService` +- **Base classes** — always extend the appropriate base: + - `ServiceTestBase` for service tests + - `StoreTestBase` for store tests + - `BaseControllerTest` for controller tests + - `S3TestBase` for integration tests + +## Build + +```bash +./mvnw clean install # Full build +./mvnw clean install -DskipDocker # Skip Docker +./mvnw verify -pl integration-tests +./mvnw ktlint:format +``` + +## CI/CD Pipeline + +All PRs and pushes are validated by the `maven-ci-and-prb.yml` GitHub Actions workflow. + +**Required gates** (all must pass before merge): +1. Compilation and build (`./mvnw clean install`) +2. Unit tests (`*Test.kt` in each module) +3. Integration tests (`*IT.kt` against Docker container) +4. ktlint (Kotlin code style) +5. Checkstyle (Java/XML code style, config in `etc/checkstyle.xml`) +6. Docker image build (unless `-DskipDocker`) + +**Additional workflows**: CodeQL (security scanning), SBOM (dependency tracking), OpenSSF Scorecard, Dependabot (automated dependency updates), Stale issue management. + +## Dependency Management + +- **All versions** are declared in the root `pom.xml` `` section +- Sub-modules inherit versions — never declare versions in sub-module POMs +- **BOMs** are preferred for multi-artifact dependencies (Kotlin BOM, Spring Boot BOM, AWS SDK BOM) +- Prefer Kotlin stdlib / JDK APIs over adding new third-party libraries +- Dependabot manages automated version updates for Maven, Docker, and GitHub Actions + +## PR & Commit Conventions + +- PRs should target `main` (active development) or version maintenance branches +- Reference related GitHub issues in PR description +- Update `CHANGELOG.md` under the current version section for user-facing changes +- Ensure all CI gates pass before requesting review +- See [PR template](.github/PULL_REQUEST_TEMPLATE.md) and [Contributing Guide](.github/CONTRIBUTING.md) + +## Version & Branch Strategy + +- **`main`** — active development for the current major version (5.x) +- **Version branches** (`s3mock-v2`, `s3mock-v3`, `s3mock-v4`) — maintenance for previous major versions +- **Tags** follow semver: `5.0.0`, `4.11.0`, etc. +- **6.x** is planned after Spring Boot 5.x — will remove JUnit/TestNG modules and target JDK 25 LTS bytecode + +## Constraints + +- Path-style URLs only (not `bucket.localhost`) +- Presigned URLs accepted but not validated +- Self-signed SSL certificate +- KMS validation only, no encryption +- Not for production + +## Common Patterns + +```kotlin +// ETag (using project's own DigestUtil, not Apache Commons) +val etag = DigestUtil.hexDigest(data) + +// Response +ResponseEntity.ok() + .eTag(normalizeEtag(s3ObjectMetadata.etag)) + .header("Last-Modified", lastModified) + .body(data) + +// Dates +Instant.now().toString() // ISO 8601 +``` diff --git a/CHANGELOG.md b/CHANGELOG.md index a8c8a6a3b..ea7c9b644 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # Changelog S3Mock follows [Semantic Versioning](https://semver.org/). It depends on lots of 3rd party libraries which are updated regularly. -Whenever a 3rd party library is updated, S3Mock will update it's MINOR version. +Whenever a 3rd party library is updated, S3Mock will update its MINOR version. * [Changelog](#changelog) @@ -115,31 +115,27 @@ Whenever a 3rd party library is updated, S3Mock will update it's MINOR version. # PLANNED - 6.x - RELEASE TBD -Version 6.x is JDK25 LTS bytecode compatible, with Docker integration. +Version 6.x will target JDK 25 LTS bytecode compatibility with Docker-only integration. -Will be released after Spring Boot 5.x, updating baselines etc. as Spring Boot 5.x requires. +Will be released after Spring Boot 5.x becomes available. -Any JUnit / direct Java usage support will most likely be dropped and only supported on a best-effort basis. -(i.e., the modules will be deleted from the code base and not released anymore. It *may* be possible to -still run S3Mock directly in Java.) -The S3Mock is a Spring Boot application and currently contains various workarounds to make it possible -to easily to run `S3MockApplication#start` from a static context. These workarounds will be deleted. +**Breaking Changes:** +- **JUnit/TestNG direct integration removed.** Test framework modules (junit4, junit5, testng) will be deleted from the codebase and no longer released. Running S3Mock embedded in the same JVM will no longer be officially supported. +- **TestContainers becomes the recommended testing approach.** Use [TestContainers](https://www.testcontainers.org/) to run S3Mock as a Docker container in your tests. +- **Standard Spring Boot application.** S3Mock will be refactored to a standard Spring Boot application, removing workarounds that enabled `S3MockApplication#start()` in static contexts. -Running S3Mock in unit tests is still supported by using [TestContainers](https://www.testcontainers.org/). - -**Once 6.x is released, 5.x may receive bug fixes and features. This will be best-effort only.** +**Support:** Once 6.x is released, version 5.x may receive bug fixes and features on a best-effort basis only. ## Planned changes -* Features and fixes +* **Features and fixes** * TBD -* Refactorings - * Looking to Remove unit test modules. This enables - * Refactoring S3Mock to a "standard" Spring Boot application. - * Removal of workarounds to use `S3MockApplication#start` from a static context -* Version updates - * Bump Spring Boot version to 5.0.0 - * Bump Spring Framework version to 8.0.0 +* **Refactorings** + * Remove unit test modules (junit4, junit5, testng) + * Refactor to standard Spring Boot application without static context workarounds +* **Version updates** + * Spring Boot 5.0.0 + * Spring Framework 8.0.0 # CURRENT - 5.x - THIS VERSION IS UNDER ACTIVE DEVELOPMENT Version 5.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration. @@ -183,13 +179,17 @@ Version 5.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Jav * Bump org.apache.maven.plugins:maven-release-plugin from 3.3.0 to 3.3.1 * Bump org.codehaus.mojo:exec-maven-plugin from 3.6.2 to 3.6.3 * Bump org.apache.maven.plugins:maven-compiler-plugin from 3.14.1 to 3.15.0 + * Bump org.apache.maven.plugins:maven-dependency-plugin from 3.9.0 to 3.10.0 + * Bump io.fabric8:docker-maven-plugin from 0.48.0 to 0.48.1 * Bump digital.pragmatech.testing:spring-test-profiler from 0.0.14 to 0.0.15 * Bump com.puppycrawl.tools:checkstyle from 12.2.0 to 13.2.0 - * Bump actions/upload-artifact from 5.0.0 to 6.0.0 * Bump actions/checkout from 6.0.1 to 6.0.2 - * Bump github/codeql-action from 4.31.6 to 4.32.2 + * Bump actions/dependency-review-action from 4.8.2 to 4.8.3 * Bump actions/setup-java from 5.0.0 to 5.2.0 - * Bump step-security/harden-runner from 2.13.3 to 2.14.1 + * Bump actions/stale from 10.1.1 to 10.2.0 + * Bump actions/upload-artifact from 5.0.0 to 6.0.0 + * Bump github/codeql-action from 4.31.6 to 4.32.4 + * Bump step-security/harden-runner from 2.13.3 to 2.14.2 # DEPRECATED - 4.x Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration. diff --git a/README.md b/README.md index 21ac9c395..c5a3328ee 100755 --- a/README.md +++ b/README.md @@ -2,89 +2,103 @@ ![Maven Build](https://github.com/adobe/S3Mock/workflows/Maven%20Build/badge.svg) [![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/) [![Docker Pulls](https://img.shields.io/docker/pulls/adobe/s3mock)](https://hub.docker.com/r/adobe/s3mock) -[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#Kotlin) -[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#Java) +[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#build--run) +[![JVM](https://img.shields.io/badge/runs%20on-JVM-RED.svg)](#build--run) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7673/badge)](https://bestpractices.coreinfrastructure.org/projects/7673) [![OpenSSF Scorecard](https://img.shields.io/ossf-scorecard/github.com/adobe/S3Mock?label=openssf%20scorecard&style=flat)](https://api.securityscorecards.dev/projects/github.com/adobe/S3Mock) [![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/) * [S3Mock](#s3mock) + * [Quick Start](#quick-start) * [Changelog](#changelog) - * [Supported S3 operations](#supported-s3-operations) + * [Version Compatibility](#version-compatibility) + * [Migration Guides](#migration-guides) + * [4.x to 5.x (Current)](#4x-to-5x-current) + * [3.x to 4.x](#3x-to-4x) + * [Supported S3 Operations](#supported-s3-operations) * [Usage](#usage) - * [Usage of AWS S3 SDKs](#usage-of-aws-s3-sdks) - * [Path-style vs. Domain-style access](#path-style-vs-domain-style-access) - * [Presigned URLs](#presigned-urls) - * [Self-signed SSL certificate](#self-signed-ssl-certificate) - * [Usage of AWS CLI](#usage-of-aws-cli) - * [Create bucket](#create-bucket) - * [Put object](#put-object) - * [Get object](#get-object) - * [Get object using HTTPS](#get-object-using-https) - * [Usage of plain HTTP / HTTPS with cURL](#usage-of-plain-http--https-with-curl) - * [Create bucket](#create-bucket-1) - * [Put object](#put-object-1) - * [Get object](#get-object-1) - * [Get object using HTTPS](#get-object-using-https-1) - * [S3Mock configuration options](#s3mock-configuration-options) - * [S3Mock Docker](#s3mock-docker) - * [Start using the command-line](#start-using-the-command-line) - * [Start using the Fabric8 Docker-Maven-Plugin](#start-using-the-fabric8-docker-maven-plugin) - * [Start using Testcontainers](#start-using-testcontainers) - * [Start using Docker compose](#start-using-docker-compose) - * [Simple example](#simple-example) - * [Expanded example](#expanded-example) - * [Start using a self-signed SSL certificate](#start-using-a-self-signed-ssl-certificate) - * [S3Mock Java](#s3mock-java) - * [Start using the JUnit5 Extension](#start-using-the-junit5-extension) - * [Start using the TestNG Listener](#start-using-the-testng-listener) - * [Start programmatically](#start-programmatically) + * [Docker (Recommended)](#docker-recommended) + * [Testcontainers](#testcontainers) + * [JUnit 5 Extension](#junit-5-extension) + * [TestNG Listener](#testng-listener) + * [AWS CLI](#aws-cli) + * [cURL](#curl) + * [Configuration](#configuration) + * [Important Limitations](#important-limitations) + * [Troubleshooting](#troubleshooting) * [File System Structure](#file-system-structure) - * [Root-Folder](#root-folder) - * [Buckets](#buckets) - * [Objects](#objects) - * [Object versions](#object-versions) - * [Multipart Uploads](#multipart-uploads) + * [Performance & Resources](#performance--resources) + * [Architecture & Development](#architecture--development) * [Build & Run](#build--run) - * [Build](#build) - * [With Docker](#with-docker) - * [Without Docker](#without-docker) - * [Run](#run) - * [Spring Application](#spring-application) - * [Docker](#docker) - * [Docker Maven plugin](#docker-maven-plugin) - * [Run integration tests](#run-integration-tests) - * [Java](#java) - * [Kotlin](#kotlin) - * [Governance model](#governance-model) - * [Vulnerability reports](#vulnerability-reports) - * [Security](#security) * [Contributing](#contributing) - * [Licensing](#licensing) - * [Powered by](#powered-by) - * [Star History](#star-history) + * [License](#license) ## S3Mock -`S3Mock` is a lightweight server that implements parts of -the [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html). -It has been created to support local integration testing by reducing infrastructure dependencies. +S3Mock is a lightweight server implementing parts of the [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) for local integration testing. It eliminates the need for actual AWS infrastructure during development and testing. -The `S3Mock` server can be started as a standalone *Docker* container, using *Testcontainers*, *JUnit5* and *TestNG* support, or programmatically. +**Recommended usage**: Run S3Mock as a Docker container or with Testcontainers to avoid classpath conflicts. + +## Quick Start + +Get up and running in 30 seconds: + +```shell +# 1. Start S3Mock +docker run -p 9090:9090 adobe/s3mock + +# 2. Create a bucket +aws s3api create-bucket --bucket my-bucket --endpoint-url http://localhost:9090 + +# 3. Upload a file +aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url http://localhost:9090 + +# 4. Download the file +aws s3api get-object --bucket my-bucket --key my-file --endpoint-url http://localhost:9090 output-file +``` + +For programmatic testing, see [Testcontainers](#testcontainers) or [JUnit 5 Extension](#junit-5-extension) below. ## Changelog -See [GitHub releases](https://github.com/adobe/S3Mock/releases). -See also the [changelog](CHANGELOG.md) for detailed information about changes in releases and changes planned for future -releases. +- [GitHub Releases](https://github.com/adobe/S3Mock/releases) +- [Detailed Changelog](CHANGELOG.md) + +## Version Compatibility + +| S3Mock | Status | Spring Boot | Kotlin | Java (target) | Java (compile) | AWS SDK v2 | Testcontainers | +|--------|-------------|-------------|---------|---------------|----------------|------------|----------------| +| 5.x | **Active** | 4.0.x | 2.3 | 17 | 25 | 2.x | 2.x | +| 4.x | Deprecated | 3.x | 2.1-2.2 | 17 | 17 | 2.x | 1.x | +| 3.x | Deprecated | 2.x | 1.x-2.0 | 17 | 17 | 2.x | 1.x | +| 2.x | End of Life | 2.x | - | 11 | 11 | 1.x/2.x | - | + +## Migration Guides + +### 4.x to 5.x (Current) +- **Jackson 3**: XML annotations updated to Jackson 3 (`tools.jackson` packages) +- **AWS SDK v1 removed**: All v1 client support has been dropped +- **JUnit 4 removed**: The `s3mock-junit4` module no longer exists +- **Controller package moved**: `com.adobe.testing.s3mock` to `com.adobe.testing.s3mock.controller` +- **Legacy properties removed**: Old-style configuration properties have been removed +- **Apache Commons removed**: `commons-compress`, `commons-codec`, `commons-lang3` replaced by Kotlin/Java stdlib +- **Owner DisplayName removed**: AWS APIs stopped returning `DisplayName` - this is a file system breaking change for existing data + +### 3.x to 4.x +- **Tomcat replaces Jetty**: Application container changed from Jetty to Tomcat +- **Versioning API**: Basic support for S3 versioning added +- **If-(Un)modified-Since**: Conditional request handling implemented -## Supported S3 operations +For full details, see the [Changelog](CHANGELOG.md). -Of -these [operations of the Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html), -all marked :white_check_mark: are supported by S3Mock: +## Supported S3 Operations + +See the [complete operations table](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) in AWS documentation. + +
+Click to expand operations table (operations marked :white_check_mark: are supported) | Operation | Support | Comment | |-----------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|------------------------| @@ -187,409 +201,97 @@ all marked :white_check_mark: are supported by S3Mock: | [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) | :white_check_mark: | | | [WriteGetObjectResponse](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html) | :x: | | -## Usage - -### Usage of AWS S3 SDKs - -S3Mock can be used with any of the available AWS S3 SDKs. - -The [Integration Tests](integration-tests) and tests in [testsupport](testsupport) contain various examples of how to use the S3Mock with the AWS SDK for Java -v1 and v2 in Kotlin. - -`S3Client` or `S3Presigner` instances are created here: - -* Kotlin: [Integration Test base class](integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt) -* Java: [S3Mock unit test starter base class](testsupport/common/src/main/java/com/adobe/testing/s3mock/testsupport/common/S3MockStarter.java) - -#### Path-style vs. Domain-style access - -AWS S3 SDKs usually use domain-style access by default. Configuration is needed for path-style access. - -S3Mock currently only supports path-style access (e.g., `http://localhost:9090/bucket/someKey`). - -Domain-style access to buckets (e.g., `http://bucket.localhost:9090/someKey`) does not work because the domain -`localhost` is special and does not allow for subdomain access without -modifications to the operating system. - -#### Presigned URLs - -S3 SDKs can be used -to [create presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html), the S3 API -supports access through those URLs. - -S3Mock will accept presigned URLs, but it *ignores all parameters*. -For instance, S3Mock does not verify the HTTP verb that the presigned uri was created with, and it does not validate -whether the link is expired or not. - -S3 SDKs can be used to create presigned URLs pointing to S3Mock if they're configured for path-style access. See the -"Usage..." section above for links to examples on how to use the SDK with presigned URLs. - -#### Self-signed SSL certificate - -S3Mock supports connections via HTTP and HTTPS. It includes a self-signed SSL certificate which is rejected by most HTTP -clients by default. -To use HTTPS, the client must accept the self-signed certificate. - -On command line, this can be done by setting the `--no-verify-ssl` option in the AWS CLI or by using the `--insecure` -option in cURL, see below. - -Java and Kotlin SDKs can be configured to trust any SSL certificate, see links to `S3Client` creation above. - -### Usage of AWS CLI - -S3Mock can be used with the AWS CLI. Setting the `--endpoint-url` enables path-style access, `--no-verify-ssl` is needed -for HTTPS access. - -Examples: - -#### Create bucket - -```shell -aws s3api create-bucket --bucket my-bucket --endpoint-url=http://localhost:9090 -``` - -#### Put object - -```shell -aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url=http://localhost:9090 -``` - -#### Get object - -```shell -aws s3api get-object --bucket my-bucket --key my-file --endpoint-url=http://localhost:9090 my-file-output -``` - -#### Get object using HTTPS - -```shell -aws s3api get-object --bucket my-bucket --key my-file --no-verify-ssl --endpoint-url=https://localhost:9191 my-file-output -``` - -### Usage of plain HTTP / HTTPS with cURL +
-As long as the requests work with the S3 API, they will work with S3Mock as well. Use `--insecure` to ignore SSL errors. - -Examples: - -#### Create bucket - -```shell -curl --request PUT "http://localhost:9090/my-test-bucket/" -``` - -#### Put object - -```shell -curl --request PUT --upload-file ./my-file http://localhost:9090/my-test-bucket/my-file -``` - -#### Get object - -```shell -curl --request GET http://localhost:9090/my-test-bucket/my-file -O -``` - -#### Get object using HTTPS +## Usage -```shell -curl --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -``` +### Docker (Recommended) -### S3Mock configuration options - -The mock can be configured with the following environment variables: - -- `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS`: list of KMS Key-Refs that are to be treated as *valid*. - - Default: none - - KMS keys must be configured as valid ARNs in the format of "`arn:aws:kms:region:acct-id:key/key-id`", for example " - `arn:aws:kms:us-east-1:1234567890:key/valid-test-key-id`" - - The list must be comma separated keys like `arn-1, arn-2` - - When requesting with KMS encryption, the key ID is passed to the SDK / CLI, in the example above this would be " - `valid-test-key-id`". - - *S3Mock does not implement KMS encryption*, if a key ID is passed in a request, S3Mock will just validate if a given - Key was configured during startup and reject the request if the given Key was not configured. -- `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS`: list of names for buckets that will be available initially. - - Default: none - - The list must be comma separated names like `bucketa, bucketb` -- `COM_ADOBE_TESTING_S3MOCK_STORE_REGION`: the region the S3Mock is supposed to mock. - - Default: `us-east-1` - - Value must be a valid AWS region identifier like `eu-west-1` -- `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT`: the base directory to place the temporary files exposed by the mock. If S3Mock is started in Docker, a volume - must be mounted as the `root` directory, see examples below. - - Default: Java temp directory -- `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT`: set to `true` to let S3Mock keep all files that were created during its lifetime. Default is - `false`, all files are removed if S3Mock shuts down. - - Default: false -- `COM_ADOBE_TESTING_S3MOCK_CONTROLLER_CONTEXT_PATH`: the base context path for all controllers. - Use if you start S3Mock within your application's unit tests and need to separate the context paths. - - Default: "" -- `debug`: set to `true` to - enable [Spring Boot's debug output](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.logging.console-output). -- `trace`: set to `true` to - enable [Spring Boot's trace output](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.logging.console-output). - -### S3Mock Docker - -The `S3Mock` Docker container is the recommended way to use `S3Mock`. -It is released to [Docker Hub](https://hub.docker.com/r/adobe/s3mock). -The container is lightweight, built on top of the official [Linux Alpine image](https://hub.docker.com/_/alpine). - -If needed, -configure [memory](https://docs.docker.com/engine/reference/commandline/run/#specify-hard-limits-on-memory-available-to-containers--m---memory) -and [cpu](https://docs.docker.com/engine/reference/commandline/run/#options) limits for the S3Mock Docker container. - -The JVM will automatically use half the available memory. - -#### Start using the command-line - -Starting on the command-line: +The Docker image is available on [Docker Hub](https://hub.docker.com/r/adobe/s3mock) and is the recommended way to run S3Mock. +**Basic usage:** ```shell -docker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock ``` -The port `9090` is for HTTP, port `9191` is for HTTPS. - -Example with configuration via environment variables: +Ports: `9090` (HTTP), `9191` (HTTPS) +**With configuration:** ```shell -docker run -p 9090:9090 -p 9191:9191 -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test -e debug=true -t adobe/s3mock -``` - -#### Start using the Fabric8 Docker-Maven-Plugin - -Our [integration tests](integration-tests) are using the Amazon S3 Client to verify the server functionality against the -S3Mock. During the Maven build, the Docker image is started using the [docker-maven-plugin](https://dmp.fabric8.io/) and -the corresponding ports are passed to the JUnit test through the `maven-failsafe-plugin`. See [`BucketIT`](integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/BucketIT.kt) -as an example on how it's used in the code. - -This way, one can easily switch between calling the S3Mock or the real S3 endpoint, and this doesn't add any additional -Java dependencies to the project. - -#### Start using Testcontainers - -The [`S3MockContainer`](testsupport/testcontainers/src/main/java/com/adobe/testing/s3mock/testcontainers/S3MockContainer.java) -is a `Testcontainer` implementation that comes pre-configured exposing HTTP and HTTPS ports. Environment variables can -be set on startup. - -The example [`S3MockContainerJupiterTest`](testsupport/testcontainers/src/test/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainerJupiterTest.kt) -demonstrates the usage with JUnit 5. The example [`S3MockContainerManualTest`](testsupport/testcontainers/src/test/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainerManualTest.kt) -demonstrates the usage with plain Kotlin. Java will be similar. - -Testcontainers provide integrations for JUnit 4, JUnit 5 and Spock. -For more information, visit the [Testcontainers](https://www.testcontainers.org/) website. - -To use the [ -`S3MockContainer`](testsupport/testcontainers/src/main/java/com/adobe/testing/s3mock/testcontainers/S3MockContainer.java), -use the following Maven artifact in `test` scope: - -```xml - - com.adobe.testing - s3mock-testcontainers - ... - test - +docker run -p 9090:9090 -p 9191:9191 \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test-bucket \ + -e debug=true \ + adobe/s3mock ``` -#### Start using Docker compose - -##### Simple example - -Create a file `docker-compose.yml` - +**Docker Compose:** ```yaml services: s3mock: image: adobe/s3mock:latest environment: - - COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=bucket1 + - COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=bucket1,bucket2 ports: - 9090:9090 + - 9191:9191 ``` -Start with - -```shell -docker compose up -d -``` - -Stop with - -```shell -docker compose down -``` - -##### Expanded example - -Suppose we want to see what S3Mock is persisting and look at the logs it generates in detail. - -A local directory is needed, let's call it `locals3root`. This directory must be mounted as a volume into the Docker -container when it's started, and that mounted volume must then be configured as the `root` for S3Mock. Let's call the -mounted volume inside the container `containers3root`. S3Mock will delete all files when it shuts down, -`COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true` tells it to leave all files instead. - -Also, to see debug logs, `debug=true` must be configured for S3Mock. - -Create a file `docker-compose.yml` - +**With persistent storage:** ```yaml services: s3mock: image: adobe/s3mock:latest environment: - - debug=true - - COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true - COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3root - ports: - - 9090:9090 - - 9191:9191 - volumes: - - ./locals3root:/containers3root -``` - -Create a directory `locals3root`. - -Start with - -```shell -docker compose up -d -``` - -Create a bucket "my-test-bucket" with - -```shell -curl --request PUT "http://localhost:9090/my-test-bucket/" -``` - -Stop with - -```shell -docker compose down -``` - -Look into the directory `locals3root` where metadata and contents of the bucket are stored. - -```shell -$ mkdir s3mock-mounttest -$ cd s3mock-mounttest -$ mkdir locals3root -$ cat docker-compose.yml -services: - s3mock: - image: adobe/s3mock:latest - environment: - - debug=true - COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true - - COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3root ports: - 9090:9090 - - 9191:9191 volumes: - ./locals3root:/containers3root - -$ docker compose up -d -[+] Running 2/2 - ✔ Network s3mock-mounttest_default Created - ✔ Container s3mock-mounttest-s3mock-1 Started -$ curl --request PUT "http://localhost:9090/my-test-bucket/" -$ docker compose down -[+] Running 2/0 - ✔ Container s3mock-mounttest-s3mock-1 Removed - ✔ Network s3mock-mounttest_default Removed - -$ ls locals3root -my-test-bucket -$ ls locals3root/my-test-bucket -bucketMetadata.json ``` -#### Start using a self-signed SSL certificate +### Testcontainers -S3Mock includes a self-signed SSL certificate: +The [`S3MockContainer`](testsupport/testcontainers/src/main/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainer.kt) provides a pre-configured Testcontainers implementation. -```shell -$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -[...] -* Server certificate: -* subject: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock -* start date: Jul 25 12:28:53 2022 GMT -* expire date: Nov 25 12:28:53 3021 GMT -* issuer: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock -* SSL certificate verify result: self signed certificate (18), continuing anyway. -[...] +**Maven dependency:** +```xml + + com.adobe.testing + s3mock-testcontainers + ... + test + ``` -To use a custom self-signed SSL certificate, derive your own Docker container from the S3Mock container: +**Usage example:** +```kotlin +@Testcontainers +class MyTest { + @Container + val s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") -```dockerfile -FROM adobe/s3mock:4.2.0 + @Test + fun test() { + val s3Client = S3Client.builder() + .endpointOverride(URI.create(s3Mock.httpEndpoint)) + .region(Region.US_EAST_1) + .credentialsProvider(StaticCredentialsProvider.create( + AwsBasicCredentials.create("foo", "bar") + )) + .build() -ENV server.ssl.key-store=/opt/customcert.jks -ENV server.ssl.key-store-password=password -ENV server.ssl.key-alias=selfsigned - -RUN keytool -genkey -keyalg RSA -alias selfsigned \ - -validity 360 \ - -keystore /opt/customcert.jks \ - -dname "cn=Test, ou=Test, o=Docker, l=NY, st=NY, c=US" \ - -storepass password -keysize 2048 \ - -ext "san=dns:localhost" + s3Client.createBucket { it.bucket("my-bucket") } + } +} ``` -```shell -$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -[...] -* Server certificate: -* subject: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test -* start date: May 9 14:33:40 2025 GMT -* expire date: May 4 14:33:40 2026 GMT -* issuer: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test -* SSL certificate verify result: self signed certificate (18), continuing anyway. -[...] -``` - -### S3Mock Java - -`S3Mock` Java libraries are released and published to the Sonatype Maven Repository and subsequently published to -the official [Maven mirrors](https://search.maven.org/search?q=g:com.adobe.testing%20a:s3mock). +### JUnit 5 Extension -| :warning: WARNING | -|:--------------------------------------------------------------------------------------------------| -| Using the Java libraries is **discouraged**, see explanation below | -| Using the Docker image is **encouraged** to insulate both S3Mock and your application at runtime. | - -`S3Mock` is built using Spring Boot, if projects use `S3Mock` by adding the dependency to their project and starting -the `S3Mock` during a JUnit test, classpaths of the tested application and of the `S3Mock` are merged, leading -to unpredictable and undesired effects such as class conflicts or dependency version conflicts. -This is especially problematic if the tested application itself is a Spring (Boot) application, as both applications -will load configurations based on the availability of certain classes in the classpath, leading to unpredictable runtime -behavior. - -_This is the opposite of what software engineers are trying to achieve when thoroughly testing code in continuous -integration..._ - -`S3Mock` dependencies are updated regularly, any update could break any number of projects. -**See also [issues labeled "dependency-problem"](https://github.com/adobe/S3Mock/issues?q=is%3Aissue+label%3Adependency-problem).** - -**See also [the Java section below](#Java)** - -#### Start using the JUnit5 Extension - -The `S3MockExtension` can currently be used in two ways: - -1. Declaratively using `@ExtendWith(S3MockExtension.class)` and by injecting a properly configured instance of - `AmazonS3` client and/or the started `S3MockApplication` to the tests. - See examples: [`S3MockExtensionDeclarativeTest`](testsupport/junit5/src/test/java/com/adobe/testing/s3mock/junit5/sdk1/S3MockExtensionDeclarativeTest.java) (for SDKv1) - or [`S3MockExtensionDeclarativeTest`](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionDeclarativeTest.kt) (for SDKv2) - -2. Programmatically using `@RegisterExtension` and by creating and configuring the `S3MockExtension` using a _builder_. - See examples: [`S3MockExtensionProgrammaticTest`](testsupport/junit5/src/test/java/com/adobe/testing/s3mock/junit5/sdk1/S3MockExtensionProgrammaticTest.java) (for SDKv1) - or [`S3MockExtensionProgrammaticTest`](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionProgrammaticTest.kt) (for SDKv2) - -To use the JUnit5 Extension, use the following Maven artifact in `test` scope: +> **Note:** This module may be removed in S3Mock 6.x. Consider using [Testcontainers](#testcontainers) instead. +**Maven dependency:** ```xml com.adobe.testing @@ -599,16 +301,24 @@ To use the JUnit5 Extension, use the following Maven artifact in `test` scope: ``` -#### Start using the TestNG Listener +**Usage:** +```kotlin +@ExtendWith(S3MockExtension::class) +class MyTest { + @Test + fun test(s3Client: S3Client) { + s3Client.createBucket { it.bucket("test-bucket") } + } +} +``` + +See examples: [Declarative](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionDeclarativeTest.kt) | [Programmatic](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionProgrammaticTest.kt) -The example [`S3MockListenerXMLConfigurationTest`](testsupport/testng/src/test/kotlin/com/adobe/testing/s3mock/testng/S3MockListenerXmlConfigurationTest.kt) -demonstrates the usage of the `S3MockListener`, which can be configured as shown in [`testng.xml`](testsupport/testng/src/test/resources/testng.xml). -The listener bootstraps the S3Mock application before TestNG execution starts and shuts down the application just before the execution terminates. -Please refer to [`IExecutionListener`](https://github.com/testng-team/testng/blob/master/testng-core-api/src/main/java/org/testng/IExecutionListener.java) -in the TestNG API. +### TestNG Listener -To use the TestNG Listener, use the following Maven artifact in `test` scope: +> **Note:** This module may be removed in S3Mock 6.x. Consider using [Testcontainers](#testcontainers) instead. +**Maven dependency:** ```xml com.adobe.testing @@ -618,206 +328,203 @@ To use the TestNG Listener, use the following Maven artifact in `test` scope: ``` -#### Start programmatically - -Include the following dependency and use one of the `start` methods in `com.adobe.testing.s3mock.S3MockApplication`: - -```xml - - com.adobe.testing - s3mock - ... - -``` - -## File System Structure +Configure in `testng.xml` - see [example configuration](testsupport/testng/src/test/resources/testng.xml). -S3Mock stores Buckets, Objects, Parts and other data on the disk. -This lets users inspect the stored data while the S3Mock is running. -If the environment variable `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` is set to `true`, this data will not be deleted when S3Mock is shut down. +### AWS CLI -| :exclamation: FYI | -|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| While it _may_ be possible to start S3Mock on a root folder from a previous run and have all data available through the S3 API, the structure and contents of the files are not considered Public API, and are subject to change in later releases. | -| Also, there are no automated test cases for this behaviour. | +Use with `--endpoint-url` and `--no-verify-ssl` (for HTTPS): -### Root-Folder +```shell +# Create bucket +aws s3api create-bucket --bucket my-bucket --endpoint-url http://localhost:9090 -S3Mock stores buckets and objects in a root-folder. +# Upload object +aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url http://localhost:9090 -This folder is expected to be empty when S3Mock starts. See also FYI above. +# Download object +aws s3api get-object --bucket my-bucket --key my-file --endpoint-url http://localhost:9090 output-file -``` -// +# HTTPS +aws s3api get-object --bucket my-bucket --key my-file --no-verify-ssl --endpoint-url https://localhost:9191 output-file ``` -### Buckets +### cURL -Buckets are stored as a folder with their name as created through the S3 API directly below the root: +```shell +# Create bucket +curl -X PUT http://localhost:9090/my-bucket/ -``` -/// -``` +# Upload object +curl -X PUT --upload-file ./my-file http://localhost:9090/my-bucket/my-file -[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) is stored in a file in the -bucket directory, serialized as JSON. -[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) contains the "key" -> "uuid" -dictionary for all objects uploaded to this bucket among other data. +# Download object +curl http://localhost:9090/my-bucket/my-file -O -``` -///bucketMetadata.json +# HTTPS (with self-signed certificate) +curl --insecure https://localhost:9191/my-bucket/my-file -O ``` -### Objects +## Configuration -Objects are stored in folders below the bucket they were created in. -A folder is created that uses the Object's UUID assigned in -the [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) as a name. +Configure S3Mock using environment variables: -``` -//// -``` +| Variable | Default | Description | +|-------------------------------------------------------|---------------------|---------------------------------------------------------------| +| `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` | Java temp directory | Base directory for file storage | +| `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` | `us-east-1` | AWS region to mock | +| `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` | none | Comma-separated list of buckets to create on startup | +| `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` | `false` | Keep files after shutdown | +| `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` | none | Comma-separated KMS key ARNs (validation only, no encryption) | +| `COM_ADOBE_TESTING_S3MOCK_CONTROLLER_CONTEXT_PATH` | `""` | Base context path for all endpoints | +| `debug` | `false` | Enable Spring Boot debug logging | +| `trace` | `false` | Enable Spring Boot trace logging | -Object data is stored below that UUID folder. +## Important Limitations -Binary data is always stored in a file `binaryData` +- **Path-style access only**: S3Mock supports `http://localhost:9090/bucket/key`, not `http://bucket.localhost:9090/key` +- **Presigned URLs**: Accepted but not validated (expiration, signature, HTTP verb not checked) +- **Self-signed SSL**: Included certificate requires clients to trust it or ignore SSL errors +- **KMS**: Key validation only - no actual encryption performed +- **Not for production**: S3Mock is a testing tool and lacks security features required for production use -``` -////binaryData -``` +## Troubleshooting -[Object metadata](server/src/main/java/com/adobe/testing/s3mock/store/S3ObjectMetadata.java) is serialized as JSON and -stored as `objectMetadata.json` +
+Click to expand troubleshooting guide -``` -////objectMetadata.json -``` +**Port already in use (`Address already in use`)** +- Ports `9090` (HTTP) and `9191` (HTTPS) must be free +- Check with: `lsof -i :9090` (macOS/Linux) or `netstat -ano | findstr :9090` (Windows) +- Stop conflicting processes or map to different ports: `docker run -p 9091:9090 adobe/s3mock` -#### Object versions +**Connection refused** +- Verify S3Mock is running: `docker ps | grep s3mock` +- Ensure you're using the correct endpoint URL (e.g., `http://localhost:9090`) +- Wait for startup to complete — check logs with `docker logs ` -If versioning is enabled for a bucket, multiple versions of an object can be stored in S3Mock. The files will be stored -alongside each other, prefixed by the versionId. +**SSL certificate errors** +- S3Mock uses a self-signed certificate on the HTTPS port (9191) +- AWS CLI: Add `--no-verify-ssl` flag +- cURL: Add `--insecure` flag +- Java/Kotlin: Configure the SDK to trust the S3Mock certificate or disable SSL verification -``` -////-binaryData -////-objectMetadata.json -``` - -### Multipart Uploads +**Docker not running (Testcontainers)** +- Testcontainers and integration tests require a running Docker daemon +- Start Docker Desktop or the Docker service before running tests +- Check with: `docker info` -Multipart Uploads are created in a bucket using object keys and an uploadId. -The object is assigned a UUID within the bucket (stored -in [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java)). -The [Multipart upload metadata](server/src/main/java/com/adobe/testing/s3mock/store/MultipartUploadInfo.java) is -currently not stored on disk. +**Classpath conflicts (JUnit 5 Extension / embedded mode)** +- S3Mock's embedded Spring Boot server may conflict with your application's dependencies +- **Recommended**: Use [Testcontainers](#testcontainers) or [Docker](#docker-recommended) instead to run S3Mock in isolation +- If using embedded mode, ensure compatible Spring Boot versions -The multiparts folder is created below the bucket folder and named with the `uploadId`: +**AWS SDK endpoint configuration** +- AWS SDK v2: Use `.endpointOverride(URI.create("http://localhost:9090"))` +- Credentials: Use any dummy credentials (e.g., `AwsBasicCredentials.create("foo", "bar")`) +- Region: Use any region (S3Mock defaults to `us-east-1`) -``` -///multiparts// -``` +**Objects not found / wrong bucket** +- S3Mock supports **path-style access only**: `http://localhost:9090/bucket/key` +- Virtual-hosted style (`http://bucket.localhost:9090/key`) is **not supported** +- Verify your SDK client is configured for path-style access -The multiparts metadata file is created below the folder named with the `uploadId`: +
-``` -///multiparts//multipartMetadata.json -``` +## File System Structure -Each part is stored in the parts folder with the `partNo` as name and `.part` as a suffix. +S3Mock stores data on disk with the following structure: ``` -///multiparts//.part +/ + / + bucketMetadata.json # Bucket metadata + / + binaryData # Object content + objectMetadata.json # Object metadata + -binaryData # Versioned object (if versioning enabled) + -objectMetadata.json + multiparts/ + / + multipartMetadata.json + .part ``` -Once the MultipartUpload is completed, the `uploadId` folder is deleted. +**Note:** The file system structure is an implementation detail and may change between releases. While files can be inspected during runtime, reusing persisted data across restarts is not officially supported. -## Build & Run +## Performance & Resources -### Build +S3Mock is designed for testing, not production workloads. Keep the following in mind: -#### With Docker +- **Disk**: All objects are stored on the local filesystem — disk usage grows with stored data +- **Memory**: Scales with concurrent multipart uploads and in-flight requests +- **CI environments**: Consider setting Docker resource limits (e.g., `--memory=512m`) to avoid contention with other services +- **Cleanup**: By default, S3Mock deletes all stored data on shutdown. Use `RETAIN_FILES_ON_EXIT=true` only when needed -To build this project, you need Docker, JDK 17 or higher, and Maven: +## Architecture & Development -```shell -./mvnw clean install +```mermaid +graph LR + Client["AWS SDK / CLI / cURL"] -->|HTTP :9090 / HTTPS :9191| Controller + subgraph S3Mock + Controller["Controller
(REST endpoints)"] -->|delegates| Service["Service
(business logic)"] + Service -->|coordinates| Store["Store
(persistence)"] + Store -->|read/write| FS["Filesystem"] + end ``` -#### Without Docker +**Module Documentation:** +- [Project Overview](AGENTS.md) - Architecture, code style, DO/DON'T guardrails +- [Server Module](server/AGENTS.md) - Core implementation (Controller→Service→Store layers) +- [Integration Tests](integration-tests/AGENTS.md) - Testing with AWS SDK v2 clients +- [Test Support](testsupport/AGENTS.md) - Testcontainers, JUnit 5, TestNG integrations -If you want to skip the Docker build, pass the optional parameter "skipDocker": +## Build & Run + +**Requirements**: Java 17+, Maven 3.9+, Docker (for Docker build and integration tests) +**Build:** ```shell +# Full build with Docker +./mvnw clean install + +# Skip Docker build ./mvnw clean install -DskipDocker ``` -### Run - -You can run the S3Mock from the sources by either of the following methods: - -#### Spring Application - -Run or Debug the class `com.adobe.testing.s3mock.S3MockApplication` in the IDE. - -#### Docker - -Use Docker: - +**Run from source:** ```shell +# As Spring Boot application +./mvnw spring-boot:run -pl server + +# As Docker container ./mvnw clean package -pl server -am -DskipTests -docker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock:latest +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock:latest ``` -#### Docker Maven plugin - -Use the Docker Maven plugin: - +**Run integration tests:** ```shell -./mvnw clean package docker:start -pl server -am -DskipTests -Ddocker.follow -Dit.s3mock.port_http=9090 -Dit.s3mock.port_https=9191 +./mvnw verify -pl integration-tests ``` -(stop with `ctrl-c`) - -### Run integration tests - -Once the application is started using default ports, you can execute the `*IT` tests from your IDE. - -### Java - -This repo is built with Java 17, output is _currently_ bytecode compatible with Java 17. -### Kotlin +**Technology:** +- Built with Kotlin 2.3.0 and Spring Boot 4.0 +- Tests written in Kotlin +- Target JVM: 17 -The [Integration Tests](integration-tests) and unit tests are built in Kotlin. - -## Governance model - -The project owner and leads make all final decisions. See the `developers` section in the [pom.xml](pom.xml) for a list -of leads. - -## Vulnerability reports - -S3Mock uses GitHub actions to produce an SBOM and to check dependencies for vulnerabilities. All vulnerabilities are -evaluated and fixed if possible. -Vulnerabilities may also be reported through the GitHub issue tracker. +## Contributing -## Security +Contributions are welcome! See [Contributing Guide](.github/CONTRIBUTING.md). -S3Mock is not intended to be used in production environments. It is a mock server meant to be used in -development and testing environments only. It does not implement all security features of AWS S3 and should not be used -as a replacement for AWS S3 in production. -It is implemented using [Spring Boot](https://github.com/spring-projects/spring-boot), which is a Java framework that is -designed to be secure by default. +**Governance**: Project leads make final decisions - see `developers` in [pom.xml](pom.xml). -## Contributing +**Security**: See [Security Policy](.github/SECURITY.md) for reporting vulnerabilities. S3Mock uses GitHub Actions for SBOM and vulnerability scanning. -Contributions are welcome! Read the [Contributing Guide](./.github/CONTRIBUTING.md) for more information. +## License -## Licensing +Licensed under the Apache License 2.0 - see [LICENSE](LICENSE). -This project is licensed under the Apache V2 License. See [LICENSE](LICENSE) for more information. +--- -## Powered by -[![IntelliJ IDEA logo.](https://resources.jetbrains.com/storage/products/company/brand/logos/IntelliJ_IDEA.svg)](https://jb.gg/OpenSourceSupport) +[![IntelliJ IDEA](https://resources.jetbrains.com/storage/products/company/brand/logos/IntelliJ_IDEA.svg)](https://jb.gg/OpenSourceSupport) -## Star History -[![Star History Chart](https://api.star-history.com/svg?repos=adobe/S3Mock&type=Date)](https://www.star-history.com/#adobe/S3Mock&Date) +[![Star History](https://api.star-history.com/svg?repos=adobe/S3Mock&type=Date)](https://www.star-history.com/#adobe/S3Mock&Date) diff --git a/docs/tasks.md b/docs/tasks.md deleted file mode 100644 index e14c2a35c..000000000 --- a/docs/tasks.md +++ /dev/null @@ -1,188 +0,0 @@ -# S3Mock Improvement Tasks - -This document contains a list of potential improvements for the S3Mock project. Each task is marked with a checkbox that can be checked off when completed. - -## Architecture Improvements - -1. [ ] Implement support for domain-style access to buckets (e.g., `http://bucket.localhost:9090/someKey`) to better match AWS S3 behavior -2. [ ] Add support for AWS S3 operations that are currently not implemented (e.g., bucket policies, website configuration, etc.) -3. [ ] Implement a plugin architecture to allow for custom extensions and behaviors -4. [ ] Add support for S3 event notifications (SNS, SQS, Lambda) -5. [ ] Improve error handling to more closely match AWS S3 error responses -6. [ ] Implement a metrics collection system to monitor performance and usage -7. [ ] Add support for S3 Select for querying object content -8. [ ] Implement support for S3 Inventory reports -9. [ ] Add support for S3 Analytics configurations -10. [ ] Implement S3 Batch Operations - -## Code Quality Improvements - -11. [ ] Increase unit test coverage for controller, service and store layers - -### Task 11 – Test Coverage Plan (Components and Actions) - -Scope and components to cover: -- Controllers (HTTP): BucketController, ObjectController, MultipartController -- Services (business logic): BucketService, ObjectService, MultipartService, Kms* services -- Stores (persistence): BucketStore, ObjectStore, MultipartStore, KmsKeyStore -- XML/DTOs and mappers: request/response XML models, serialization utils -- Utilities: digest, ETag, headers, Range requests, SSE -- Configuration: StoreConfiguration, controller advice/error mapping - -Priorities (short-to-long horizon): -1) High-value happy-path and error-path coverage for controllers with mocked services (fast feedback). -2) Store layer correctness with Spring Boot WebEnvironment.NONE tests (file IO, metadata persistence, edge cases). -3) Service layer behavior with mocked stores (parameter validation, branching, SSE/KMS interactions). -4) XML serialization/deserialization fidelity for commonly used operations. -5) Regression tests for known corner cases (range requests, conditional headers, multipart completion ordering, KMS key validation). - -Concrete test additions (incremental): -- Controllers - - BucketController - - listBuckets returns empty and non-empty results; XML schema shape - - createBucket duplicate name -> proper S3 error code - - deleteBucket non-empty -> proper error - - ObjectController - - putObject with/without Content-MD5; mismatched MD5 -> error - - getObject with Range header (single range) -> 206 + Content-Range - - getObject nonexistent -> 404 S3-style error - - headObject verifies metadata and headers (ETag, Content-Length) - - MultipartController - - initiateMultipartUpload returns uploadId - - uploadPart with invalid partNumber -> error mapping - - completeMultipartUpload out-of-order parts -> consistent ETag behavior -- Services - - ObjectService.storeObject validates metadata, handles SSE headers routing to KMS - - BucketService.deleteBucket checks emptiness guard -- Stores - - ObjectStore - - storeS3ObjectMetadata and getS3ObjectMetadata roundtrip - - list with prefix/delimiter, max-keys, continuation - - delete removes metadata and data file - - BucketStore - - create, list, delete, exist checks - - MultipartStore - - init, addPart, complete, abort state transitions -- XML/DTOs - - Serialize/deserialize ListAllMyBucketsResult, CompleteMultipartUploadResult - -Suggested file locations (server module): -- Controllers: server/src/test/kotlin/com/adobe/testing/s3mock/itlike/controller/*Test.kt (extending BaseControllerTest) -- Services: server/src/test/kotlin/com/adobe/testing/s3mock/service/*Test.kt -- Stores: server/src/test/kotlin/com/adobe/testing/s3mock/store/*Test.kt (extend StoreTestBase) -- DTOs: server/src/test/kotlin/com/adobe/testing/s3mock/xml/*Test.kt - -Execution (fast path per repo guidelines): -- One test class: ./mvnw -pl server test -Dtest=ObjectStoreTest -- One method: ./mvnw -pl server test -Dtest=ObjectStoreTest#testStoreAndGetObject -- Or via tool: run_test server/src/test/kotlin/com/adobe/testing/s3mock/store/ObjectStoreTest.kt - -Acceptance targets for Task 11 completion: -- +10–15% line coverage increase in server module, focusing on controllers and stores -- At least one new test per component category listed above -- Error-path assertions include correct HTTP status and S3 error codes/messages - -Notes: -- Avoid ITs unless Docker available; prefer WebEnvironment.RANDOM_PORT controller tests with mocked services. -- Use provided test bases: BaseControllerTest, StoreTestBase, ServiceTestBase. -- Reuse existing sample files: server/src/test/resources/sampleFile.txt, sampleFile_large.txt, sampleKMSFile.txt. -12. [ ] Refactor synchronization mechanisms in store classes to improve concurrency handling -13. [ ] Implement more comprehensive input validation for S3 API parameters -14. [ ] Add more detailed logging throughout the application for better debugging -15. [ ] Refactor XML serialization/deserialization to use more type-safe approaches -16. [ ] Improve exception handling and error messages -17. [ ] Implement more robust cleanup of temporary files -18. [ ] Add comprehensive JavaDoc to all public APIs -19. [ ] Refactor controller classes to reduce code duplication -20. [ ] Implement more thorough validation of S3 object metadata - -## Performance Improvements - -21. [x] Optimize file storage for large objects -22. [ ] Implement caching for frequently accessed objects -23. [ ] Optimize list operations for buckets with many objects -24. [ ] Improve multipart upload performance -25. [x] Reduce memory usage when handling large files -26. [ ] Optimize XML serialization/deserialization -27. [x] Keep object metadata storage as plain text (JSON) for inspectability (decided against more efficient/binary storage) -28. [ ] Add support for conditional requests to reduce unnecessary data transfer -29. [ ] Optimize concurrent access patterns -30. [ ] Implement more efficient bucket and object locking mechanisms - -## Security Improvements - -31. [ ] Add support for bucket policies -32. [ ] Implement proper authentication and authorization mechanisms -33. [ ] Add support for AWS IAM-style access control -34. [ ] Implement proper handling of AWS signatures for authentication -35. [ ] Add support for server-side encryption with customer-provided keys -36. [ ] Improve SSL/TLS configuration and certificate management -37. [ ] Implement proper input sanitization to prevent injection attacks -38. [ ] Add support for VPC endpoint policies -39. [ ] Implement proper access logging -40. [ ] Add support for AWS CloudTrail-style audit logging - -## Documentation Improvements - -41. [ ] Create comprehensive API documentation -42. [ ] Add more examples for different usage scenarios -43. [ ] Improve README with more detailed setup instructions -44. [ ] Create troubleshooting guide -45. [ ] Add architecture diagrams -46. [ ] Document internal design decisions -47. [ ] Create contributor guidelines -48. [ ] Add more code examples for different programming languages -49. [ ] Create a user guide with common use cases -50. [ ] Document performance characteristics and limitations - -## DevOps Improvements - -51. [ ] Add Docker Compose examples for different scenarios -52. [ ] Implement health checks for container orchestration -53. [ ] Add Kubernetes deployment examples -54. [ ] Improve CI/CD pipeline -55. [ ] Add performance benchmarking to CI process -56. [ ] Implement automated security scanning -57. [ ] Add support for configuration through environment variables for all settings -58. [ ] Improve Docker image size and build time -59. [ ] Add support for container orchestration tools -60. [ ] Implement graceful shutdown and startup procedures - -## Dependency and Technology Updates - -61. [ ] Evaluate migration to Java 21 for improved performance and features -62. [ ] Update to latest Spring Boot version when available -63. [ ] Consider using reactive programming model with Spring WebFlux -64. [ ] Evaluate using non-blocking I/O for file operations -65. [ ] Consider using a more efficient serialization format for metadata storage -66. [ ] Evaluate using a database for metadata storage instead of files -67. [ ] Update AWS SDK dependencies regularly -68. [ ] Consider using GraalVM native image compilation for faster startup -69. [ ] Evaluate using Project Loom virtual threads for improved concurrency -70. [ ] Consider using modern Java language features more extensively (records, sealed classes, etc.) - -## Testing Improvements - -71. [ ] Add more integration tests for edge cases -72. [ ] Implement property-based testing for robust validation -73. [ ] Add performance regression tests -74. [ ] Implement chaos testing for resilience verification -75. [ ] Add more unit tests for utility classes -76. [ ] Implement contract tests for S3 API compatibility -77. [ ] Add tests for concurrent access scenarios -78. [ ] Improve test coverage for error conditions -79. [ ] Add tests for different storage backends -80. [ ] Implement automated compatibility testing with different AWS SDK versions - -## User Experience Improvements - -81. [ ] Add a web-based UI for browsing buckets and objects -82. [ ] Implement a CLI tool for interacting with S3Mock -83. [ ] Add better error messages and troubleshooting information -84. [ ] Improve startup time -85. [ ] Add support for custom endpoints -86. [ ] Implement request/response logging for debugging -87. [ ] Add support for request tracing -88. [ ] Improve configuration options documentation -89. [ ] Add support for programmatic configuration -90. [ ] Implement better defaults for common use cases diff --git a/integration-tests/AGENTS.md b/integration-tests/AGENTS.md new file mode 100644 index 000000000..002947f63 --- /dev/null +++ b/integration-tests/AGENTS.md @@ -0,0 +1,118 @@ +# Agent Context for S3Mock Integration Tests + +Integration tests verifying S3Mock with real AWS SDK v2 clients. + +## Structure + +``` +integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/ +├── S3TestBase.kt # Base class with s3Client (v2) +├── BucketIT.kt # Bucket operations +├── ObjectIT.kt # Object operations +├── MultipartUploadIT.kt # Multipart uploads +└── *IT.kt # Other tests +``` + +## Base Class + +Extend `S3TestBase` for access to: +- `s3Client` - AWS SDK v2 +- `serviceEndpoint`, `serviceEndpointHttp`, `serviceEndpointHttps` + +## DO / DON'T + +### DO +- Extend `S3TestBase` for all integration tests +- Use **backtick test names**: `` fun `should create bucket and upload object`(testInfo: TestInfo) `` +- Use **unique bucket names** via `givenBucket(testInfo)` or `UUID.randomUUID()` +- Use **Arrange-Act-Assert** pattern consistently +- Accept `testInfo: TestInfo` parameter in test methods for unique resource naming +- Verify HTTP codes, headers (ETag, Content-Type), XML/JSON bodies, and error responses +- Refactor legacy `testSomething` camelCase names to backtick style when touching tests + +### DON'T +- DON'T use AWS SDK v1 — it has been removed in 5.x +- DON'T share state between tests — each test must be self-contained +- DON'T hardcode bucket names — use `UUID.randomUUID()` for uniqueness +- DON'T mock AWS SDK clients — use actual SDK clients against S3Mock + +## Test Pattern + +Arrange-Act-Assert with `testInfo` for unique bucket names: + +```kotlin +internal class MyFeatureIT : S3TestBase() { + @Test + fun `should perform operation`(testInfo: TestInfo) { + // Arrange + val bucketName = givenBucket(testInfo) + + // Act + s3Client.putObject( + PutObjectRequest.builder().bucket(bucketName).key("key").build(), + RequestBody.fromString("content") + ) + + // Assert + val response = s3Client.getObject( + GetObjectRequest.builder().bucket(bucketName).key("key").build() + ) + assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") + } +} +``` + +## Common Patterns + +**Bucket creation**: Use `givenBucket(testInfo)` from `S3TestBase` — don't create your own helper. + +**Multipart**: Initiate → Upload parts → Complete → Verify + +**Errors** (use AssertJ, not JUnit `assertThrows`): +```kotlin +assertThatThrownBy { s3Client.deleteBucket { it.bucket(bucketName) } } + .isInstanceOf(AwsServiceException::class.java) + .hasMessageContaining("Status Code: 409") +``` + +**Metadata**: +```kotlin +s3Client.putObject( + PutObjectRequest.builder() + .metadata(mapOf("key" to "value")) + .build(), + RequestBody.fromString("content") +) +``` + +## Assertions + +AssertJ: +```kotlin +assertThat(result).isNotNull() +assertThat(list).hasSize(3).contains(item) +assertThat(buckets.buckets()).anyMatch { it.name() == bucketName } +``` + +## Running + +```bash +./mvnw verify -pl integration-tests +./mvnw verify -pl integration-tests -Dit.test=BucketIT +./mvnw verify -pl integration-tests -Dit.test=BucketIT#shouldCreateBucket +./mvnw verify -pl integration-tests -DskipDocker +``` + +## Best Practices + +1. Independent tests (no shared state) +2. Unique bucket names with UUID +3. Use AWS SDK v2 (`s3Client`) — SDK v1 has been removed +4. Verify: HTTP codes, headers (ETag, Content-Type), XML/JSON bodies, error responses +5. Use actual AWS SDK clients (not mocks) + +## Debugging + +- Check Docker container logs +- Verify endpoint connectivity +- Compare with AWS S3 API docs diff --git a/server/AGENTS.md b/server/AGENTS.md new file mode 100644 index 000000000..ac96740f6 --- /dev/null +++ b/server/AGENTS.md @@ -0,0 +1,156 @@ +# Agent Context for S3Mock Server Module + +Core S3Mock server implementation. + +## Structure + +``` +server/src/main/kotlin/com/adobe/testing/s3mock/ +├── S3MockApplication.kt # Spring Boot entry +├── S3MockConfiguration.kt # Top-level config +├── S3MockProperties.kt # Properties binding +├── controller/ +│ ├── *Controller.kt # REST endpoints +│ ├── ControllerConfiguration.kt # Controller beans + exception handlers +│ └── ControllerProperties.kt +├── dto/ # XML/JSON models (*Result, request/response) +├── service/ +│ ├── *Service.kt # Business logic +│ └── ServiceConfiguration.kt # Service beans (used in @SpringBootTest) +├── store/ +│ ├── *Store.kt # Persistence +│ ├── StoreConfiguration.kt # Store beans (used in @SpringBootTest) +│ └── StoreProperties.kt +└── util/ + ├── DigestUtil.kt # ETag/checksum computation (replaces Apache Commons) + ├── EtagUtil.kt # ETag normalization + ├── HeaderUtil.kt # HTTP header helpers + └── AwsHttpHeaders.kt # AWS-specific header constants +``` + +## Implementation Flow + +**Adding S3 operation**: DTO (Jackson XML) → Store (filesystem) → Service (validation, logic) → Controller (HTTP mapping) + +### 1. DTOs (`dto/`) +```kotlin +@JacksonXmlRootElement(localName = "CreateBucketConfiguration") +data class CreateBucketConfiguration( + @JacksonXmlProperty(localName = "LocationConstraint") + val locationConstraint: LocationConstraint? = null +) +``` + +Key annotations: `@JacksonXmlRootElement`, `@JacksonXmlProperty`, `@JacksonXmlElementWrapper(useWrapping = false)` + +### 2. Store (`store/`) +- Filesystem path resolution, binary storage, metadata JSON +- `BucketStore`, `ObjectStore` for CRUD operations +- `BucketMetadata`, `S3ObjectMetadata` models + +### 3. Service (`service/`) +```kotlin +@Service +class ObjectService( + private val bucketStore: BucketStore, + private val objectStore: ObjectStore +) { + fun getObject(bucketName: String, key: String): S3Object { + val bucket = bucketStore.getBucketMetadata(bucketName) + ?: throw S3Exception.NO_SUCH_BUCKET + return objectStore.getObject(bucket, key) + ?: throw S3Exception.NO_SUCH_KEY + } +} +``` + +Responsibilities: Validation, S3 exceptions, coordinate stores + +### 4. Controller (`controller/`) +```kotlin +@RestController +class ObjectController(private val objectService: ObjectService) { + @GetMapping("/{bucketName:.+}/{*key}") + fun getObject(@PathVariable bucketName: String, @PathVariable key: String) = + objectService.getObject(bucketName, key).let { + ResponseEntity.ok() + .header("ETag", "\"${it.etag}\"") + .body(it.dataStream) + } +} +``` + +Responsibilities: HTTP mapping, headers, streaming responses + +## Error Handling + +- Services throw **`S3Exception`** constants (e.g., `S3Exception.NO_SUCH_KEY`, `S3Exception.INVALID_BUCKET_NAME`) +- Controllers never catch exceptions — Spring's `@ExceptionHandler` in `ControllerConfiguration` handles them +- `S3MockExceptionHandler` converts `S3Exception` → XML `ErrorResponse` with the correct HTTP status +- `IllegalStateExceptionHandler` converts unexpected errors → `500 InternalError` +- DON'T create new exception classes — add new constants to `S3Exception` + +## DO / DON'T + +### DO +- Follow the **DTO → Store → Service → Controller** flow when adding new S3 operations +- Add **Jackson XML annotations** matching the AWS API naming exactly (verify against [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)) +- Use **`@SpringBootTest`** with **`@MockitoBean`** for all tests +- Use **backtick test names**: `` fun `should return object with correct etag`() `` +- Throw **`S3Exception` constants** (e.g., `S3Exception.NO_SUCH_BUCKET`, `S3Exception.NO_SUCH_KEY`) from the Service layer + +### DON'T +- DON'T put business logic in controllers — controllers only map HTTP requests and delegate to services +- DON'T use `@Autowired` in production code — use constructor injection +- DON'T return raw strings — use typed DTOs for XML/JSON responses +- DON'T use `@ExtendWith(MockitoExtension::class)`, `@Mock`, or `@InjectMocks` — use `@SpringBootTest` with `@MockitoBean` +- DON'T use legacy `testSomething` naming — refactor to backtick style when touching tests + +## Testing + +Spring Boot tests with `@MockitoBean`: +```kotlin +@SpringBootTest(classes = [ServiceConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) +@MockitoBean(types = [BucketService::class, MultipartService::class, MultipartStore::class]) +internal class ObjectServiceTest : ServiceTestBase() { + @Autowired + private lateinit var iut: ObjectService + + @Test + fun `should get object`() { + whenever(bucketStore.getBucketMetadata("bucket")).thenReturn(bucket) + whenever(objectStore.getObject(bucket, "key")).thenReturn(s3Object) + assertThat(iut.getObject("bucket", "key")).isEqualTo(s3Object) + } +} +``` + +## Configuration + +Three `@ConfigurationProperties` classes bind environment variables to typed properties: +- `StoreProperties` (`com.adobe.testing.s3mock.store.*`) — storage root, buckets, KMS, region +- `ControllerProperties` (`com.adobe.testing.s3mock.controller.*`) — context path +- `S3MockProperties` (`com.adobe.testing.s3mock.*`) — top-level settings + +Spring Boot relaxed binding maps properties to environment variables automatically: +`com.adobe.testing.s3mock.store.initial-buckets` → `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` + +```kotlin +@JvmRecord +@ConfigurationProperties("com.adobe.testing.s3mock.store") +data class StoreProperties( + @param:DefaultValue("false") val retainFilesOnExit: Boolean, + @param:DefaultValue("") val root: String, + @param:DefaultValue("") val validKmsKeys: Set, + @param:DefaultValue("") val initialBuckets: List, + @param:DefaultValue("us-east-1") val region: Region +) +``` + +## Running + +```bash +./mvnw spring-boot:run -pl server +./mvnw package -pl server -am && java -jar server/target/s3mock-*.jar +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock:latest +``` diff --git a/testsupport/AGENTS.md b/testsupport/AGENTS.md new file mode 100644 index 000000000..7ad6cc948 --- /dev/null +++ b/testsupport/AGENTS.md @@ -0,0 +1,89 @@ +# Agent Context for S3Mock Test Support + +Test framework integrations for using S3Mock: JUnit 5, Testcontainers, TestNG. + +> **Deprecation Notice (6.x):** The JUnit 5, TestNG, and all direct-integration modules will be +> removed in S3Mock 6.x. Testcontainers will become the only officially supported testing approach. +> See [CHANGELOG.md](../CHANGELOG.md) for details. + +## Framework Selection + +- **Testcontainers** (`testcontainers/`) - Docker container isolation, works with any framework. **Recommended.** +- **JUnit 5** (`junit5/`) - Extension with parameter injection. Will be removed in 6.x. +- **TestNG** (`testng/`) - Listener-based integration. Will be removed in 6.x. +- **Common** (`common/`) - Shared `S3MockStarter` base class and utilities. + +> **Note:** JUnit 4 support (`junit4/`) was removed in 5.x. + +## DO / DON'T + +### DO +- Prefer **Testcontainers** for new test setups — it provides the best isolation +- Keep framework modules **thin** — delegate to `common/` for shared logic +- Maintain backward compatibility for existing public APIs + +### DON'T +- DON'T add new features to `junit4/` — it has been removed in 5.x +- DON'T invest heavily in `junit5/` or `testng/` — they will be removed in 6.x +- DON'T add framework-specific logic that belongs in `common/` + +## JUnit 5 Extension + +```kotlin +@ExtendWith(S3MockExtension::class) +class MyTest { + @Test + fun test(s3Client: S3Client) { + s3Client.createBucket { it.bucket("test-bucket") } + } +} +``` + +Builder alternative: +```kotlin +@RegisterExtension +val s3Mock = S3MockExtension.builder() + .withInitialBuckets("bucket1") + .withValidKmsKeys("arn:aws:kms:...") + .build() +``` + +## Testcontainers + +```kotlin +@Container +val s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + .withValidKmsKeys("key-id") + +// Access via s3Mock.httpEndpoint or s3Mock.httpsEndpoint +``` + +Configuration methods: `withInitialBuckets()`, `withValidKmsKeys()`, `withRetainFilesOnExit()`, `withRoot()`, `withRegion()`. + +## TestNG Listener + +Configure in `testng.xml`: +```xml + +``` + +Access via system properties: +```kotlin +val endpoint = System.getProperty("s3mock.httpEndpoint") +``` + +Parameters: `s3mock.httpPort` (9090), `s3mock.httpsPort` (9191), `s3mock.initialBuckets`, `s3mock.root`. + +## Design Principles + +1. Framework-agnostic core in `common` module +2. Simple API with sensible defaults +3. Proper lifecycle management (start before tests, stop after) +4. Resource cleanup after tests + +## Common Issues + +- **Not starting**: Check ports 9090/9191 availability, Docker running (Testcontainers) +- **Connection refused**: Verify S3Mock started, correct endpoint URL +- **Test interference**: Use unique bucket names (UUID), ensure test isolation