From 9516d1c902946e45ff82d4a550713cca9fb2d42e Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sat, 21 Feb 2026 14:29:50 +0100 Subject: [PATCH 01/36] chore: remove junie files --- .junie/guidelines.md | 386 ------------------------------------------- docs/tasks.md | 188 --------------------- 2 files changed, 574 deletions(-) delete mode 100644 .junie/guidelines.md delete mode 100644 docs/tasks.md diff --git a/.junie/guidelines.md b/.junie/guidelines.md deleted file mode 100644 index 7d0d5129a..000000000 --- a/.junie/guidelines.md +++ /dev/null @@ -1,386 +0,0 @@ -# S3Mock Development Guidelines (Concise) - -Essential info for working on S3Mock. This top section is a quick, no‑frills guide. Details follow below. - -Quickstart TL;DR -- Build (fast): ./mvnw clean install -DskipDocker -- Build (full): ./mvnw clean install -- Server tests only: ./mvnw -pl server test -- All tests incl. ITs (requires Docker): ./mvnw verify -- One server test: ./mvnw -pl server test -Dtest=ObjectStoreTest -- One IT: ./mvnw -pl integration-tests -am verify -Dit.test=BucketIT - -Requirements -- JDK 17+ -- Docker (only for integration tests and Docker image build) - -Common config (env vars) -- COM_ADOBE_TESTING_S3MOCK_STORE_REGION (default: us-east-1) -- COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS -- COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT (default: false) -- debug / trace (Spring Boot flags) - -Tips -- IDE runs: use server tests; ITs need Docker via Maven lifecycle. -- Debug server on 9090/9191; trust self‑signed cert or use HTTP. -- Before declaring done: run a full Maven build successfully. - -Troubleshooting -- Connection refused: ensure S3Mock is running on expected ports. -- SSL errors: trust self‑signed cert or switch to HTTP. -- Docker errors: ensure Docker is running and you have permissions. - -## Junie Operations Playbook (Critical) -To ensure tests execute successfully in this environment, follow these strict rules: - -- Default test scope: server module only. Do NOT run full project builds by default. -- Use the test tool, not shell, to run tests: - - Preferred: run_test on specific test files, e.g., "server/src/test/kotlin/com/adobe/testing/s3mock/store/ObjectStoreTest.kt". - - One test by name: use run_test with full path and the test method name parameter. - - Note: Directory-wide runs via run_test may not be supported in this environment. If you need to run all server tests, use Maven with: ./mvnw -pl server -DskipDocker test. -- Avoid integration tests unless explicitly requested and Docker availability is confirmed. If requested, run via Maven lifecycle only. -- If a build is required, prefer fast builds: - - Use ./mvnw -pl server -am -DskipDocker clean test or rely on run_test which compiles as needed. - - Only run ./mvnw clean install (full) when the user explicitly asks for a full build or cross-module changes demand it. -- Never run mvnw verify without confirming Docker is available; if not available, add -DskipDocker. -- Java 17+ required; if build fails due to JDK, report and stop, do not retry with different commands. -- Decision tree: - 1) Need to validate changes in server module? -> run_test on one or more specific test files (fast path). If you truly need all server tests, use: ./mvnw -pl server -DskipDocker test. - 2) Need a specific server test? -> run_test on that file. - 3) Need ITs and Docker is confirmed? -> mvnw -pl integration-tests -am verify; otherwise skip. - 4) Need a build artifact quickly? -> mvnw clean install -DskipDocker. - -Note: Always summarize which scope you ran and why. - -— - -## Build and Configuration Instructions - -### Building the Project - -S3Mock uses Maven for building. The project includes a Maven wrapper (`mvnw`) so you don't need to install Maven separately. - -#### Basic Build Commands - -```bash -# Build the entire project -./mvnw clean install - -# Build without running Docker (useful for quick builds) -./mvnw clean install -DskipDocker - -# Build a specific module -./mvnw clean install -pl server -am -``` - -#### Build Requirements - -- JDK 17 or higher -- Docker (for building the Docker image and running integration tests) - -### Configuration Options - -S3Mock can be configured with the following environment variables: - -| Environment Variable | Legacy Name | Description | Default | -|-------------------------------------------------------|-----------------------------------|-------------------------------------------------------|---------------------| -| `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` | `validKmsKeys` | List of KMS Key-Refs that are treated as valid | none | -| `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` | `initialBuckets` | List of bucket names that will be available initially | none | -| `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` | `COM_ADOBE_TESTING_S3MOCK_REGION` | The region the S3Mock is supposed to mock | `us-east-1` | -| `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` | `root` | Base directory for temporary files | Java temp directory | -| `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` | `retainFilesOnExit` | Set to `true` to keep files after shutdown | `false` | -| `debug` | - | Enable Spring Boot's debug output | `false` | -| `trace` | - | Enable Spring Boot's trace output | `false` | - -## Testing Information - -### Test Structure - -The project uses JUnit 5 for testing. There are two main types of tests in the project: - -1. **Integration Tests**: Located in the `integration-tests` module and written in Kotlin. These tests verify the end-to-end functionality of S3Mock by starting a Docker container and making actual HTTP requests to it. - -2. **Server Module Tests**: Located in the `server` module and primarily written in Kotlin. These include: - - Unit tests for utility classes and DTOs - - Spring Boot tests for controllers, services, and stores - - XML serialization/deserialization tests - -#### Integration Tests - -The main test base class for integration tests is `S3TestBase` which provides utility methods for: -- Creating S3 clients -- Managing buckets and objects -- Handling SSL certificates -- Generating test data - -#### Server Module Tests - -The server module contains several types of tests: - -1. **Controller Tests**: Use `@SpringBootTest` with `WebEnvironment.RANDOM_PORT` and `TestRestTemplate` to test HTTP endpoints. These tests mock the service layer using `@MockitoBean`. - -2. **Store Tests**: Use `@SpringBootTest` with `WebEnvironment.NONE` to test the data storage layer. These tests often use `@Autowired` to inject the component under test. - -3. **Service Tests**: Test the service layer with mocked dependencies. - -4. **Unit Tests**: Simple tests for utility classes and DTOs without Spring context. - -Common base classes and utilities: -- `BaseControllerTest`: Sets up XML serialization/deserialization for controller tests -- `StoreTestBase`: Common setup for store tests -- `ServiceTestBase`: Common setup for service tests - -### Running Tests - -#### Running Server Module Tests - -The server module tests can be run without Docker: - -```bash -# Run all server module tests -./mvnw -pl server test - -# Run a specific test class -./mvnw -pl server test -Dtest=ObjectStoreTest - -# Run a specific test method -./mvnw -pl server test -Dtest=ObjectStoreTest#testStoreObject -``` - -The server module uses Spring Test Profiler to analyze test performance. Test execution times are reported in the `target/spring-test-profiler` directory. - -#### Running Integration Tests - -Integration tests require Docker to be running as they start an S3Mock Docker container. The integration tests can only be executed through Maven because they need the S3Mock to run. The process works as follows: - -1. The Docker Maven plugin (io.fabric8:docker-maven-plugin) starts the S3Mock Docker container in the pre-integration-test phase -2. The Maven Failsafe plugin runs the integration tests against the running container -3. The Docker Maven plugin stops the container in the post-integration-test phase - -```bash -# Run all integration tests -./mvnw verify - -# Run a specific integration test -./mvnw -pl integration-tests -am verify -Dit.test=BucketIT - -# Run tests without Docker (will skip integration tests) -./mvnw verify -DskipDocker -``` - -Note that attempting to run integration tests directly from your IDE without starting the S3Mock Docker container will result in connection errors, as the tests expect S3Mock to be running on specific ports. - -### Writing New Tests - -#### Writing Integration Tests - -To create a new integration test: - -1. Create a new Kotlin class in the `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its` directory -2. Extend the `S3TestBase` class to inherit utility methods -3. Name your test class with an `IT` suffix to be recognized as an integration test -4. Use the provided S3 client methods to interact with S3Mock - -Example integration test: - -```kotlin -// ExampleIT.kt -internal class ExampleIT : S3TestBase() { - private val s3Client = createS3Client() - - @Test - fun testPutAndGetObject(testInfo: TestInfo) { - // Create a bucket - val bucketName = givenBucket(testInfo) - - // Create test content - val content = "This is a test file content" - val contentBytes = content.toByteArray(StandardCharsets.UTF_8) - - // Put an object into the bucket - val key = "test-object.txt" - val putObjectResponse = s3Client.putObject( - PutObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build(), - RequestBody.fromBytes(contentBytes) - ) - - // Verify the object was uploaded successfully - assertThat(putObjectResponse.eTag()).isNotBlank() - - // Get the object back - val getObjectResponse = s3Client.getObject( - GetObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - // Read the content and verify it matches what we uploaded - val retrievedContent = getObjectResponse.readAllBytes() - assertThat(retrievedContent).isEqualTo(contentBytes) - - // Clean up - s3Client.deleteObject { it.bucket(bucketName).key(key) } - } -} -``` - -#### Writing Server Module Tests - -The server module uses different testing approaches depending on what's being tested: - -1. **Controller Tests**: - - Extend `BaseControllerTest` to inherit XML serialization setup - - Use `@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)` - - Use `@MockitoBean` to mock service dependencies - - Inject `TestRestTemplate` to make HTTP requests to the controller - -Example controller test: - -```kotlin -// BucketControllerTest.kt -@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) -@MockitoBean(classes = [BucketService::class, ObjectService::class, MultipartService::class]) -internal class BucketControllerTest : BaseControllerTest() { - @Autowired - private lateinit var restTemplate: TestRestTemplate - - @MockitoBean - private lateinit var bucketService: BucketService - - @Test - fun testListBuckets() { - // Mock service response - whenever(bucketService.listBuckets()).thenReturn(givenBuckets(2)) - - // Make HTTP request - val response = restTemplate.getForEntity("/", ListAllMyBucketsResult::class.java) - - // Verify response - assertThat(response.statusCode).isEqualTo(HttpStatus.OK) - assertThat(response.body.buckets.bucket).hasSize(2) - } -} -``` - -2. **Store Tests**: - - Extend `StoreTestBase` for common setup - - Use `@SpringBootTest(webEnvironment = WebEnvironment.NONE)` - - Use `@Autowired` to inject the component under test - -Example store test: - -```kotlin -// ObjectStoreTest.kt -@SpringBootTest(classes = [StoreConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) -@MockitoBean(classes = [KmsKeyStore::class, BucketStore::class]) -internal class ObjectStoreTest : StoreTestBase() { - @Autowired - private lateinit var objectStore: ObjectStore - - @Test - fun testStoreAndGetObject() { - // Test storing and retrieving an object - val sourceFile = File(TEST_FILE_PATH) - val id = UUID.randomUUID().toString() - val name = sourceFile.name - - objectStore.storeS3ObjectMetadata( - metadataFrom("test-bucket"), id, name, "text/plain", - emptyMap(), sourceFile.toPath(), - emptyMap(), emptyMap(), null, emptyList(), - null, null, Owner.DEFAULT_OWNER, StorageClass.STANDARD - ) - - val metadata = objectStore.getS3ObjectMetadata(metadataFrom("test-bucket"), id, null) - - assertThat(metadata.key).isEqualTo(name) - assertThat(metadata.contentType).isEqualTo("text/plain") - } -} -``` - -3. **Unit Tests**: - - Use standard JUnit 5 tests without Spring context - - Focus on testing a single class or method in isolation - -Example unit test: - -```kotlin -// DigestUtilTest.kt -internal class DigestUtilTest { - @Test - fun testHexDigest() { - val input = "test data".toByteArray() - val expected = DigestUtils.md5Hex(input) - - assertThat(DigestUtil.hexDigest(input)).isEqualTo(expected) - } -} -``` - -## Additional Development Information - -### Project Structure - -S3Mock is a multi-module Maven project: - -- `build-config`: Build configuration files -- `docker`: Docker image build module -- `integration-tests`: Integration tests -- `server`: Core S3Mock implementation -- `testsupport`: Test support modules for different testing frameworks - -### Code Style - -The project uses Checkstyle for Java code style checking. The configuration is in `build-config/checkstyle.xml`. - -### License Headers - -S3Mock uses the Apache License 2.0 and enforces proper license headers in all source files through the `license-maven-plugin`. Important rules: - -- All source files must include the proper license header -- When modifying a file, the copyright year range in the header must be updated to include the current year -- The format is: `Copyright 2017-YYYY Adobe.` where YYYY is the current year -- The license check runs automatically during the build process -- To fix license headers, run: `./mvnw license:format` -- To check license headers without modifying files: `./mvnw license:check` - -### Debugging - -To debug the S3Mock server: - -1. Run the S3MockApplication class in your IDE with debug mode -2. The server will start on ports 9090 (HTTP) and 9191 (HTTPS) -3. Configure your S3 client to connect to these ports - -Alternatively, you can run the Docker container with debug enabled: - -```bash -docker run -p 9090:9090 -p 9191:9191 -e debug=true -t adobe/s3mock -``` - -### Common Issues - -1. **Connection refused errors**: Ensure the S3Mock server is running and the ports are correctly configured. - -2. **SSL certificate errors**: S3Mock uses a self-signed certificate. Configure your client to trust all certificates or use HTTP instead. - -3. **Docker-related errors**: Make sure Docker is running, and you have permissions to create containers. - -### Recommended Development Workflow - -1. Make changes to the code -2. Validate changes with server module tests first (fast path) - - Use the run_test tool on "server/src/test" or on a specific test file/method. - - Prefer this over invoking Maven directly; run_test compiles as needed. -3. Only run a full Maven build when explicitly requested or when cross-module changes demand it - - If building in this environment, prefer fast builds: ./mvnw -pl server -am -DskipDocker clean test - - Do not run mvnw verify unless Docker is confirmed; otherwise add -DskipDocker -4. Run integration tests only when Docker availability is confirmed and when explicitly requested - - Execute via Maven lifecycle: ./mvnw -pl integration-tests -am verify (or add -DskipDocker to skip ITs) -5. Optionally build the Docker image to verify packaging when needed -6. Test with your application to verify real-world usage diff --git a/docs/tasks.md b/docs/tasks.md deleted file mode 100644 index e14c2a35c..000000000 --- a/docs/tasks.md +++ /dev/null @@ -1,188 +0,0 @@ -# S3Mock Improvement Tasks - -This document contains a list of potential improvements for the S3Mock project. Each task is marked with a checkbox that can be checked off when completed. - -## Architecture Improvements - -1. [ ] Implement support for domain-style access to buckets (e.g., `http://bucket.localhost:9090/someKey`) to better match AWS S3 behavior -2. [ ] Add support for AWS S3 operations that are currently not implemented (e.g., bucket policies, website configuration, etc.) -3. [ ] Implement a plugin architecture to allow for custom extensions and behaviors -4. [ ] Add support for S3 event notifications (SNS, SQS, Lambda) -5. [ ] Improve error handling to more closely match AWS S3 error responses -6. [ ] Implement a metrics collection system to monitor performance and usage -7. [ ] Add support for S3 Select for querying object content -8. [ ] Implement support for S3 Inventory reports -9. [ ] Add support for S3 Analytics configurations -10. [ ] Implement S3 Batch Operations - -## Code Quality Improvements - -11. [ ] Increase unit test coverage for controller, service and store layers - -### Task 11 – Test Coverage Plan (Components and Actions) - -Scope and components to cover: -- Controllers (HTTP): BucketController, ObjectController, MultipartController -- Services (business logic): BucketService, ObjectService, MultipartService, Kms* services -- Stores (persistence): BucketStore, ObjectStore, MultipartStore, KmsKeyStore -- XML/DTOs and mappers: request/response XML models, serialization utils -- Utilities: digest, ETag, headers, Range requests, SSE -- Configuration: StoreConfiguration, controller advice/error mapping - -Priorities (short-to-long horizon): -1) High-value happy-path and error-path coverage for controllers with mocked services (fast feedback). -2) Store layer correctness with Spring Boot WebEnvironment.NONE tests (file IO, metadata persistence, edge cases). -3) Service layer behavior with mocked stores (parameter validation, branching, SSE/KMS interactions). -4) XML serialization/deserialization fidelity for commonly used operations. -5) Regression tests for known corner cases (range requests, conditional headers, multipart completion ordering, KMS key validation). - -Concrete test additions (incremental): -- Controllers - - BucketController - - listBuckets returns empty and non-empty results; XML schema shape - - createBucket duplicate name -> proper S3 error code - - deleteBucket non-empty -> proper error - - ObjectController - - putObject with/without Content-MD5; mismatched MD5 -> error - - getObject with Range header (single range) -> 206 + Content-Range - - getObject nonexistent -> 404 S3-style error - - headObject verifies metadata and headers (ETag, Content-Length) - - MultipartController - - initiateMultipartUpload returns uploadId - - uploadPart with invalid partNumber -> error mapping - - completeMultipartUpload out-of-order parts -> consistent ETag behavior -- Services - - ObjectService.storeObject validates metadata, handles SSE headers routing to KMS - - BucketService.deleteBucket checks emptiness guard -- Stores - - ObjectStore - - storeS3ObjectMetadata and getS3ObjectMetadata roundtrip - - list with prefix/delimiter, max-keys, continuation - - delete removes metadata and data file - - BucketStore - - create, list, delete, exist checks - - MultipartStore - - init, addPart, complete, abort state transitions -- XML/DTOs - - Serialize/deserialize ListAllMyBucketsResult, CompleteMultipartUploadResult - -Suggested file locations (server module): -- Controllers: server/src/test/kotlin/com/adobe/testing/s3mock/itlike/controller/*Test.kt (extending BaseControllerTest) -- Services: server/src/test/kotlin/com/adobe/testing/s3mock/service/*Test.kt -- Stores: server/src/test/kotlin/com/adobe/testing/s3mock/store/*Test.kt (extend StoreTestBase) -- DTOs: server/src/test/kotlin/com/adobe/testing/s3mock/xml/*Test.kt - -Execution (fast path per repo guidelines): -- One test class: ./mvnw -pl server test -Dtest=ObjectStoreTest -- One method: ./mvnw -pl server test -Dtest=ObjectStoreTest#testStoreAndGetObject -- Or via tool: run_test server/src/test/kotlin/com/adobe/testing/s3mock/store/ObjectStoreTest.kt - -Acceptance targets for Task 11 completion: -- +10–15% line coverage increase in server module, focusing on controllers and stores -- At least one new test per component category listed above -- Error-path assertions include correct HTTP status and S3 error codes/messages - -Notes: -- Avoid ITs unless Docker available; prefer WebEnvironment.RANDOM_PORT controller tests with mocked services. -- Use provided test bases: BaseControllerTest, StoreTestBase, ServiceTestBase. -- Reuse existing sample files: server/src/test/resources/sampleFile.txt, sampleFile_large.txt, sampleKMSFile.txt. -12. [ ] Refactor synchronization mechanisms in store classes to improve concurrency handling -13. [ ] Implement more comprehensive input validation for S3 API parameters -14. [ ] Add more detailed logging throughout the application for better debugging -15. [ ] Refactor XML serialization/deserialization to use more type-safe approaches -16. [ ] Improve exception handling and error messages -17. [ ] Implement more robust cleanup of temporary files -18. [ ] Add comprehensive JavaDoc to all public APIs -19. [ ] Refactor controller classes to reduce code duplication -20. [ ] Implement more thorough validation of S3 object metadata - -## Performance Improvements - -21. [x] Optimize file storage for large objects -22. [ ] Implement caching for frequently accessed objects -23. [ ] Optimize list operations for buckets with many objects -24. [ ] Improve multipart upload performance -25. [x] Reduce memory usage when handling large files -26. [ ] Optimize XML serialization/deserialization -27. [x] Keep object metadata storage as plain text (JSON) for inspectability (decided against more efficient/binary storage) -28. [ ] Add support for conditional requests to reduce unnecessary data transfer -29. [ ] Optimize concurrent access patterns -30. [ ] Implement more efficient bucket and object locking mechanisms - -## Security Improvements - -31. [ ] Add support for bucket policies -32. [ ] Implement proper authentication and authorization mechanisms -33. [ ] Add support for AWS IAM-style access control -34. [ ] Implement proper handling of AWS signatures for authentication -35. [ ] Add support for server-side encryption with customer-provided keys -36. [ ] Improve SSL/TLS configuration and certificate management -37. [ ] Implement proper input sanitization to prevent injection attacks -38. [ ] Add support for VPC endpoint policies -39. [ ] Implement proper access logging -40. [ ] Add support for AWS CloudTrail-style audit logging - -## Documentation Improvements - -41. [ ] Create comprehensive API documentation -42. [ ] Add more examples for different usage scenarios -43. [ ] Improve README with more detailed setup instructions -44. [ ] Create troubleshooting guide -45. [ ] Add architecture diagrams -46. [ ] Document internal design decisions -47. [ ] Create contributor guidelines -48. [ ] Add more code examples for different programming languages -49. [ ] Create a user guide with common use cases -50. [ ] Document performance characteristics and limitations - -## DevOps Improvements - -51. [ ] Add Docker Compose examples for different scenarios -52. [ ] Implement health checks for container orchestration -53. [ ] Add Kubernetes deployment examples -54. [ ] Improve CI/CD pipeline -55. [ ] Add performance benchmarking to CI process -56. [ ] Implement automated security scanning -57. [ ] Add support for configuration through environment variables for all settings -58. [ ] Improve Docker image size and build time -59. [ ] Add support for container orchestration tools -60. [ ] Implement graceful shutdown and startup procedures - -## Dependency and Technology Updates - -61. [ ] Evaluate migration to Java 21 for improved performance and features -62. [ ] Update to latest Spring Boot version when available -63. [ ] Consider using reactive programming model with Spring WebFlux -64. [ ] Evaluate using non-blocking I/O for file operations -65. [ ] Consider using a more efficient serialization format for metadata storage -66. [ ] Evaluate using a database for metadata storage instead of files -67. [ ] Update AWS SDK dependencies regularly -68. [ ] Consider using GraalVM native image compilation for faster startup -69. [ ] Evaluate using Project Loom virtual threads for improved concurrency -70. [ ] Consider using modern Java language features more extensively (records, sealed classes, etc.) - -## Testing Improvements - -71. [ ] Add more integration tests for edge cases -72. [ ] Implement property-based testing for robust validation -73. [ ] Add performance regression tests -74. [ ] Implement chaos testing for resilience verification -75. [ ] Add more unit tests for utility classes -76. [ ] Implement contract tests for S3 API compatibility -77. [ ] Add tests for concurrent access scenarios -78. [ ] Improve test coverage for error conditions -79. [ ] Add tests for different storage backends -80. [ ] Implement automated compatibility testing with different AWS SDK versions - -## User Experience Improvements - -81. [ ] Add a web-based UI for browsing buckets and objects -82. [ ] Implement a CLI tool for interacting with S3Mock -83. [ ] Add better error messages and troubleshooting information -84. [ ] Improve startup time -85. [ ] Add support for custom endpoints -86. [ ] Implement request/response logging for debugging -87. [ ] Add support for request tracing -88. [ ] Improve configuration options documentation -89. [ ] Add support for programmatic configuration -90. [ ] Implement better defaults for common use cases From e492c5420ec4fe854c21335de9e95d19ea2aec6e Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sat, 21 Feb 2026 17:04:03 +0100 Subject: [PATCH 02/36] feat: let Claude generate skills --- .claude/skills/document/SKILL.md | 126 +++++++++ .claude/skills/implement/SKILL.md | 418 ++++++++++++++++++++++++++++++ .claude/skills/test/SKILL.md | 307 ++++++++++++++++++++++ 3 files changed, 851 insertions(+) create mode 100644 .claude/skills/document/SKILL.md create mode 100644 .claude/skills/implement/SKILL.md create mode 100644 .claude/skills/test/SKILL.md diff --git a/.claude/skills/document/SKILL.md b/.claude/skills/document/SKILL.md new file mode 100644 index 000000000..fec482c45 --- /dev/null +++ b/.claude/skills/document/SKILL.md @@ -0,0 +1,126 @@ +--- +name: document +description: Generate or update project documentation including README, API docs, and architecture guides. Use when asked to document code, create documentation, or explain project structure. +--- + +# Documentation Skill for S3Mock + +This skill helps generate and maintain comprehensive documentation for the S3Mock project. + +## When to Use + +- Creating or updating README files +- Generating API documentation +- Documenting architecture and design decisions +- Creating usage guides and examples +- Explaining S3 operations and implementations + +## Instructions + +When documenting S3Mock, follow these steps: + +1. **Understand Context** + - Review the specific component or feature to document + - Check existing documentation structure in README.md + - Identify the target audience (developers, users, contributors) + +2. **Gather Information** + - Read relevant source code in `server/src/main/` + - Review test files for usage examples + - Check integration tests in `integration-tests/src/test/` + - Reference AWS S3 API documentation for accuracy + +3. **Document Structure** + - Use clear, concise language + - Include code examples where applicable + - Add links to related documentation + - Follow existing documentation style in README.md + - Include both Kotlin and Java examples when relevant + +4. **Key Areas to Document** + - **Configuration**: Environment variables and setup options + - **API Operations**: Supported S3 operations and limitations + - **Usage Examples**: Docker, TestContainers, JUnit, TestNG + - **File System**: Storage structure and metadata + - **Integration**: SDK configuration and best practices + +5. **Documentation Types** + + **API Documentation**: + - Operation name and AWS S3 API link + - Support status (✓ or ✗) + - Limitations or known issues + - Example usage with AWS SDK v1 and v2 + + **Configuration Documentation**: + - Environment variable name + - Default value + - Description and use cases + - Example values + + **Architecture Documentation**: + - Component purpose and responsibilities + - Key classes and their relationships + - Data flow and storage patterns + - Design decisions and trade-offs + +6. **Examples and Code Snippets** + - Provide complete, working examples + - Include necessary imports and setup + - Show both HTTP/HTTPS usage + - Demonstrate error handling + - Reference real test files when possible + +7. **Quality Checklist** + - [ ] Technical accuracy verified + - [ ] Code examples tested + - [ ] Links are valid + - [ ] Consistent with existing docs + - [ ] Clear and understandable + - [ ] Proper markdown formatting + +## Project-Specific Guidelines + +### S3Mock Technical Details + +- **Primary Language**: Kotlin with Java support +- **Framework**: Spring Boot 4.x +- **Testing**: JUnit 5, TestNG, Testcontainers +- **Build Tool**: Maven +- **Container**: Docker (Alpine Linux base) +- **Ports**: HTTP (9090), HTTPS (9191) + +### Important Limitations to Document + +- Path-style access only (no domain-style) +- Presigned URLs accepted but parameters ignored +- Self-signed SSL certificate included +- Mock implementation - not for production +- Limited KMS encryption support (validation only) + +### Documentation Style + +- Use GitHub-flavored Markdown +- Include badges for status and versions +- Provide table of contents for long documents +- Use code blocks with language specification +- Add shell prompts (`$`) for CLI examples +- Include "⚠️ WARNING" or "ℹ️ FYI" callouts for important notes + +## Output Format + +Provide documentation in Markdown format, ready to be integrated into: +- README.md sections +- Separate documentation files +- Code comments and KDoc/Javadoc +- GitHub wiki pages +- CONTRIBUTING.md guidelines + +## Resources + +Refer to these files for context: +- `/README.md` - Main project documentation +- `/CONTRIBUTING.md` - Contribution guidelines +- `/server/src/main/kotlin/` - Core implementation +- `/integration-tests/` - Usage examples +- `/testsupport/` - Test framework integrations diff --git a/.claude/skills/implement/SKILL.md b/.claude/skills/implement/SKILL.md new file mode 100644 index 000000000..dd82217de --- /dev/null +++ b/.claude/skills/implement/SKILL.md @@ -0,0 +1,418 @@ +--- +name: implement +description: Implement new features, fix bugs, or refactor code in S3Mock. Use when asked to add S3 operations, modify existing functionality, or improve code structure. +--- + +# Implementation Skill for S3Mock + +This skill helps implement features and fix bugs in the S3Mock project following best practices. + +## When to Use + +- Implementing new S3 API operations +- Adding features or enhancements +- Fixing bugs and issues +- Refactoring existing code +- Optimizing performance +- Updating dependencies or configurations + +## Instructions + +When implementing changes to S3Mock, follow these steps: + +1. **Understand the Requirement** + - Review AWS S3 API documentation for the feature + - Check existing similar implementations + - Understand expected behavior and edge cases + - Identify affected components (controller, service, store, DTO) + +2. **Architecture Overview** + + S3Mock follows a layered architecture: + + ``` + Controller Layer (HTTP endpoints) + ↓ + Service Layer (Business logic) + ↓ + Store Layer (File system persistence) + ``` + + **Key Packages**: + - `controller/` - REST API endpoints, request/response handling + - `service/` - Business logic, S3 operation implementation + - `store/` - File system storage, metadata management + - `dto/` - Data Transfer Objects, XML/JSON serialization + +3. **Implementation Workflow** + + **Step 1: Define DTOs** + - Create data classes in `dto/` for request/response + - Add Jackson XML annotations for serialization + - Follow AWS S3 XML structure exactly + + ```kotlin + @JacksonXmlRootElement(localName = "CreateBucketConfiguration") + data class CreateBucketConfiguration( + @JacksonXmlProperty(localName = "LocationConstraint") + val locationConstraint: LocationConstraint? = null + ) + ``` + + **Step 2: Update Store Layer** + - Modify `store/` classes for data persistence + - Update `BucketMetadata` or `S3ObjectMetadata` as needed + - Ensure proper file system structure + - Handle versioning if applicable + + ```kotlin + // In BucketStore or ObjectStore + fun storeSomething(bucketName: String, data: SomeData) { + val bucket = getBucketMetadata(bucketName) + // Store data to file system + Files.write(getDataPath(bucket), serialize(data)) + } + ``` + + **Step 3: Implement Service Layer** + - Add business logic in `service/ObjectService` or create new service + - Validate inputs and handle errors + - Use store layer for persistence + - Generate appropriate responses + + ```kotlin + @Service + class ObjectService( + private val bucketStore: BucketStore, + private val objectStore: ObjectStore + ) { + fun performOperation(bucket: String, key: String): Result { + // Validate + bucketStore.getBucketMetadata(bucket) ?: throw NoSuchBucketException() + + // Business logic + val result = objectStore.doSomething(bucket, key) + + // Return + return Result(result) + } + } + ``` + + **Step 4: Add Controller Endpoint** + - Create or update controller in `controller/` + - Map HTTP method and path + - Handle request headers and query parameters + - Set response headers (ETag, Content-Type, etc.) + - Return proper HTTP status codes + + ```kotlin + @RestController + @RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") + class ObjectController( + private val objectService: ObjectService + ) { + @GetMapping( + "/{bucketName:.+}/{*key}", + produces = [MediaType.APPLICATION_XML_VALUE] + ) + fun getObject( + @PathVariable bucketName: String, + @PathVariable key: String, + @RequestHeader headers: Map + ): ResponseEntity { + val result = objectService.getObject(bucketName, key) + + return ResponseEntity + .ok() + .header("ETag", result.etag) + .header("Content-Type", result.contentType) + .body(result.data) + } + } + ``` + +4. **Code Style and Conventions** + + **Language**: Kotlin with Java interoperability + + **Naming**: + - Classes: PascalCase + - Functions: camelCase + - Properties: camelCase + - Constants: SCREAMING_SNAKE_CASE + - Packages: lowercase + + **Kotlin Conventions**: + ```kotlin + // Use data classes for DTOs + data class Result(val value: String) + + // Use nullable types appropriately + fun find(id: String): Result? + + // Use expression bodies for simple functions + fun isValid(value: String): Boolean = value.isNotEmpty() + + // Use named parameters for clarity + createObject( + bucket = "test-bucket", + key = "test-key", + data = byteArray + ) + ``` + + **Spring Annotations**: + - `@Service` for service classes + - `@RestController` for controllers + - `@Component` for other beans + - `@Autowired` or constructor injection (preferred) + +5. **Error Handling** + + **S3 Exceptions**: + ```kotlin + // Use custom exceptions from com.adobe.testing.s3mock.dto + throw NoSuchBucketException("Bucket does not exist: $bucketName") + throw NoSuchKeyException("Key does not exist: $key") + throw BucketAlreadyExistsException("Bucket already exists: $bucketName") + ``` + + **HTTP Status Codes**: + - 200 OK - Successful operation + - 204 No Content - Successful delete + - 404 Not Found - Bucket/object not found + - 409 Conflict - Resource conflict + - 500 Internal Server Error - Unexpected errors + +6. **XML Serialization** + + Use Jackson XML annotations: + ```kotlin + @JacksonXmlRootElement(localName = "ListBucketResult") + data class ListBucketResult( + @JacksonXmlProperty(localName = "Name") + val name: String, + + @JacksonXmlProperty(localName = "Prefix") + val prefix: String? = null, + + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "Contents") + val contents: List = emptyList() + ) + ``` + +7. **Testing** + + **Write Tests**: + - Add unit tests in `server/src/test/` + - Add integration tests in `integration-tests/src/test/` + - Test both success and error cases + - Test with AWS SDK v1 and v2 if applicable + + **Test Coverage**: + - New features: 80%+ coverage + - Bug fixes: Add test reproducing the bug + - Refactoring: Maintain existing coverage + +8. **Documentation** + + **KDoc Comments**: + ```kotlin + /** + * Retrieves an object from the specified bucket. + * + * @param bucketName The bucket name + * @param key The object key + * @return The object data and metadata + * @throws NoSuchBucketException if bucket doesn't exist + * @throws NoSuchKeyException if object doesn't exist + */ + fun getObject(bucketName: String, key: String): S3Object + ``` + + **Update README**: Add to supported operations table if implementing new S3 API + +9. **Build and Verify** + + ```bash + # Format code + ./mvnw ktlint:format + + # Run checkstyle + ./mvnw checkstyle:check + + # Run unit tests + ./mvnw test + + # Run integration tests + ./mvnw verify + + # Build without Docker + ./mvnw clean install -DskipDocker + ``` + +10. **Implementation Checklist** + - [ ] AWS S3 API documentation reviewed + - [ ] DTOs created with proper XML annotations + - [ ] Store layer updated for persistence + - [ ] Service layer implements business logic + - [ ] Controller endpoint added with proper mapping + - [ ] Error handling implemented + - [ ] Unit tests written + - [ ] Integration tests written + - [ ] Code formatted and linted + - [ ] Documentation updated + - [ ] Build passes locally + +## Project-Specific Guidelines + +### S3Mock Technology Stack + +- **Language**: Kotlin 2.3.0 (target: JVM 17) +- **Framework**: Spring Boot 4.0.x +- **Build**: Maven 3.9+ +- **Serialization**: Jackson XML/JSON +- **Testing**: JUnit 5, Mockito, AssertJ +- **Container**: Docker Alpine Linux + +### File System Structure + +Objects stored at: +``` +////binaryData +////objectMetadata.json +///bucketMetadata.json +``` + +With versioning: +``` +////-binaryData +////-objectMetadata.json +``` + +### Configuration + +Use environment variables (see `S3MockProperties.kt`): +```kotlin +@ConfigurationProperties(prefix = "com.adobe.testing.s3mock.store") +data class StoreProperties( + val root: Path, + val retainFilesOnExit: Boolean = false, + val validKmsKeys: Set = emptySet(), + val initialBuckets: Set = emptySet(), + val region: Region = Region.US_EAST_1 +) +``` + +### Common Patterns + +**ETag Generation**: +```kotlin +val etag = DigestUtils.md5Hex(data) +``` + +**UUID for Objects**: +```kotlin +val uuid = UUID.randomUUID().toString() +``` + +**Date Formatting**: +```kotlin +// ISO 8601 format for S3 API +val formatted = Instant.now().toString() +``` + +**Response Headers**: +```kotlin +ResponseEntity + .ok() + .header("ETag", "\"$etag\"") + .header("Last-Modified", lastModified) + .header("Content-Type", contentType) + .header("Content-Length", size.toString()) + .body(data) +``` + +### Implementing New S3 Operations + +1. Check AWS S3 API docs: https://docs.aws.amazon.com/AmazonS3/latest/API/ +2. Find operation in table (lines 84-188 in README.md) +3. Create branch from `main` or maintenance branch +4. Follow implementation workflow above +5. Update README.md supported operations table +6. Run full test suite +7. Create PR with clear description + +### Performance Considerations + +- Use streaming for large files +- Avoid loading entire files into memory +- Use `Files.copy()` for file operations +- Consider parallel streams for batch operations +- Profile with large datasets if needed + +### Security Considerations + +- Validate all inputs (bucket names, keys, headers) +- Sanitize file paths to prevent directory traversal +- Use secure random for UUIDs +- Don't log sensitive data +- Follow Spring Security best practices + +## Common Implementation Patterns + +### Adding a New Bucket Operation + +```kotlin +// 1. DTO (if needed) +@JacksonXmlRootElement(localName = "OperationRequest") +data class OperationRequest(...) + +// 2. Service +fun performBucketOperation(bucketName: String): Result { + val bucket = bucketStore.getBucketMetadata(bucketName) + ?: throw NoSuchBucketException(bucketName) + // Logic here + return Result(...) +} + +// 3. Controller +@PostMapping("/{bucketName:.+}?operation") +fun bucketOperation( + @PathVariable bucketName: String, + @RequestBody request: OperationRequest +): ResponseEntity { + val result = bucketService.performBucketOperation(bucketName) + return ResponseEntity.ok(result) +} +``` + +### Adding a New Object Operation + +```kotlin +// Similar to bucket but with key parameter +@GetMapping("/{bucketName:.+}/{*key}") +fun objectOperation( + @PathVariable bucketName: String, + @PathVariable key: String +): ResponseEntity { + val result = objectService.performOperation(bucketName, key) + return ResponseEntity.ok(result) +} +``` + +## Resources + +- **AWS S3 API**: https://docs.aws.amazon.com/AmazonS3/latest/API/ +- **Spring Boot Docs**: https://docs.spring.io/spring-boot/docs/current/reference/ +- **Kotlin Docs**: https://kotlinlang.org/docs/home.html +- **Jackson XML**: https://github.com/FasterXML/jackson-dataformat-xml +- **S3Mock Main Classes**: + - `S3MockApplication.kt` - Application entry point + - `ObjectController.kt` - Object operations + - `BucketController.kt` - Bucket operations + - `ObjectService.kt` - Object business logic + - `BucketStore.kt` - Bucket storage + - `ObjectStore.kt` - Object storage diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md new file mode 100644 index 000000000..afc66ebde --- /dev/null +++ b/.claude/skills/test/SKILL.md @@ -0,0 +1,307 @@ +--- +name: test +description: Write, update, or fix tests for S3Mock including unit tests, integration tests, and test configurations. Use when asked to test code, create test cases, or fix failing tests. +--- + +# Test Skill for S3Mock + +This skill helps create and maintain comprehensive tests for the S3Mock project. + +## When to Use + +- Writing new unit or integration tests +- Fixing failing tests +- Adding test coverage for new features +- Updating tests after code changes +- Creating test data and fixtures +- Debugging test failures + +## Instructions + +When writing tests for S3Mock, follow these steps: + +1. **Identify Test Type** + - **Unit Tests**: Test individual components in isolation (`server/src/test/`) + - **Integration Tests**: Test S3Mock with real AWS SDK clients (`integration-tests/src/test/`) + - **Support Tests**: Test JUnit5, TestNG, Testcontainers integrations (`testsupport/*/src/test/`) + +2. **Understand Test Context** + - Locate relevant source code to test + - Review existing test patterns in the module + - Identify test framework (JUnit 5, TestNG, Mockito) + - Determine if S3Mock instance is needed + +3. **Test Structure for Integration Tests** + + ```kotlin + // Extend S3TestBase for S3Mock setup + class MyFeatureIT : S3TestBase() { + + @Test + fun `should perform S3 operation`() { + // Arrange - setup buckets, objects, test data + val bucketName = "test-bucket" + s3Client.createBucket(bucketName) + + // Act - perform the operation + val result = s3Client.someOperation(bucketName, "key") + + // Assert - verify behavior + assertThat(result).isNotNull() + } + } + ``` + +4. **Test Structure for Unit Tests** + + ```kotlin + @ExtendWith(MockitoExtension::class) + internal class MyServiceTest { + + @Mock + private lateinit var dependency: SomeDependency + + @InjectMocks + private lateinit var service: MyService + + @Test + fun `should handle valid input`() { + // Arrange + whenever(dependency.method()).thenReturn("result") + + // Act + val result = service.doSomething() + + // Assert + assertThat(result).isEqualTo("expected") + verify(dependency).method() + } + } + ``` + +5. **Test Naming Conventions** + - Integration tests: End with `IT` (e.g., `BucketIT.kt`) + - Unit tests: End with `Test` (e.g., `BucketServiceTest.kt`) + - Use descriptive test names with backticks in Kotlin + - Format: `` `should when ` `` + +6. **S3Mock Test Setup** + + **For Integration Tests**: + ```kotlin + // Tests extend S3TestBase which provides: + // - s3Client: S3Client (AWS SDK v2) + // - s3ClientV1: AmazonS3 (AWS SDK v1) + // - httpClient: Configured HTTP client + // - serviceEndpoint: S3Mock endpoint URL + ``` + + **For JUnit 5 Extensions**: + ```kotlin + @ExtendWith(S3MockExtension::class) + class MyTest { + @Test + fun test(s3Client: S3Client) { + // S3Client injected automatically + } + } + ``` + + **For Testcontainers**: + ```kotlin + val s3MockContainer = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + s3MockContainer.start() + ``` + +7. **Common Test Patterns** + + **Testing Object Operations**: + ```kotlin + @Test + fun `should store and retrieve object`() { + val bucketName = "test-bucket" + val key = "test-key" + val content = "test content" + + s3Client.createBucket(bucketName) + s3Client.putObject(bucketName, key, content.toByteArray()) + + val response = s3Client.getObject(bucketName, key) + val retrieved = response.readAllBytes().decodeToString() + + assertThat(retrieved).isEqualTo(content) + } + ``` + + **Testing Bucket Operations**: + ```kotlin + @Test + fun `should create and list buckets`() { + val bucketName = "my-bucket" + + s3Client.createBucket(bucketName) + val buckets = s3Client.listBuckets() + + assertThat(buckets).anyMatch { it.name() == bucketName } + } + ``` + + **Testing Multipart Uploads**: + ```kotlin + @Test + fun `should complete multipart upload`() { + val bucketName = "test-bucket" + val key = "large-file" + + s3Client.createBucket(bucketName) + + val uploadId = s3Client.createMultipartUpload( + CreateMultipartUploadRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ).uploadId() + + val parts = listOf( + uploadPart(bucketName, key, uploadId, 1, "part1".toByteArray()), + uploadPart(bucketName, key, uploadId, 2, "part2".toByteArray()) + ) + + s3Client.completeMultipartUpload( + CompleteMultipartUploadRequest.builder() + .bucket(bucketName) + .key(key) + .uploadId(uploadId) + .multipartUpload(CompletedMultipartUpload.builder().parts(parts).build()) + .build() + ) + + val obj = s3Client.getObject(bucketName, key) + assertThat(obj.readAllBytes().decodeToString()).isEqualTo("part1part2") + } + ``` + + **Testing Error Cases**: + ```kotlin + @Test + fun `should throw exception for non-existent bucket`() { + assertThrows { + s3Client.getObject("non-existent-bucket", "key") + } + } + ``` + +8. **Assertions and Matchers** + - Use AssertJ: `assertThat(value).isEqualTo(expected)` + - Use XMLUnit for XML: `assertThat(xml).and(expected).areIdentical()` + - Use Kotlin test assertions: `assertThrows { ... }` + - Verify mock interactions: `verify(mock).method()` + +9. **Test Data Management** + - Create test data in test setup + - Use meaningful test data that reflects real usage + - Clean up resources (usually automatic with test lifecycle) + - Use random bucket names if needed: `"bucket-${UUID.randomUUID()}"` + +10. **Testing Best Practices** + - **Isolation**: Each test should be independent + - **Clarity**: Test one thing per test method + - **Speed**: Keep tests fast, prefer unit tests + - **Coverage**: Test happy path and error cases + - **Readability**: Use descriptive names and clear assertions + - **Maintenance**: Update tests with code changes + +11. **Running Tests** + + ```bash + # Run all tests + ./mvnw clean verify + + # Run unit tests only + ./mvnw test + + # Run integration tests + ./mvnw verify -pl integration-tests + + # Run specific test + ./mvnw test -Dtest=BucketIT + + # Skip Docker build + ./mvnw verify -DskipDocker + ``` + +12. **Test Quality Checklist** + - [ ] Test name clearly describes what is tested + - [ ] Arrange-Act-Assert structure followed + - [ ] Test is independent and repeatable + - [ ] Both success and failure cases covered + - [ ] Assertions are specific and meaningful + - [ ] No hard-coded waits or sleeps + - [ ] Resources properly managed + - [ ] Test runs successfully locally + +## Project-Specific Guidelines + +### S3Mock Test Architecture + +- **Base Classes**: `S3TestBase` provides S3Client setup +- **Test Location**: + - Unit: `server/src/test/kotlin/com/adobe/testing/s3mock/` + - Integration: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/` +- **Languages**: Tests written in Kotlin +- **Frameworks**: JUnit 5, Mockito, AssertJ +- **Execution**: Parallel test execution enabled + +### Testing S3 Operations + +When testing S3 operations: +1. Verify response structure matches S3 API +2. Check HTTP status codes +3. Validate XML/JSON response bodies +4. Test with both SDK v1 and v2 if applicable +5. Verify headers (ETag, Content-Type, etc.) +6. Test presigned URLs if relevant + +### Testing File System Integration + +When testing storage: +1. Verify files created in correct location +2. Check metadata serialization +3. Validate bucket and object UUIDs +4. Test versioning if enabled +5. Verify cleanup on shutdown (if configured) + +### Mock Configuration in Tests + +```kotlin +// Configure test-specific settings +@TestPropertySource( + properties = [ + "com.adobe.testing.s3mock.store.retainFilesOnExit=false", + "com.adobe.testing.s3mock.store.validKmsKeys=test-key-id" + ] +) +``` + +## Common Issues and Solutions + +**Issue**: Test fails with "Connection refused" +- **Solution**: Ensure S3Mock is started before test runs, check port configuration + +**Issue**: Test fails intermittently +- **Solution**: Check for race conditions, ensure proper test isolation + +**Issue**: Test can't find bucket +- **Solution**: Verify bucket creation in test setup, check bucket name + +**Issue**: XML comparison fails +- **Solution**: Use XMLUnit's `areIdentical()` ignoring whitespace and comments + +## Resources + +- **Test Base**: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt` +- **Examples**: All `*IT.kt` files in `integration-tests/src/test/` +- **SDK Usage**: AWS SDK v2 documentation +- **Assertions**: AssertJ documentation +- **Mocking**: Mockito-Kotlin documentation From 3b23edcec21d27fefc98969860dab1ef82c17ff7 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sat, 21 Feb 2026 17:13:34 +0100 Subject: [PATCH 03/36] feat: let Claude generate generic skills These skills are reusable across projects --- .claude/skills/document/SKILL.md | 138 ++------- .claude/skills/implement/SKILL.md | 446 +++--------------------------- .claude/skills/test/SKILL.md | 325 +++------------------- 3 files changed, 105 insertions(+), 804 deletions(-) diff --git a/.claude/skills/document/SKILL.md b/.claude/skills/document/SKILL.md index fec482c45..35a91122a 100644 --- a/.claude/skills/document/SKILL.md +++ b/.claude/skills/document/SKILL.md @@ -1,126 +1,44 @@ --- name: document -description: Generate or update project documentation including README, API docs, and architecture guides. Use when asked to document code, create documentation, or explain project structure. +description: Generate or update project documentation. Use when asked to document code, create docs, or explain features. --- -# Documentation Skill for S3Mock +# Documentation Skill -This skill helps generate and maintain comprehensive documentation for the S3Mock project. +Generate and maintain project documentation. ## When to Use - Creating or updating README files - Generating API documentation -- Documenting architecture and design decisions -- Creating usage guides and examples -- Explaining S3 operations and implementations +- Documenting features and usage +- Writing guides and examples ## Instructions -When documenting S3Mock, follow these steps: - 1. **Understand Context** - - Review the specific component or feature to document - - Check existing documentation structure in README.md - - Identify the target audience (developers, users, contributors) + - Review the code or feature to document + - Check existing documentation style + - Identify target audience 2. **Gather Information** - - Read relevant source code in `server/src/main/` - - Review test files for usage examples - - Check integration tests in `integration-tests/src/test/` - - Reference AWS S3 API documentation for accuracy - -3. **Document Structure** - - Use clear, concise language - - Include code examples where applicable - - Add links to related documentation - - Follow existing documentation style in README.md - - Include both Kotlin and Java examples when relevant - -4. **Key Areas to Document** - - **Configuration**: Environment variables and setup options - - **API Operations**: Supported S3 operations and limitations - - **Usage Examples**: Docker, TestContainers, JUnit, TestNG - - **File System**: Storage structure and metadata - - **Integration**: SDK configuration and best practices - -5. **Documentation Types** - - **API Documentation**: - - Operation name and AWS S3 API link - - Support status (✓ or ✗) - - Limitations or known issues - - Example usage with AWS SDK v1 and v2 - - **Configuration Documentation**: - - Environment variable name - - Default value - - Description and use cases - - Example values - - **Architecture Documentation**: - - Component purpose and responsibilities - - Key classes and their relationships - - Data flow and storage patterns - - Design decisions and trade-offs - -6. **Examples and Code Snippets** - - Provide complete, working examples - - Include necessary imports and setup - - Show both HTTP/HTTPS usage - - Demonstrate error handling - - Reference real test files when possible - -7. **Quality Checklist** - - [ ] Technical accuracy verified - - [ ] Code examples tested - - [ ] Links are valid - - [ ] Consistent with existing docs - - [ ] Clear and understandable - - [ ] Proper markdown formatting - -## Project-Specific Guidelines - -### S3Mock Technical Details - -- **Primary Language**: Kotlin with Java support -- **Framework**: Spring Boot 4.x -- **Testing**: JUnit 5, TestNG, Testcontainers -- **Build Tool**: Maven -- **Container**: Docker (Alpine Linux base) -- **Ports**: HTTP (9090), HTTPS (9191) - -### Important Limitations to Document - -- Path-style access only (no domain-style) -- Presigned URLs accepted but parameters ignored -- Self-signed SSL certificate included -- Mock implementation - not for production -- Limited KMS encryption support (validation only) - -### Documentation Style - -- Use GitHub-flavored Markdown -- Include badges for status and versions -- Provide table of contents for long documents -- Use code blocks with language specification -- Add shell prompts (`$`) for CLI examples -- Include "⚠️ WARNING" or "ℹ️ FYI" callouts for important notes - -## Output Format - -Provide documentation in Markdown format, ready to be integrated into: -- README.md sections -- Separate documentation files -- Code comments and KDoc/Javadoc -- GitHub wiki pages -- CONTRIBUTING.md guidelines - -## Resources - -Refer to these files for context: -- `/README.md` - Main project documentation -- `/CONTRIBUTING.md` - Contribution guidelines -- `/server/src/main/kotlin/` - Core implementation -- `/integration-tests/` - Usage examples -- `/testsupport/` - Test framework integrations + - Read source code and comments + - Review test files for examples + - Check external references (APIs, specs) + +3. **Write Clear Documentation** + - Use concise, clear language + - Include code examples + - Add links to related docs + - Follow project conventions (see AGENTS.md) + +4. **Quality Check** + - Technical accuracy verified + - Examples work correctly + - Links are valid + - Consistent style + - Proper formatting + +## Output + +Provide documentation ready to integrate into project files. diff --git a/.claude/skills/implement/SKILL.md b/.claude/skills/implement/SKILL.md index dd82217de..addf077c7 100644 --- a/.claude/skills/implement/SKILL.md +++ b/.claude/skills/implement/SKILL.md @@ -1,418 +1,58 @@ --- name: implement -description: Implement new features, fix bugs, or refactor code in S3Mock. Use when asked to add S3 operations, modify existing functionality, or improve code structure. +description: Implement features, fix bugs, or refactor code. Use when asked to add functionality, modify code, or improve structure. --- -# Implementation Skill for S3Mock +# Implementation Skill -This skill helps implement features and fix bugs in the S3Mock project following best practices. +Implement features and fix bugs following best practices. ## When to Use -- Implementing new S3 API operations -- Adding features or enhancements -- Fixing bugs and issues -- Refactoring existing code +- Adding new features +- Fixing bugs +- Refactoring code - Optimizing performance -- Updating dependencies or configurations +- Updating dependencies ## Instructions -When implementing changes to S3Mock, follow these steps: - 1. **Understand the Requirement** - - Review AWS S3 API documentation for the feature + - Review specifications or issue description - Check existing similar implementations - Understand expected behavior and edge cases - - Identify affected components (controller, service, store, DTO) - -2. **Architecture Overview** - - S3Mock follows a layered architecture: - - ``` - Controller Layer (HTTP endpoints) - ↓ - Service Layer (Business logic) - ↓ - Store Layer (File system persistence) - ``` - - **Key Packages**: - - `controller/` - REST API endpoints, request/response handling - - `service/` - Business logic, S3 operation implementation - - `store/` - File system storage, metadata management - - `dto/` - Data Transfer Objects, XML/JSON serialization - -3. **Implementation Workflow** - - **Step 1: Define DTOs** - - Create data classes in `dto/` for request/response - - Add Jackson XML annotations for serialization - - Follow AWS S3 XML structure exactly - - ```kotlin - @JacksonXmlRootElement(localName = "CreateBucketConfiguration") - data class CreateBucketConfiguration( - @JacksonXmlProperty(localName = "LocationConstraint") - val locationConstraint: LocationConstraint? = null - ) - ``` - - **Step 2: Update Store Layer** - - Modify `store/` classes for data persistence - - Update `BucketMetadata` or `S3ObjectMetadata` as needed - - Ensure proper file system structure - - Handle versioning if applicable - - ```kotlin - // In BucketStore or ObjectStore - fun storeSomething(bucketName: String, data: SomeData) { - val bucket = getBucketMetadata(bucketName) - // Store data to file system - Files.write(getDataPath(bucket), serialize(data)) - } - ``` - - **Step 3: Implement Service Layer** - - Add business logic in `service/ObjectService` or create new service - - Validate inputs and handle errors - - Use store layer for persistence - - Generate appropriate responses - - ```kotlin - @Service - class ObjectService( - private val bucketStore: BucketStore, - private val objectStore: ObjectStore - ) { - fun performOperation(bucket: String, key: String): Result { - // Validate - bucketStore.getBucketMetadata(bucket) ?: throw NoSuchBucketException() - - // Business logic - val result = objectStore.doSomething(bucket, key) - - // Return - return Result(result) - } - } - ``` - - **Step 4: Add Controller Endpoint** - - Create or update controller in `controller/` - - Map HTTP method and path - - Handle request headers and query parameters - - Set response headers (ETag, Content-Type, etc.) - - Return proper HTTP status codes - - ```kotlin - @RestController - @RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") - class ObjectController( - private val objectService: ObjectService - ) { - @GetMapping( - "/{bucketName:.+}/{*key}", - produces = [MediaType.APPLICATION_XML_VALUE] - ) - fun getObject( - @PathVariable bucketName: String, - @PathVariable key: String, - @RequestHeader headers: Map - ): ResponseEntity { - val result = objectService.getObject(bucketName, key) - - return ResponseEntity - .ok() - .header("ETag", result.etag) - .header("Content-Type", result.contentType) - .body(result.data) - } - } - ``` - -4. **Code Style and Conventions** - - **Language**: Kotlin with Java interoperability - - **Naming**: - - Classes: PascalCase - - Functions: camelCase - - Properties: camelCase - - Constants: SCREAMING_SNAKE_CASE - - Packages: lowercase - - **Kotlin Conventions**: - ```kotlin - // Use data classes for DTOs - data class Result(val value: String) - - // Use nullable types appropriately - fun find(id: String): Result? - - // Use expression bodies for simple functions - fun isValid(value: String): Boolean = value.isNotEmpty() - - // Use named parameters for clarity - createObject( - bucket = "test-bucket", - key = "test-key", - data = byteArray - ) - ``` - - **Spring Annotations**: - - `@Service` for service classes - - `@RestController` for controllers - - `@Component` for other beans - - `@Autowired` or constructor injection (preferred) - -5. **Error Handling** - - **S3 Exceptions**: - ```kotlin - // Use custom exceptions from com.adobe.testing.s3mock.dto - throw NoSuchBucketException("Bucket does not exist: $bucketName") - throw NoSuchKeyException("Key does not exist: $key") - throw BucketAlreadyExistsException("Bucket already exists: $bucketName") - ``` - - **HTTP Status Codes**: - - 200 OK - Successful operation - - 204 No Content - Successful delete - - 404 Not Found - Bucket/object not found - - 409 Conflict - Resource conflict - - 500 Internal Server Error - Unexpected errors - -6. **XML Serialization** - - Use Jackson XML annotations: - ```kotlin - @JacksonXmlRootElement(localName = "ListBucketResult") - data class ListBucketResult( - @JacksonXmlProperty(localName = "Name") - val name: String, - - @JacksonXmlProperty(localName = "Prefix") - val prefix: String? = null, - - @JacksonXmlElementWrapper(useWrapping = false) - @JacksonXmlProperty(localName = "Contents") - val contents: List = emptyList() - ) - ``` - -7. **Testing** - - **Write Tests**: - - Add unit tests in `server/src/test/` - - Add integration tests in `integration-tests/src/test/` - - Test both success and error cases - - Test with AWS SDK v1 and v2 if applicable - - **Test Coverage**: - - New features: 80%+ coverage - - Bug fixes: Add test reproducing the bug - - Refactoring: Maintain existing coverage - -8. **Documentation** - - **KDoc Comments**: - ```kotlin - /** - * Retrieves an object from the specified bucket. - * - * @param bucketName The bucket name - * @param key The object key - * @return The object data and metadata - * @throws NoSuchBucketException if bucket doesn't exist - * @throws NoSuchKeyException if object doesn't exist - */ - fun getObject(bucketName: String, key: String): S3Object - ``` - - **Update README**: Add to supported operations table if implementing new S3 API - -9. **Build and Verify** - - ```bash - # Format code - ./mvnw ktlint:format - - # Run checkstyle - ./mvnw checkstyle:check - - # Run unit tests - ./mvnw test - - # Run integration tests - ./mvnw verify - - # Build without Docker - ./mvnw clean install -DskipDocker - ``` - -10. **Implementation Checklist** - - [ ] AWS S3 API documentation reviewed - - [ ] DTOs created with proper XML annotations - - [ ] Store layer updated for persistence - - [ ] Service layer implements business logic - - [ ] Controller endpoint added with proper mapping - - [ ] Error handling implemented - - [ ] Unit tests written - - [ ] Integration tests written - - [ ] Code formatted and linted - - [ ] Documentation updated - - [ ] Build passes locally - -## Project-Specific Guidelines - -### S3Mock Technology Stack - -- **Language**: Kotlin 2.3.0 (target: JVM 17) -- **Framework**: Spring Boot 4.0.x -- **Build**: Maven 3.9+ -- **Serialization**: Jackson XML/JSON -- **Testing**: JUnit 5, Mockito, AssertJ -- **Container**: Docker Alpine Linux - -### File System Structure - -Objects stored at: -``` -////binaryData -////objectMetadata.json -///bucketMetadata.json -``` - -With versioning: -``` -////-binaryData -////-objectMetadata.json -``` - -### Configuration - -Use environment variables (see `S3MockProperties.kt`): -```kotlin -@ConfigurationProperties(prefix = "com.adobe.testing.s3mock.store") -data class StoreProperties( - val root: Path, - val retainFilesOnExit: Boolean = false, - val validKmsKeys: Set = emptySet(), - val initialBuckets: Set = emptySet(), - val region: Region = Region.US_EAST_1 -) -``` - -### Common Patterns - -**ETag Generation**: -```kotlin -val etag = DigestUtils.md5Hex(data) -``` - -**UUID for Objects**: -```kotlin -val uuid = UUID.randomUUID().toString() -``` - -**Date Formatting**: -```kotlin -// ISO 8601 format for S3 API -val formatted = Instant.now().toString() -``` - -**Response Headers**: -```kotlin -ResponseEntity - .ok() - .header("ETag", "\"$etag\"") - .header("Last-Modified", lastModified) - .header("Content-Type", contentType) - .header("Content-Length", size.toString()) - .body(data) -``` - -### Implementing New S3 Operations - -1. Check AWS S3 API docs: https://docs.aws.amazon.com/AmazonS3/latest/API/ -2. Find operation in table (lines 84-188 in README.md) -3. Create branch from `main` or maintenance branch -4. Follow implementation workflow above -5. Update README.md supported operations table -6. Run full test suite -7. Create PR with clear description - -### Performance Considerations - -- Use streaming for large files -- Avoid loading entire files into memory -- Use `Files.copy()` for file operations -- Consider parallel streams for batch operations -- Profile with large datasets if needed - -### Security Considerations - -- Validate all inputs (bucket names, keys, headers) -- Sanitize file paths to prevent directory traversal -- Use secure random for UUIDs -- Don't log sensitive data -- Follow Spring Security best practices - -## Common Implementation Patterns - -### Adding a New Bucket Operation - -```kotlin -// 1. DTO (if needed) -@JacksonXmlRootElement(localName = "OperationRequest") -data class OperationRequest(...) - -// 2. Service -fun performBucketOperation(bucketName: String): Result { - val bucket = bucketStore.getBucketMetadata(bucketName) - ?: throw NoSuchBucketException(bucketName) - // Logic here - return Result(...) -} - -// 3. Controller -@PostMapping("/{bucketName:.+}?operation") -fun bucketOperation( - @PathVariable bucketName: String, - @RequestBody request: OperationRequest -): ResponseEntity { - val result = bucketService.performBucketOperation(bucketName) - return ResponseEntity.ok(result) -} -``` - -### Adding a New Object Operation - -```kotlin -// Similar to bucket but with key parameter -@GetMapping("/{bucketName:.+}/{*key}") -fun objectOperation( - @PathVariable bucketName: String, - @PathVariable key: String -): ResponseEntity { - val result = objectService.performOperation(bucketName, key) - return ResponseEntity.ok(result) -} -``` - -## Resources - -- **AWS S3 API**: https://docs.aws.amazon.com/AmazonS3/latest/API/ -- **Spring Boot Docs**: https://docs.spring.io/spring-boot/docs/current/reference/ -- **Kotlin Docs**: https://kotlinlang.org/docs/home.html -- **Jackson XML**: https://github.com/FasterXML/jackson-dataformat-xml -- **S3Mock Main Classes**: - - `S3MockApplication.kt` - Application entry point - - `ObjectController.kt` - Object operations - - `BucketController.kt` - Bucket operations - - `ObjectService.kt` - Object business logic - - `BucketStore.kt` - Bucket storage - - `ObjectStore.kt` - Object storage + - Review project architecture (see AGENTS.md) + +2. **Plan the Implementation** + - Identify affected components + - Consider architectural patterns + - Plan for error handling + - Think about testing + +3. **Write Clean Code** + - Follow project style guide (see AGENTS.md) + - Use appropriate naming conventions + - Add necessary comments + - Handle errors properly + - Consider edge cases + +4. **Follow Project Standards** + - Match existing code style + - Use project utilities and patterns + - Follow architectural guidelines (see AGENTS.md) + - Maintain consistency + +5. **Verify Implementation** + - Code compiles/builds + - Tests pass + - Linting passes + - No regressions + +6. **Document Changes** + - Add/update code comments + - Update relevant documentation + - Document breaking changes + +## Output + +Provide clean, tested, well-documented code following project conventions. diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md index afc66ebde..cd94da828 100644 --- a/.claude/skills/test/SKILL.md +++ b/.claude/skills/test/SKILL.md @@ -1,307 +1,50 @@ --- name: test -description: Write, update, or fix tests for S3Mock including unit tests, integration tests, and test configurations. Use when asked to test code, create test cases, or fix failing tests. +description: Write, update, or fix tests. Use when asked to test code, create test cases, or debug failing tests. --- -# Test Skill for S3Mock +# Test Skill -This skill helps create and maintain comprehensive tests for the S3Mock project. +Create and maintain comprehensive tests. ## When to Use - Writing new unit or integration tests - Fixing failing tests -- Adding test coverage for new features -- Updating tests after code changes -- Creating test data and fixtures -- Debugging test failures +- Adding test coverage +- Creating test fixtures ## Instructions -When writing tests for S3Mock, follow these steps: - 1. **Identify Test Type** - - **Unit Tests**: Test individual components in isolation (`server/src/test/`) - - **Integration Tests**: Test S3Mock with real AWS SDK clients (`integration-tests/src/test/`) - - **Support Tests**: Test JUnit5, TestNG, Testcontainers integrations (`testsupport/*/src/test/`) + - Unit tests: Test components in isolation + - Integration tests: Test system interactions + - End-to-end tests: Test full workflows 2. **Understand Test Context** - - Locate relevant source code to test - - Review existing test patterns in the module - - Identify test framework (JUnit 5, TestNG, Mockito) - - Determine if S3Mock instance is needed - -3. **Test Structure for Integration Tests** - - ```kotlin - // Extend S3TestBase for S3Mock setup - class MyFeatureIT : S3TestBase() { - - @Test - fun `should perform S3 operation`() { - // Arrange - setup buckets, objects, test data - val bucketName = "test-bucket" - s3Client.createBucket(bucketName) - - // Act - perform the operation - val result = s3Client.someOperation(bucketName, "key") - - // Assert - verify behavior - assertThat(result).isNotNull() - } - } - ``` - -4. **Test Structure for Unit Tests** - - ```kotlin - @ExtendWith(MockitoExtension::class) - internal class MyServiceTest { - - @Mock - private lateinit var dependency: SomeDependency - - @InjectMocks - private lateinit var service: MyService - - @Test - fun `should handle valid input`() { - // Arrange - whenever(dependency.method()).thenReturn("result") - - // Act - val result = service.doSomething() - - // Assert - assertThat(result).isEqualTo("expected") - verify(dependency).method() - } - } - ``` - -5. **Test Naming Conventions** - - Integration tests: End with `IT` (e.g., `BucketIT.kt`) - - Unit tests: End with `Test` (e.g., `BucketServiceTest.kt`) - - Use descriptive test names with backticks in Kotlin - - Format: `` `should when ` `` - -6. **S3Mock Test Setup** - - **For Integration Tests**: - ```kotlin - // Tests extend S3TestBase which provides: - // - s3Client: S3Client (AWS SDK v2) - // - s3ClientV1: AmazonS3 (AWS SDK v1) - // - httpClient: Configured HTTP client - // - serviceEndpoint: S3Mock endpoint URL - ``` - - **For JUnit 5 Extensions**: - ```kotlin - @ExtendWith(S3MockExtension::class) - class MyTest { - @Test - fun test(s3Client: S3Client) { - // S3Client injected automatically - } - } - ``` - - **For Testcontainers**: - ```kotlin - val s3MockContainer = S3MockContainer("latest") - .withInitialBuckets("test-bucket") - s3MockContainer.start() - ``` - -7. **Common Test Patterns** - - **Testing Object Operations**: - ```kotlin - @Test - fun `should store and retrieve object`() { - val bucketName = "test-bucket" - val key = "test-key" - val content = "test content" - - s3Client.createBucket(bucketName) - s3Client.putObject(bucketName, key, content.toByteArray()) - - val response = s3Client.getObject(bucketName, key) - val retrieved = response.readAllBytes().decodeToString() - - assertThat(retrieved).isEqualTo(content) - } - ``` - - **Testing Bucket Operations**: - ```kotlin - @Test - fun `should create and list buckets`() { - val bucketName = "my-bucket" - - s3Client.createBucket(bucketName) - val buckets = s3Client.listBuckets() - - assertThat(buckets).anyMatch { it.name() == bucketName } - } - ``` - - **Testing Multipart Uploads**: - ```kotlin - @Test - fun `should complete multipart upload`() { - val bucketName = "test-bucket" - val key = "large-file" - - s3Client.createBucket(bucketName) - - val uploadId = s3Client.createMultipartUpload( - CreateMultipartUploadRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ).uploadId() - - val parts = listOf( - uploadPart(bucketName, key, uploadId, 1, "part1".toByteArray()), - uploadPart(bucketName, key, uploadId, 2, "part2".toByteArray()) - ) - - s3Client.completeMultipartUpload( - CompleteMultipartUploadRequest.builder() - .bucket(bucketName) - .key(key) - .uploadId(uploadId) - .multipartUpload(CompletedMultipartUpload.builder().parts(parts).build()) - .build() - ) - - val obj = s3Client.getObject(bucketName, key) - assertThat(obj.readAllBytes().decodeToString()).isEqualTo("part1part2") - } - ``` - - **Testing Error Cases**: - ```kotlin - @Test - fun `should throw exception for non-existent bucket`() { - assertThrows { - s3Client.getObject("non-existent-bucket", "key") - } - } - ``` - -8. **Assertions and Matchers** - - Use AssertJ: `assertThat(value).isEqualTo(expected)` - - Use XMLUnit for XML: `assertThat(xml).and(expected).areIdentical()` - - Use Kotlin test assertions: `assertThrows { ... }` - - Verify mock interactions: `verify(mock).method()` - -9. **Test Data Management** - - Create test data in test setup - - Use meaningful test data that reflects real usage - - Clean up resources (usually automatic with test lifecycle) - - Use random bucket names if needed: `"bucket-${UUID.randomUUID()}"` - -10. **Testing Best Practices** - - **Isolation**: Each test should be independent - - **Clarity**: Test one thing per test method - - **Speed**: Keep tests fast, prefer unit tests - - **Coverage**: Test happy path and error cases - - **Readability**: Use descriptive names and clear assertions - - **Maintenance**: Update tests with code changes - -11. **Running Tests** - - ```bash - # Run all tests - ./mvnw clean verify - - # Run unit tests only - ./mvnw test - - # Run integration tests - ./mvnw verify -pl integration-tests - - # Run specific test - ./mvnw test -Dtest=BucketIT - - # Skip Docker build - ./mvnw verify -DskipDocker - ``` - -12. **Test Quality Checklist** - - [ ] Test name clearly describes what is tested - - [ ] Arrange-Act-Assert structure followed - - [ ] Test is independent and repeatable - - [ ] Both success and failure cases covered - - [ ] Assertions are specific and meaningful - - [ ] No hard-coded waits or sleeps - - [ ] Resources properly managed - - [ ] Test runs successfully locally - -## Project-Specific Guidelines - -### S3Mock Test Architecture - -- **Base Classes**: `S3TestBase` provides S3Client setup -- **Test Location**: - - Unit: `server/src/test/kotlin/com/adobe/testing/s3mock/` - - Integration: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/` -- **Languages**: Tests written in Kotlin -- **Frameworks**: JUnit 5, Mockito, AssertJ -- **Execution**: Parallel test execution enabled - -### Testing S3 Operations - -When testing S3 operations: -1. Verify response structure matches S3 API -2. Check HTTP status codes -3. Validate XML/JSON response bodies -4. Test with both SDK v1 and v2 if applicable -5. Verify headers (ETag, Content-Type, etc.) -6. Test presigned URLs if relevant - -### Testing File System Integration - -When testing storage: -1. Verify files created in correct location -2. Check metadata serialization -3. Validate bucket and object UUIDs -4. Test versioning if enabled -5. Verify cleanup on shutdown (if configured) - -### Mock Configuration in Tests - -```kotlin -// Configure test-specific settings -@TestPropertySource( - properties = [ - "com.adobe.testing.s3mock.store.retainFilesOnExit=false", - "com.adobe.testing.s3mock.store.validKmsKeys=test-key-id" - ] -) -``` - -## Common Issues and Solutions - -**Issue**: Test fails with "Connection refused" -- **Solution**: Ensure S3Mock is started before test runs, check port configuration - -**Issue**: Test fails intermittently -- **Solution**: Check for race conditions, ensure proper test isolation - -**Issue**: Test can't find bucket -- **Solution**: Verify bucket creation in test setup, check bucket name - -**Issue**: XML comparison fails -- **Solution**: Use XMLUnit's `areIdentical()` ignoring whitespace and comments - -## Resources - -- **Test Base**: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt` -- **Examples**: All `*IT.kt` files in `integration-tests/src/test/` -- **SDK Usage**: AWS SDK v2 documentation -- **Assertions**: AssertJ documentation -- **Mocking**: Mockito-Kotlin documentation + - Locate source code to test + - Review existing test patterns + - Identify test framework and conventions (see AGENTS.md) + +3. **Write Tests** + - Use Arrange-Act-Assert pattern + - Test both success and failure cases + - Use descriptive test names + - Keep tests focused and independent + +4. **Follow Project Standards** + - Match existing test style + - Use project test utilities + - Follow naming conventions (see AGENTS.md) + - Set up proper test fixtures + +5. **Quality Check** + - Tests pass locally + - Good coverage of edge cases + - Clear assertions + - No flaky tests + - Fast execution + +## Output + +Provide complete, runnable tests following project conventions. From 73a870f8d25a4cfb80c03c2ba8631671c6700701 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sat, 21 Feb 2026 17:14:40 +0100 Subject: [PATCH 04/36] feat: let Claude generate specific "AGENTS.md" These files contain context-specific information that agents can read if they read in / make changes in that context. --- AGENTS.md | 221 +++++++++++++++++++ integration-tests/AGENTS.md | 411 ++++++++++++++++++++++++++++++++++++ server/AGENTS.md | 270 +++++++++++++++++++++++ testsupport/AGENTS.md | 324 ++++++++++++++++++++++++++++ 4 files changed, 1226 insertions(+) create mode 100644 AGENTS.md create mode 100644 integration-tests/AGENTS.md create mode 100644 server/AGENTS.md create mode 100644 testsupport/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..8df3c1d9c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,221 @@ +# Agent Context for S3Mock + +This file provides context-specific information for AI agents working on the S3Mock project. + +## Project Overview + +**S3Mock** is a lightweight server implementing parts of the Amazon S3 API for local integration testing. + +- **Language**: Kotlin 2.3.0 (with Java interoperability, target JVM 17) +- **Framework**: Spring Boot 4.0.x +- **Build Tool**: Maven 3.9+ +- **Testing**: JUnit 5, Mockito, AssertJ, Testcontainers +- **Container**: Docker (Alpine Linux base) + +## Project Structure + +``` +/ +├── server/ # Core S3Mock implementation +│ ├── src/main/kotlin/ # Application code +│ └── src/test/ # Unit tests +├── integration-tests/ # Integration tests with AWS SDKs +├── testsupport/ # Test framework integrations +│ ├── junit4/ +│ ├── junit5/ +│ ├── testcontainers/ +│ └── testng/ +└── docker/ # Docker image build +``` + +## Architecture + +S3Mock follows a layered architecture: + +``` +Controller Layer (HTTP endpoints, REST API) + ↓ +Service Layer (Business logic, S3 operations) + ↓ +Store Layer (File system persistence, metadata) +``` + +### Key Packages + +- `controller/` - REST API endpoints, request/response handling +- `service/` - Business logic for S3 operations +- `store/` - File system storage and metadata management +- `dto/` - Data Transfer Objects with XML/JSON serialization + +## Code Style + +### Language Conventions + +- **Primary**: Kotlin with idiomatic patterns +- **Naming**: PascalCase for classes, camelCase for functions/properties +- **Data Classes**: Use for DTOs and value objects +- **Null Safety**: Leverage Kotlin's type system +- **Expression Bodies**: Use for simple functions + +### Spring Conventions + +- Constructor injection (preferred over field injection) +- `@Service` for service layer +- `@RestController` for controllers +- `@Component` for other managed beans + +### Example + +```kotlin +@RestController +@RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") +class ObjectController( + private val objectService: ObjectService +) { + @GetMapping("/{bucketName:.+}/{*key}") + fun getObject( + @PathVariable bucketName: String, + @PathVariable key: String + ): ResponseEntity { + val result = objectService.getObject(bucketName, key) + return ResponseEntity.ok() + .header("ETag", result.etag) + .body(result.data) + } +} +``` + +## XML Serialization + +S3Mock uses Jackson XML for AWS S3 API compatibility. Follow AWS XML structure exactly. + +```kotlin +@JacksonXmlRootElement(localName = "ListBucketResult") +data class ListBucketResult( + @JacksonXmlProperty(localName = "Name") + val name: String, + + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "Contents") + val contents: List = emptyList() +) +``` + +## File System Structure + +S3Mock persists data to the file system: + +``` +// + // + bucketMetadata.json + // + binaryData + objectMetadata.json + +# With versioning: + /-binaryData + /-objectMetadata.json + +# Multipart uploads: + /multiparts/ + // + multipartMetadata.json + /.part +``` + +## Configuration + +Environment variables (see `S3MockProperties.kt`): + +- `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` - Base directory for files +- `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` - Keep files after shutdown +- `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` - AWS region (default: us-east-1) +- `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` - Comma-separated bucket names +- `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` - Valid KMS key ARNs + +## Error Handling + +Use S3-specific exceptions: + +```kotlin +throw NoSuchBucketException("Bucket does not exist: $bucketName") +throw NoSuchKeyException("Key does not exist: $key") +throw BucketAlreadyExistsException("Bucket already exists: $bucketName") +``` + +HTTP status codes: 200 OK, 204 No Content, 404 Not Found, 409 Conflict, 500 Internal Error + +## Testing Philosophy + +- **Unit Tests**: Test components in isolation, mock dependencies +- **Integration Tests**: Test with real AWS SDK clients against S3Mock +- **Test Naming**: End with `Test` (unit) or `IT` (integration) +- **Test Independence**: Each test should be self-contained + +## Build Commands + +```bash +# Full build with tests and Docker +./mvnw clean install + +# Build without Docker +./mvnw clean install -DskipDocker + +# Run unit tests only +./mvnw test + +# Run integration tests +./mvnw verify -pl integration-tests + +# Format code +./mvnw ktlint:format + +# Check code style +./mvnw checkstyle:check +``` + +## AWS S3 API Reference + +When implementing S3 operations, refer to: +- [AWS S3 API Documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/) +- README.md for list of supported operations + +## Important Constraints + +- **Path-style access only** (no domain-style: `http://bucket.localhost`) +- **Presigned URLs**: Accepted but parameters ignored (no validation) +- **Self-signed SSL**: Included certificate, clients must trust it +- **Mock implementation**: Not for production use +- **KMS**: Validation only, no actual encryption + +## Common Patterns + +### ETag Generation +```kotlin +val etag = DigestUtils.md5Hex(data) +``` + +### Response Headers +```kotlin +ResponseEntity.ok() + .header("ETag", "\"$etag\"") + .header("Last-Modified", lastModified) + .header("Content-Type", contentType) + .body(data) +``` + +### Date Formatting +```kotlin +val formatted = Instant.now().toString() // ISO 8601 +``` + +## Documentation + +- **KDoc**: Use for public APIs +- **README**: Update supported operations table when adding features +- **CHANGELOG**: Document changes in releases + +## Contributing + +See `.github/CONTRIBUTING.md` for contribution guidelines. diff --git a/integration-tests/AGENTS.md b/integration-tests/AGENTS.md new file mode 100644 index 000000000..6d00bc238 --- /dev/null +++ b/integration-tests/AGENTS.md @@ -0,0 +1,411 @@ +# Agent Context for S3Mock Integration Tests + +This module contains integration tests that verify S3Mock against real AWS SDK clients. + +## Module Structure + +``` +integration-tests/ +├── src/test/kotlin/ +│ └── com/adobe/testing/s3mock/its/ +│ ├── S3TestBase.kt # Base class for all integration tests +│ ├── BucketIT.kt # Bucket operation tests +│ ├── ObjectIT.kt # Object operation tests +│ ├── MultipartUploadIT.kt # Multipart upload tests +│ └── *IT.kt # Other integration tests +└── pom.xml +``` + +## Test Base Class + +All integration tests extend `S3TestBase`: + +```kotlin +abstract class S3TestBase { + companion object { + lateinit var s3Client: S3Client // AWS SDK v2 + lateinit var s3ClientV1: AmazonS3 // AWS SDK v1 + lateinit var serviceEndpoint: String // S3Mock endpoint + lateinit var serviceEndpointHttp: String // HTTP endpoint + lateinit var serviceEndpointHttps: String // HTTPS endpoint + } +} +``` + +### Provided Clients + +- `s3Client` - AWS SDK v2 client (default) +- `s3ClientV1` - AWS SDK v1 client (for v1 compatibility tests) +- `httpClient` - Configured HTTP client for direct HTTP requests + +## Test Structure + +Follow the Arrange-Act-Assert pattern: + +```kotlin +class MyFeatureIT : S3TestBase() { + + @Test + fun `should perform operation successfully`() { + // Arrange - Setup test data + val bucketName = "test-bucket-${UUID.randomUUID()}" + val key = "test-key" + s3Client.createBucket { it.bucket(bucketName) } + + // Act - Perform the operation + s3Client.putObject( + PutObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build(), + RequestBody.fromString("test content") + ) + + // Assert - Verify results + val response = s3Client.getObject( + GetObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ) + + assertThat(response.response().contentLength()).isEqualTo(12L) + assertThat(response.readAllBytes().decodeToString()).isEqualTo("test content") + } +} +``` + +## Test Naming Conventions + +- Class names: End with `IT` (e.g., `BucketIT.kt`) +- Test methods: Use descriptive names with backticks + - `` `should perform action when condition` `` + - `` `should throw exception when invalid input` `` + +## Common Test Patterns + +### Testing Object Operations + +```kotlin +@Test +fun `should store and retrieve object`() { + val bucketName = givenBucket() + val key = "my-key" + val content = "test content" + + s3Client.putObject( + PutObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build(), + RequestBody.fromString(content) + ) + + val response = s3Client.getObject( + GetObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ) + + assertThat(response.readAllBytes().decodeToString()).isEqualTo(content) +} +``` + +### Testing Bucket Operations + +```kotlin +@Test +fun `should create and list buckets`() { + val bucketName = "test-bucket-${UUID.randomUUID()}" + + s3Client.createBucket { it.bucket(bucketName) } + + val buckets = s3Client.listBuckets() + + assertThat(buckets.buckets()) + .anyMatch { it.name() == bucketName } +} +``` + +### Testing Multipart Uploads + +```kotlin +@Test +fun `should complete multipart upload`() { + val bucketName = givenBucket() + val key = "large-file" + + // Initiate + val uploadId = s3Client.createMultipartUpload( + CreateMultipartUploadRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ).uploadId() + + // Upload parts + val part1 = s3Client.uploadPart( + UploadPartRequest.builder() + .bucket(bucketName) + .key(key) + .uploadId(uploadId) + .partNumber(1) + .build(), + RequestBody.fromString("part1") + ) + + val part2 = s3Client.uploadPart( + UploadPartRequest.builder() + .bucket(bucketName) + .key(key) + .uploadId(uploadId) + .partNumber(2) + .build(), + RequestBody.fromString("part2") + ) + + // Complete + s3Client.completeMultipartUpload( + CompleteMultipartUploadRequest.builder() + .bucket(bucketName) + .key(key) + .uploadId(uploadId) + .multipartUpload( + CompletedMultipartUpload.builder() + .parts( + CompletedPart.builder().partNumber(1).eTag(part1.eTag()).build(), + CompletedPart.builder().partNumber(2).eTag(part2.eTag()).build() + ) + .build() + ) + .build() + ) + + // Verify + val response = s3Client.getObject( + GetObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ) + + assertThat(response.readAllBytes().decodeToString()).isEqualTo("part1part2") +} +``` + +### Testing Error Conditions + +```kotlin +@Test +fun `should throw NoSuchBucket exception`() { + assertThrows { + s3Client.getObject( + GetObjectRequest.builder() + .bucket("non-existent-bucket") + .key("key") + .build() + ) + } +} +``` + +### Testing with Headers + +```kotlin +@Test +fun `should handle custom metadata`() { + val bucketName = givenBucket() + val key = "key-with-metadata" + + s3Client.putObject( + PutObjectRequest.builder() + .bucket(bucketName) + .key(key) + .metadata(mapOf("custom-key" to "custom-value")) + .build(), + RequestBody.fromString("content") + ) + + val response = s3Client.headObject( + HeadObjectRequest.builder() + .bucket(bucketName) + .key(key) + .build() + ) + + assertThat(response.metadata()).containsEntry("custom-key", "custom-value") +} +``` + +## Helper Methods + +### Bucket Creation + +```kotlin +private fun givenBucket(): String { + val bucketName = "test-bucket-${UUID.randomUUID()}" + s3Client.createBucket { it.bucket(bucketName) } + return bucketName +} +``` + +### Random Test Data + +```kotlin +private fun randomBytes(size: Int): ByteArray { + val bytes = ByteArray(size) + Random.nextBytes(bytes) + return bytes +} +``` + +## Assertions + +Use AssertJ for fluent assertions: + +```kotlin +// Basic assertions +assertThat(result).isNotNull() +assertThat(result).isEqualTo(expected) +assertThat(list).hasSize(3) +assertThat(list).contains(item) + +// Collection assertions +assertThat(buckets.buckets()) + .hasSize(1) + .anyMatch { it.name() == bucketName } + +// Exception assertions +assertThrows { + s3Client.deleteBucket { it.bucket("non-existent") } +} +``` + +## XML Response Verification + +For testing XML responses directly: + +```kotlin +import org.xmlunit.assertj3.XmlAssert + +@Test +fun `should return valid XML`() { + val xml = """ + + test-bucket + + """.trimIndent() + + XmlAssert.assertThat(xml) + .nodesByXPath("//Name") + .exist() +} +``` + +## Test Execution + +```bash +# Run all integration tests +./mvnw verify -pl integration-tests + +# Run specific test class +./mvnw verify -pl integration-tests -Dit.test=BucketIT + +# Run specific test method +./mvnw verify -pl integration-tests -Dit.test=BucketIT#shouldCreateBucket + +# Skip Docker build +./mvnw verify -pl integration-tests -DskipDocker +``` + +## S3Mock Test Configuration + +S3Mock is started automatically via Docker Maven Plugin. Configuration in `pom.xml`: + +```xml + + io.fabric8 + docker-maven-plugin + + + + adobe/s3mock:latest + + + it.s3mock.port_http:9090 + it.s3mock.port_https:9191 + + + + + + +``` + +## Test Best Practices + +1. **Isolation**: Each test is independent, no shared state +2. **Cleanup**: Tests clean up automatically (or rely on fresh S3Mock instance) +3. **Naming**: Use unique bucket names with UUID to avoid conflicts +4. **Speed**: Keep tests fast, avoid unnecessary delays +5. **Coverage**: Test both AWS SDK v1 and v2 when applicable +6. **Real SDK**: Use actual AWS SDK clients, not mocks + +## Testing Both SDK Versions + +When testing features with both SDK versions: + +```kotlin +@Test +fun `should work with SDK v1`() { + val bucketName = givenBucket() + + // Use s3ClientV1 (AWS SDK v1) + s3ClientV1.putObject(bucketName, "key", "content") + + val content = s3ClientV1.getObjectAsString(bucketName, "key") + assertThat(content).isEqualTo("content") +} + +@Test +fun `should work with SDK v2`() { + val bucketName = givenBucket() + + // Use s3Client (AWS SDK v2) + s3Client.putObject( + PutObjectRequest.builder().bucket(bucketName).key("key").build(), + RequestBody.fromString("content") + ) + + val response = s3Client.getObject( + GetObjectRequest.builder().bucket(bucketName).key("key").build() + ) + + assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") +} +``` + +## Verifying S3Mock Behavior + +Integration tests verify: +- Correct HTTP status codes +- Proper response headers (ETag, Content-Type, etc.) +- Valid XML/JSON response bodies +- Error responses match AWS S3 +- Edge cases and boundary conditions + +## Debugging Failed Tests + +1. Check S3Mock logs (Docker container logs) +2. Verify endpoint connectivity +3. Check request/response with HTTP client debugging +4. Validate test data and assumptions +5. Compare with AWS S3 API documentation + +## References + +- Main project context: `/AGENTS.md` +- Server implementation: `/server/AGENTS.md` +- AWS SDK v2: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/ +- AWS SDK v1: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/ diff --git a/server/AGENTS.md b/server/AGENTS.md new file mode 100644 index 000000000..32eb566c8 --- /dev/null +++ b/server/AGENTS.md @@ -0,0 +1,270 @@ +# Agent Context for S3Mock Server Module + +This module contains the core S3Mock server implementation. + +## Module Structure + +``` +server/ +├── src/main/kotlin/ +│ └── com/adobe/testing/s3mock/ +│ ├── S3MockApplication.kt # Spring Boot application entry point +│ ├── S3MockConfiguration.kt # Main configuration +│ ├── S3MockProperties.kt # Configuration properties +│ ├── controller/ # REST API endpoints +│ │ ├── BucketController.kt # Bucket operations +│ │ ├── ObjectController.kt # Object operations +│ │ └── *Converter.kt # Header/parameter converters +│ ├── dto/ # Data Transfer Objects +│ │ ├── *Result.kt # Response objects +│ │ └── *.kt # Request/response models +│ ├── service/ # Business logic +│ │ └── ObjectService.kt # Object operations service +│ └── store/ # Persistence layer +│ ├── BucketStore.kt # Bucket storage +│ ├── ObjectStore.kt # Object storage +│ ├── BucketMetadata.kt # Bucket metadata model +│ └── S3ObjectMetadata.kt # Object metadata model +└── src/test/ # Unit tests +``` + +## Implementation Workflow + +When implementing new S3 operations: + +1. **Define DTOs** (`dto/`) + - Create request/response data classes + - Add Jackson XML annotations + - Match AWS S3 XML structure exactly + +2. **Update Store Layer** (`store/`) + - Add persistence methods to `BucketStore` or `ObjectStore` + - Update metadata classes if needed + - Handle file system operations + +3. **Implement Service** (`service/`) + - Add business logic methods + - Validate inputs + - Use store layer for data access + - Handle errors with S3 exceptions + +4. **Add Controller Endpoint** (`controller/`) + - Map HTTP method and path + - Handle headers and parameters + - Call service layer + - Return proper response with headers + +## DTO Patterns + +All DTOs must serialize to XML matching AWS S3 API: + +```kotlin +@JacksonXmlRootElement(localName = "CreateBucketConfiguration") +data class CreateBucketConfiguration( + @JacksonXmlProperty(localName = "LocationConstraint") + val locationConstraint: LocationConstraint? = null, + + @JacksonXmlProperty(localName = "Bucket") + val bucket: BucketInfo? = null +) +``` + +### Common Annotations + +- `@JacksonXmlRootElement` - Root element name +- `@JacksonXmlProperty` - Element/attribute name +- `@JacksonXmlElementWrapper(useWrapping = false)` - Unwrap collections +- Custom serializers/deserializers for special cases (Region, Instant, etc.) + +## Store Layer Patterns + +The store layer manages file system persistence: + +```kotlin +@Component +class ObjectStore( + private val storeProperties: StoreProperties +) { + fun storeObject(bucket: BucketMetadata, key: String, data: InputStream): S3Object { + val uuid = UUID.randomUUID().toString() + val objectPath = getObjectPath(bucket, uuid) + Files.createDirectories(objectPath) + + // Store binary data + val dataPath = objectPath.resolve("binaryData") + Files.copy(data, dataPath, StandardCopyOption.REPLACE_EXISTING) + + // Store metadata + val metadata = S3ObjectMetadata(...) + storeMetadata(objectPath, metadata) + + return S3Object(...) + } +} +``` + +### Key Responsibilities + +- File system path resolution +- Binary data storage +- Metadata serialization/deserialization +- Cleanup and deletion + +## Service Layer Patterns + +Business logic and S3 operation implementation: + +```kotlin +@Service +class ObjectService( + private val bucketStore: BucketStore, + private val objectStore: ObjectStore +) { + fun getObject(bucketName: String, key: String): S3Object { + // Validate bucket exists + val bucket = bucketStore.getBucketMetadata(bucketName) + ?: throw NoSuchBucketException(bucketName) + + // Get object + val s3Object = objectStore.getObject(bucket, key) + ?: throw NoSuchKeyException(key) + + return s3Object + } +} +``` + +### Key Responsibilities + +- Input validation +- Error handling with S3 exceptions +- Coordinate store operations +- Implement S3 operation logic + +## Controller Layer Patterns + +REST endpoints implementing S3 API: + +```kotlin +@RestController +@RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") +class ObjectController( + private val objectService: ObjectService +) { + @GetMapping( + value = ["/{bucketName:.+}/{*key}"], + produces = [MediaType.APPLICATION_OCTET_STREAM_VALUE] + ) + fun getObject( + @PathVariable bucketName: String, + @PathVariable key: String, + @RequestHeader(required = false) headers: Map + ): ResponseEntity { + val s3Object = objectService.getObject(bucketName, key) + + return ResponseEntity + .ok() + .header("ETag", "\"${s3Object.etag}\"") + .header("Content-Type", s3Object.contentType) + .header("Last-Modified", s3Object.lastModified) + .body(s3Object.dataStream) + } +} +``` + +### Key Responsibilities + +- HTTP request mapping +- Header and parameter parsing +- Response construction with proper headers +- Stream large responses when needed + +## Testing + +### Unit Tests + +Test individual components with mocked dependencies: + +```kotlin +@ExtendWith(MockitoExtension::class) +internal class ObjectServiceTest { + @Mock + private lateinit var bucketStore: BucketStore + + @Mock + private lateinit var objectStore: ObjectStore + + @InjectMocks + private lateinit var objectService: ObjectService + + @Test + fun `should get object`() { + val bucket = BucketMetadata(...) + val s3Object = S3Object(...) + + whenever(bucketStore.getBucketMetadata("bucket")).thenReturn(bucket) + whenever(objectStore.getObject(bucket, "key")).thenReturn(s3Object) + + val result = objectService.getObject("bucket", "key") + + assertThat(result).isEqualTo(s3Object) + } +} +``` + +## Configuration + +Configuration bound from environment variables: + +```kotlin +@ConfigurationProperties(prefix = "com.adobe.testing.s3mock.store") +data class StoreProperties( + val root: Path = Files.createTempDirectory("s3mock"), + val retainFilesOnExit: Boolean = false, + val validKmsKeys: Set = emptySet(), + val initialBuckets: Set = emptySet(), + val region: Region = Region.US_EAST_1 +) +``` + +## Running the Server + +```bash +# As Spring Boot application +./mvnw spring-boot:run -pl server + +# Build executable JAR +./mvnw package -pl server -am +java -jar server/target/s3mock-*.jar + +# With Docker +./mvnw clean package -pl server -am -DskipTests +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock:latest +``` + +## Common Tasks + +### Adding a New S3 Operation + +1. Check AWS S3 API documentation +2. Create request/response DTOs in `dto/` +3. Add store methods in `store/` if needed +4. Implement service method in `service/` +5. Add controller endpoint in `controller/` +6. Write unit tests +7. Test with integration tests (see `integration-tests/AGENTS.md`) + +### Updating Existing Operations + +1. Identify affected layers (controller, service, store) +2. Update DTOs if request/response changed +3. Modify service logic +4. Update store operations if persistence changed +5. Update tests +6. Verify with integration tests + +## References + +- Main project context: `/AGENTS.md` +- Integration testing: `/integration-tests/AGENTS.md` +- AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/ diff --git a/testsupport/AGENTS.md b/testsupport/AGENTS.md new file mode 100644 index 000000000..4b0d26297 --- /dev/null +++ b/testsupport/AGENTS.md @@ -0,0 +1,324 @@ +# Agent Context for S3Mock Test Support + +This module provides test framework integrations for using S3Mock in various testing environments. + +## Module Structure + +``` +testsupport/ +├── common/ # Shared test support utilities +├── junit4/ # JUnit 4 Rule +├── junit5/ # JUnit 5 Extension +├── testcontainers/ # Testcontainers support +└── testng/ # TestNG Listener +``` + +## Common Module + +Shared utilities used by all test framework integrations. + +### Key Classes + +- `S3MockStarter` - Base class for starting S3Mock programmatically +- Configuration helpers +- Common test utilities + +## JUnit 5 Extension + +Provides `S3MockExtension` for JUnit 5 tests. + +### Usage Patterns + +**Declarative with Injection**: + +```kotlin +@ExtendWith(S3MockExtension::class) +class MyTest { + @Test + fun test(s3Client: S3Client) { + // S3Client injected automatically + s3Client.createBucket { it.bucket("test-bucket") } + } +} +``` + +**Programmatic with Builder**: + +```kotlin +class MyTest { + @RegisterExtension + val s3Mock = S3MockExtension.builder() + .withInitialBuckets("bucket1", "bucket2") + .withValidKmsKeys("arn:aws:kms:us-east-1:1234567890:key/test-key") + .build() + + @Test + fun test() { + val s3Client = s3Mock.createS3ClientV2() + // Use client + } +} +``` + +### Implementation Notes + +- Starts S3Mock before all tests +- Stops S3Mock after all tests +- Provides parameter injection for S3Client +- Supports both HTTP and HTTPS endpoints + +## JUnit 4 Rule + +Provides `S3MockRule` for JUnit 4 tests. + +### Usage + +```java +public class MyTest { + @Rule + public S3MockRule s3MockRule = S3MockRule.builder() + .withInitialBuckets("test-bucket") + .build(); + + @Test + public void test() { + AmazonS3 s3Client = s3MockRule.createS3Client(); + s3Client.createBucket("my-bucket"); + } +} +``` + +## Testcontainers Support + +Provides `S3MockContainer` - a Testcontainers implementation. + +### Usage + +**With JUnit 5**: + +```kotlin +@Testcontainers +class MyTest { + @Container + val s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + .withValidKmsKeys("test-key-id") + + @Test + fun test() { + val s3Client = S3Client.builder() + .endpointOverride(URI.create(s3Mock.httpEndpoint)) + .region(Region.US_EAST_1) + .credentialsProvider( + StaticCredentialsProvider.create( + AwsBasicCredentials.create("foo", "bar") + ) + ) + .build() + + s3Client.createBucket { it.bucket("my-bucket") } + } +} +``` + +**Manual Management**: + +```kotlin +class MyTest { + private lateinit var s3Mock: S3MockContainer + + @BeforeEach + fun setup() { + s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + s3Mock.start() + } + + @AfterEach + fun teardown() { + s3Mock.stop() + } + + @Test + fun test() { + // Use s3Mock.httpEndpoint or s3Mock.httpsEndpoint + } +} +``` + +### Container Configuration + +```kotlin +S3MockContainer("latest") + .withInitialBuckets("bucket1", "bucket2") + .withValidKmsKeys("key1", "key2") + .withRetainFilesOnExit(true) + .withRoot("/custom/path") + .withRegion("us-west-2") +``` + +### Key Methods + +- `httpEndpoint` - Get HTTP endpoint URL +- `httpsEndpoint` - Get HTTPS endpoint URL +- `withInitialBuckets()` - Pre-create buckets +- `withValidKmsKeys()` - Configure valid KMS keys +- `withRetainFilesOnExit()` - Keep files after container stops + +## TestNG Listener + +Provides `S3MockListener` for TestNG tests. + +### Usage + +Configure in `testng.xml`: + +```xml + + + + + + + + + + + + + +``` + +In test class: + +```kotlin +class MyTest { + @Test + fun test() { + val endpoint = System.getProperty("s3mock.httpEndpoint") + val s3Client = S3Client.builder() + .endpointOverride(URI.create(endpoint)) + // ... configure + .build() + + s3Client.createBucket { it.bucket("my-bucket") } + } +} +``` + +### Configuration Parameters + +- `s3mock.httpPort` - HTTP port (default: 9090) +- `s3mock.httpsPort` - HTTPS port (default: 9191) +- `s3mock.initialBuckets` - Comma-separated bucket names +- `s3mock.root` - Storage root directory + +## Writing Test Support Code + +When modifying or extending test support: + +### Design Principles + +1. **Framework-agnostic core**: Keep common logic in `common` module +2. **Simple API**: Minimize configuration complexity +3. **Sensible defaults**: Work out of the box for common cases +4. **Lifecycle management**: Proper startup/shutdown +5. **Resource cleanup**: Clean up after tests + +### Example: Adding New Configuration + +```kotlin +// In common module +abstract class S3MockStarter { + var customOption: String? = null + + protected fun buildEnvironment(): Map { + val env = mutableMapOf() + customOption?.let { env["CUSTOM_ENV_VAR"] = it } + return env + } +} + +// In JUnit 5 extension +class S3MockExtension { + class Builder { + fun withCustomOption(value: String): Builder { + this.customOption = value + return this + } + } +} +``` + +## Testing Test Support + +Each module includes tests: + +```kotlin +// Test the extension itself +@ExtendWith(S3MockExtension::class) +class S3MockExtensionTest { + @Test + fun `extension should start S3Mock`(s3Client: S3Client) { + // Verify S3Mock is accessible + assertThat(s3Client.listBuckets()).isNotNull() + } + + @Test + fun `should create initial buckets`() { + // Test initial bucket creation + } +} +``` + +## Maven Dependencies + +Users add test support as test dependencies: + +**JUnit 5**: +```xml + + com.adobe.testing + s3mock-junit5 + ${s3mock.version} + test + +``` + +**Testcontainers**: +```xml + + com.adobe.testing + s3mock-testcontainers + ${s3mock.version} + test + +``` + +## Common Issues + +### S3Mock Not Starting + +- Check port availability (9090, 9191) +- Verify Docker is running (for Testcontainers) +- Check logs for startup errors + +### Connection Refused + +- Ensure S3Mock started before tests run +- Verify endpoint URL is correct +- Check firewall/network settings + +### Tests Interfere with Each Other + +- Use unique bucket names (UUID) +- Ensure proper test isolation +- Consider separate S3Mock instances per test class + +## References + +- Main project context: `/AGENTS.md` +- Integration tests: `/integration-tests/AGENTS.md` +- JUnit 5: https://junit.org/junit5/ +- Testcontainers: https://www.testcontainers.org/ +- TestNG: https://testng.org/ From 3ff26ba40a5046607e969a730bf8f7c09cd4f2ca Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sat, 21 Feb 2026 17:22:11 +0100 Subject: [PATCH 05/36] feat: let Claude refactor README.md The README is now much shorter while retaining all essential information, making it faster to scan and easier to find specific usage patterns. --- README.md | 741 ++++++++++++------------------------------------------ 1 file changed, 158 insertions(+), 583 deletions(-) diff --git a/README.md b/README.md index 21ac9c395..5bea70fe8 100755 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ ![Maven Build](https://github.com/adobe/S3Mock/workflows/Maven%20Build/badge.svg) [![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/) [![Docker Pulls](https://img.shields.io/docker/pulls/adobe/s3mock)](https://hub.docker.com/r/adobe/s3mock) -[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#Kotlin) -[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#Java) +[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#kotlin) +[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#java) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7673/badge)](https://bestpractices.coreinfrastructure.org/projects/7673) [![OpenSSF Scorecard](https://img.shields.io/ossf-scorecard/github.com/adobe/S3Mock?label=openssf%20scorecard&style=flat)](https://api.securityscorecards.dev/projects/github.com/adobe/S3Mock) [![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/) @@ -11,80 +11,38 @@ * [S3Mock](#s3mock) * [Changelog](#changelog) - * [Supported S3 operations](#supported-s3-operations) + * [Supported S3 Operations](#supported-s3-operations) * [Usage](#usage) - * [Usage of AWS S3 SDKs](#usage-of-aws-s3-sdks) - * [Path-style vs. Domain-style access](#path-style-vs-domain-style-access) - * [Presigned URLs](#presigned-urls) - * [Self-signed SSL certificate](#self-signed-ssl-certificate) - * [Usage of AWS CLI](#usage-of-aws-cli) - * [Create bucket](#create-bucket) - * [Put object](#put-object) - * [Get object](#get-object) - * [Get object using HTTPS](#get-object-using-https) - * [Usage of plain HTTP / HTTPS with cURL](#usage-of-plain-http--https-with-curl) - * [Create bucket](#create-bucket-1) - * [Put object](#put-object-1) - * [Get object](#get-object-1) - * [Get object using HTTPS](#get-object-using-https-1) - * [S3Mock configuration options](#s3mock-configuration-options) - * [S3Mock Docker](#s3mock-docker) - * [Start using the command-line](#start-using-the-command-line) - * [Start using the Fabric8 Docker-Maven-Plugin](#start-using-the-fabric8-docker-maven-plugin) - * [Start using Testcontainers](#start-using-testcontainers) - * [Start using Docker compose](#start-using-docker-compose) - * [Simple example](#simple-example) - * [Expanded example](#expanded-example) - * [Start using a self-signed SSL certificate](#start-using-a-self-signed-ssl-certificate) - * [S3Mock Java](#s3mock-java) - * [Start using the JUnit5 Extension](#start-using-the-junit5-extension) - * [Start using the TestNG Listener](#start-using-the-testng-listener) - * [Start programmatically](#start-programmatically) + * [Docker (Recommended)](#docker-recommended) + * [Testcontainers](#testcontainers) + * [JUnit 5 Extension](#junit-5-extension) + * [TestNG Listener](#testng-listener) + * [AWS CLI](#aws-cli) + * [cURL](#curl) + * [Configuration](#configuration) + * [Important Limitations](#important-limitations) * [File System Structure](#file-system-structure) - * [Root-Folder](#root-folder) - * [Buckets](#buckets) - * [Objects](#objects) - * [Object versions](#object-versions) - * [Multipart Uploads](#multipart-uploads) * [Build & Run](#build--run) - * [Build](#build) - * [With Docker](#with-docker) - * [Without Docker](#without-docker) - * [Run](#run) - * [Spring Application](#spring-application) - * [Docker](#docker) - * [Docker Maven plugin](#docker-maven-plugin) - * [Run integration tests](#run-integration-tests) - * [Java](#java) - * [Kotlin](#kotlin) - * [Governance model](#governance-model) - * [Vulnerability reports](#vulnerability-reports) - * [Security](#security) * [Contributing](#contributing) - * [Licensing](#licensing) - * [Powered by](#powered-by) - * [Star History](#star-history) + * [License](#license) ## S3Mock -`S3Mock` is a lightweight server that implements parts of -the [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html). -It has been created to support local integration testing by reducing infrastructure dependencies. +S3Mock is a lightweight server implementing parts of the [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) for local integration testing. It eliminates the need for actual AWS infrastructure during development and testing. -The `S3Mock` server can be started as a standalone *Docker* container, using *Testcontainers*, *JUnit5* and *TestNG* support, or programmatically. +**Recommended usage**: Run S3Mock as a Docker container or with Testcontainers to avoid classpath conflicts. ## Changelog -See [GitHub releases](https://github.com/adobe/S3Mock/releases). -See also the [changelog](CHANGELOG.md) for detailed information about changes in releases and changes planned for future -releases. +- [GitHub Releases](https://github.com/adobe/S3Mock/releases) +- [Detailed Changelog](CHANGELOG.md) -## Supported S3 operations +## Supported S3 Operations -Of -these [operations of the Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html), -all marked :white_check_mark: are supported by S3Mock: +See the [complete operations table](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) in AWS documentation. + +Operations marked :white_check_mark: below are supported by S3Mock: | Operation | Support | Comment | |-----------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|------------------------| @@ -189,635 +147,252 @@ all marked :white_check_mark: are supported by S3Mock: ## Usage -### Usage of AWS S3 SDKs - -S3Mock can be used with any of the available AWS S3 SDKs. - -The [Integration Tests](integration-tests) and tests in [testsupport](testsupport) contain various examples of how to use the S3Mock with the AWS SDK for Java -v1 and v2 in Kotlin. - -`S3Client` or `S3Presigner` instances are created here: - -* Kotlin: [Integration Test base class](integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt) -* Java: [S3Mock unit test starter base class](testsupport/common/src/main/java/com/adobe/testing/s3mock/testsupport/common/S3MockStarter.java) - -#### Path-style vs. Domain-style access - -AWS S3 SDKs usually use domain-style access by default. Configuration is needed for path-style access. - -S3Mock currently only supports path-style access (e.g., `http://localhost:9090/bucket/someKey`). - -Domain-style access to buckets (e.g., `http://bucket.localhost:9090/someKey`) does not work because the domain -`localhost` is special and does not allow for subdomain access without -modifications to the operating system. - -#### Presigned URLs - -S3 SDKs can be used -to [create presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html), the S3 API -supports access through those URLs. - -S3Mock will accept presigned URLs, but it *ignores all parameters*. -For instance, S3Mock does not verify the HTTP verb that the presigned uri was created with, and it does not validate -whether the link is expired or not. - -S3 SDKs can be used to create presigned URLs pointing to S3Mock if they're configured for path-style access. See the -"Usage..." section above for links to examples on how to use the SDK with presigned URLs. - -#### Self-signed SSL certificate +### Docker (Recommended) -S3Mock supports connections via HTTP and HTTPS. It includes a self-signed SSL certificate which is rejected by most HTTP -clients by default. -To use HTTPS, the client must accept the self-signed certificate. - -On command line, this can be done by setting the `--no-verify-ssl` option in the AWS CLI or by using the `--insecure` -option in cURL, see below. - -Java and Kotlin SDKs can be configured to trust any SSL certificate, see links to `S3Client` creation above. - -### Usage of AWS CLI - -S3Mock can be used with the AWS CLI. Setting the `--endpoint-url` enables path-style access, `--no-verify-ssl` is needed -for HTTPS access. - -Examples: - -#### Create bucket +The Docker image is available on [Docker Hub](https://hub.docker.com/r/adobe/s3mock) and is the recommended way to run S3Mock. +**Basic usage:** ```shell -aws s3api create-bucket --bucket my-bucket --endpoint-url=http://localhost:9090 +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock ``` -#### Put object +Ports: `9090` (HTTP), `9191` (HTTPS) +**With configuration:** ```shell -aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url=http://localhost:9090 +docker run -p 9090:9090 -p 9191:9191 \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test-bucket \ + -e debug=true \ + adobe/s3mock ``` -#### Get object - -```shell -aws s3api get-object --bucket my-bucket --key my-file --endpoint-url=http://localhost:9090 my-file-output -``` - -#### Get object using HTTPS - -```shell -aws s3api get-object --bucket my-bucket --key my-file --no-verify-ssl --endpoint-url=https://localhost:9191 my-file-output -``` - -### Usage of plain HTTP / HTTPS with cURL - -As long as the requests work with the S3 API, they will work with S3Mock as well. Use `--insecure` to ignore SSL errors. - -Examples: - -#### Create bucket - -```shell -curl --request PUT "http://localhost:9090/my-test-bucket/" -``` - -#### Put object - -```shell -curl --request PUT --upload-file ./my-file http://localhost:9090/my-test-bucket/my-file -``` - -#### Get object - -```shell -curl --request GET http://localhost:9090/my-test-bucket/my-file -O -``` - -#### Get object using HTTPS - -```shell -curl --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -``` - -### S3Mock configuration options - -The mock can be configured with the following environment variables: - -- `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS`: list of KMS Key-Refs that are to be treated as *valid*. - - Default: none - - KMS keys must be configured as valid ARNs in the format of "`arn:aws:kms:region:acct-id:key/key-id`", for example " - `arn:aws:kms:us-east-1:1234567890:key/valid-test-key-id`" - - The list must be comma separated keys like `arn-1, arn-2` - - When requesting with KMS encryption, the key ID is passed to the SDK / CLI, in the example above this would be " - `valid-test-key-id`". - - *S3Mock does not implement KMS encryption*, if a key ID is passed in a request, S3Mock will just validate if a given - Key was configured during startup and reject the request if the given Key was not configured. -- `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS`: list of names for buckets that will be available initially. - - Default: none - - The list must be comma separated names like `bucketa, bucketb` -- `COM_ADOBE_TESTING_S3MOCK_STORE_REGION`: the region the S3Mock is supposed to mock. - - Default: `us-east-1` - - Value must be a valid AWS region identifier like `eu-west-1` -- `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT`: the base directory to place the temporary files exposed by the mock. If S3Mock is started in Docker, a volume - must be mounted as the `root` directory, see examples below. - - Default: Java temp directory -- `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT`: set to `true` to let S3Mock keep all files that were created during its lifetime. Default is - `false`, all files are removed if S3Mock shuts down. - - Default: false -- `COM_ADOBE_TESTING_S3MOCK_CONTROLLER_CONTEXT_PATH`: the base context path for all controllers. - Use if you start S3Mock within your application's unit tests and need to separate the context paths. - - Default: "" -- `debug`: set to `true` to - enable [Spring Boot's debug output](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.logging.console-output). -- `trace`: set to `true` to - enable [Spring Boot's trace output](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.logging.console-output). - -### S3Mock Docker - -The `S3Mock` Docker container is the recommended way to use `S3Mock`. -It is released to [Docker Hub](https://hub.docker.com/r/adobe/s3mock). -The container is lightweight, built on top of the official [Linux Alpine image](https://hub.docker.com/_/alpine). - -If needed, -configure [memory](https://docs.docker.com/engine/reference/commandline/run/#specify-hard-limits-on-memory-available-to-containers--m---memory) -and [cpu](https://docs.docker.com/engine/reference/commandline/run/#options) limits for the S3Mock Docker container. - -The JVM will automatically use half the available memory. - -#### Start using the command-line - -Starting on the command-line: - -```shell -docker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock -``` - -The port `9090` is for HTTP, port `9191` is for HTTPS. - -Example with configuration via environment variables: - -```shell -docker run -p 9090:9090 -p 9191:9191 -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test -e debug=true -t adobe/s3mock -``` - -#### Start using the Fabric8 Docker-Maven-Plugin - -Our [integration tests](integration-tests) are using the Amazon S3 Client to verify the server functionality against the -S3Mock. During the Maven build, the Docker image is started using the [docker-maven-plugin](https://dmp.fabric8.io/) and -the corresponding ports are passed to the JUnit test through the `maven-failsafe-plugin`. See [`BucketIT`](integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/BucketIT.kt) -as an example on how it's used in the code. - -This way, one can easily switch between calling the S3Mock or the real S3 endpoint, and this doesn't add any additional -Java dependencies to the project. - -#### Start using Testcontainers - -The [`S3MockContainer`](testsupport/testcontainers/src/main/java/com/adobe/testing/s3mock/testcontainers/S3MockContainer.java) -is a `Testcontainer` implementation that comes pre-configured exposing HTTP and HTTPS ports. Environment variables can -be set on startup. - -The example [`S3MockContainerJupiterTest`](testsupport/testcontainers/src/test/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainerJupiterTest.kt) -demonstrates the usage with JUnit 5. The example [`S3MockContainerManualTest`](testsupport/testcontainers/src/test/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainerManualTest.kt) -demonstrates the usage with plain Kotlin. Java will be similar. - -Testcontainers provide integrations for JUnit 4, JUnit 5 and Spock. -For more information, visit the [Testcontainers](https://www.testcontainers.org/) website. - -To use the [ -`S3MockContainer`](testsupport/testcontainers/src/main/java/com/adobe/testing/s3mock/testcontainers/S3MockContainer.java), -use the following Maven artifact in `test` scope: - -```xml - - com.adobe.testing - s3mock-testcontainers - ... - test - -``` - -#### Start using Docker compose - -##### Simple example - -Create a file `docker-compose.yml` - +**Docker Compose:** ```yaml services: s3mock: image: adobe/s3mock:latest environment: - - COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=bucket1 + - COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=bucket1,bucket2 ports: - 9090:9090 + - 9191:9191 ``` -Start with - -```shell -docker compose up -d -``` - -Stop with - -```shell -docker compose down -``` - -##### Expanded example - -Suppose we want to see what S3Mock is persisting and look at the logs it generates in detail. - -A local directory is needed, let's call it `locals3root`. This directory must be mounted as a volume into the Docker -container when it's started, and that mounted volume must then be configured as the `root` for S3Mock. Let's call the -mounted volume inside the container `containers3root`. S3Mock will delete all files when it shuts down, -`COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true` tells it to leave all files instead. - -Also, to see debug logs, `debug=true` must be configured for S3Mock. - -Create a file `docker-compose.yml` - +**With persistent storage:** ```yaml services: s3mock: image: adobe/s3mock:latest environment: - - debug=true - - COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true - COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3root - ports: - - 9090:9090 - - 9191:9191 - volumes: - - ./locals3root:/containers3root -``` - -Create a directory `locals3root`. - -Start with - -```shell -docker compose up -d -``` - -Create a bucket "my-test-bucket" with - -```shell -curl --request PUT "http://localhost:9090/my-test-bucket/" -``` - -Stop with - -```shell -docker compose down -``` - -Look into the directory `locals3root` where metadata and contents of the bucket are stored. - -```shell -$ mkdir s3mock-mounttest -$ cd s3mock-mounttest -$ mkdir locals3root -$ cat docker-compose.yml -services: - s3mock: - image: adobe/s3mock:latest - environment: - - debug=true - COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true - - COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3root ports: - 9090:9090 - - 9191:9191 volumes: - ./locals3root:/containers3root - -$ docker compose up -d -[+] Running 2/2 - ✔ Network s3mock-mounttest_default Created - ✔ Container s3mock-mounttest-s3mock-1 Started -$ curl --request PUT "http://localhost:9090/my-test-bucket/" -$ docker compose down -[+] Running 2/0 - ✔ Container s3mock-mounttest-s3mock-1 Removed - ✔ Network s3mock-mounttest_default Removed - -$ ls locals3root -my-test-bucket -$ ls locals3root/my-test-bucket -bucketMetadata.json -``` - -#### Start using a self-signed SSL certificate - -S3Mock includes a self-signed SSL certificate: - -```shell -$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -[...] -* Server certificate: -* subject: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock -* start date: Jul 25 12:28:53 2022 GMT -* expire date: Nov 25 12:28:53 3021 GMT -* issuer: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock -* SSL certificate verify result: self signed certificate (18), continuing anyway. -[...] -``` - -To use a custom self-signed SSL certificate, derive your own Docker container from the S3Mock container: - -```dockerfile -FROM adobe/s3mock:4.2.0 - -ENV server.ssl.key-store=/opt/customcert.jks -ENV server.ssl.key-store-password=password -ENV server.ssl.key-alias=selfsigned - -RUN keytool -genkey -keyalg RSA -alias selfsigned \ - -validity 360 \ - -keystore /opt/customcert.jks \ - -dname "cn=Test, ou=Test, o=Docker, l=NY, st=NY, c=US" \ - -storepass password -keysize 2048 \ - -ext "san=dns:localhost" -``` - -```shell -$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O -[...] -* Server certificate: -* subject: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test -* start date: May 9 14:33:40 2025 GMT -* expire date: May 4 14:33:40 2026 GMT -* issuer: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test -* SSL certificate verify result: self signed certificate (18), continuing anyway. -[...] ``` -### S3Mock Java - -`S3Mock` Java libraries are released and published to the Sonatype Maven Repository and subsequently published to -the official [Maven mirrors](https://search.maven.org/search?q=g:com.adobe.testing%20a:s3mock). - -| :warning: WARNING | -|:--------------------------------------------------------------------------------------------------| -| Using the Java libraries is **discouraged**, see explanation below | -| Using the Docker image is **encouraged** to insulate both S3Mock and your application at runtime. | - -`S3Mock` is built using Spring Boot, if projects use `S3Mock` by adding the dependency to their project and starting -the `S3Mock` during a JUnit test, classpaths of the tested application and of the `S3Mock` are merged, leading -to unpredictable and undesired effects such as class conflicts or dependency version conflicts. -This is especially problematic if the tested application itself is a Spring (Boot) application, as both applications -will load configurations based on the availability of certain classes in the classpath, leading to unpredictable runtime -behavior. - -_This is the opposite of what software engineers are trying to achieve when thoroughly testing code in continuous -integration..._ - -`S3Mock` dependencies are updated regularly, any update could break any number of projects. -**See also [issues labeled "dependency-problem"](https://github.com/adobe/S3Mock/issues?q=is%3Aissue+label%3Adependency-problem).** +### Testcontainers -**See also [the Java section below](#Java)** - -#### Start using the JUnit5 Extension - -The `S3MockExtension` can currently be used in two ways: - -1. Declaratively using `@ExtendWith(S3MockExtension.class)` and by injecting a properly configured instance of - `AmazonS3` client and/or the started `S3MockApplication` to the tests. - See examples: [`S3MockExtensionDeclarativeTest`](testsupport/junit5/src/test/java/com/adobe/testing/s3mock/junit5/sdk1/S3MockExtensionDeclarativeTest.java) (for SDKv1) - or [`S3MockExtensionDeclarativeTest`](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionDeclarativeTest.kt) (for SDKv2) - -2. Programmatically using `@RegisterExtension` and by creating and configuring the `S3MockExtension` using a _builder_. - See examples: [`S3MockExtensionProgrammaticTest`](testsupport/junit5/src/test/java/com/adobe/testing/s3mock/junit5/sdk1/S3MockExtensionProgrammaticTest.java) (for SDKv1) - or [`S3MockExtensionProgrammaticTest`](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionProgrammaticTest.kt) (for SDKv2) - -To use the JUnit5 Extension, use the following Maven artifact in `test` scope: +The [`S3MockContainer`](testsupport/testcontainers/src/main/kotlin/com/adobe/testing/s3mock/testcontainers/S3MockContainer.kt) provides a pre-configured Testcontainers implementation. +**Maven dependency:** ```xml com.adobe.testing - s3mock-junit5 + s3mock-testcontainers ... test ``` -#### Start using the TestNG Listener +**Usage example:** +```kotlin +@Testcontainers +class MyTest { + @Container + val s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + + @Test + fun test() { + val s3Client = S3Client.builder() + .endpointOverride(URI.create(s3Mock.httpEndpoint)) + .region(Region.US_EAST_1) + .credentialsProvider(StaticCredentialsProvider.create( + AwsBasicCredentials.create("foo", "bar") + )) + .build() -The example [`S3MockListenerXMLConfigurationTest`](testsupport/testng/src/test/kotlin/com/adobe/testing/s3mock/testng/S3MockListenerXmlConfigurationTest.kt) -demonstrates the usage of the `S3MockListener`, which can be configured as shown in [`testng.xml`](testsupport/testng/src/test/resources/testng.xml). -The listener bootstraps the S3Mock application before TestNG execution starts and shuts down the application just before the execution terminates. -Please refer to [`IExecutionListener`](https://github.com/testng-team/testng/blob/master/testng-core-api/src/main/java/org/testng/IExecutionListener.java) -in the TestNG API. + s3Client.createBucket { it.bucket("my-bucket") } + } +} +``` -To use the TestNG Listener, use the following Maven artifact in `test` scope: +### JUnit 5 Extension +**Maven dependency:** ```xml com.adobe.testing - s3mock-testng + s3mock-junit5 ... test ``` -#### Start programmatically +**Usage:** +```kotlin +@ExtendWith(S3MockExtension::class) +class MyTest { + @Test + fun test(s3Client: S3Client) { + s3Client.createBucket { it.bucket("test-bucket") } + } +} +``` + +See examples: [Declarative](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionDeclarativeTest.kt) | [Programmatic](testsupport/junit5/src/test/kotlin/com/adobe/testing/s3mock/junit5/sdk2/S3MockExtensionProgrammaticTest.kt) -Include the following dependency and use one of the `start` methods in `com.adobe.testing.s3mock.S3MockApplication`: +### TestNG Listener +**Maven dependency:** ```xml com.adobe.testing - s3mock + s3mock-testng ... + test ``` -## File System Structure - -S3Mock stores Buckets, Objects, Parts and other data on the disk. -This lets users inspect the stored data while the S3Mock is running. -If the environment variable `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` is set to `true`, this data will not be deleted when S3Mock is shut down. - -| :exclamation: FYI | -|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| While it _may_ be possible to start S3Mock on a root folder from a previous run and have all data available through the S3 API, the structure and contents of the files are not considered Public API, and are subject to change in later releases. | -| Also, there are no automated test cases for this behaviour. | - -### Root-Folder - -S3Mock stores buckets and objects in a root-folder. - -This folder is expected to be empty when S3Mock starts. See also FYI above. - -``` -// -``` - -### Buckets - -Buckets are stored as a folder with their name as created through the S3 API directly below the root: - -``` -/// -``` - -[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) is stored in a file in the -bucket directory, serialized as JSON. -[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) contains the "key" -> "uuid" -dictionary for all objects uploaded to this bucket among other data. - -``` -///bucketMetadata.json -``` +Configure in `testng.xml` - see [example configuration](testsupport/testng/src/test/resources/testng.xml). -### Objects +### AWS CLI -Objects are stored in folders below the bucket they were created in. -A folder is created that uses the Object's UUID assigned in -the [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) as a name. +Use with `--endpoint-url` and `--no-verify-ssl` (for HTTPS): -``` -//// -``` +```shell +# Create bucket +aws s3api create-bucket --bucket my-bucket --endpoint-url http://localhost:9090 -Object data is stored below that UUID folder. +# Upload object +aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url http://localhost:9090 -Binary data is always stored in a file `binaryData` +# Download object +aws s3api get-object --bucket my-bucket --key my-file --endpoint-url http://localhost:9090 output-file -``` -////binaryData +# HTTPS +aws s3api get-object --bucket my-bucket --key my-file --no-verify-ssl --endpoint-url https://localhost:9191 output-file ``` -[Object metadata](server/src/main/java/com/adobe/testing/s3mock/store/S3ObjectMetadata.java) is serialized as JSON and -stored as `objectMetadata.json` +### cURL -``` -////objectMetadata.json -``` +```shell +# Create bucket +curl -X PUT http://localhost:9090/my-bucket/ -#### Object versions +# Upload object +curl -X PUT --upload-file ./my-file http://localhost:9090/my-bucket/my-file -If versioning is enabled for a bucket, multiple versions of an object can be stored in S3Mock. The files will be stored -alongside each other, prefixed by the versionId. +# Download object +curl http://localhost:9090/my-bucket/my-file -O -``` -////-binaryData -////-objectMetadata.json +# HTTPS (with self-signed certificate) +curl --insecure https://localhost:9191/my-bucket/my-file -O ``` -### Multipart Uploads +## Configuration -Multipart Uploads are created in a bucket using object keys and an uploadId. -The object is assigned a UUID within the bucket (stored -in [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java)). -The [Multipart upload metadata](server/src/main/java/com/adobe/testing/s3mock/store/MultipartUploadInfo.java) is -currently not stored on disk. +Configure S3Mock using environment variables: -The multiparts folder is created below the bucket folder and named with the `uploadId`: +| Variable | Default | Description | +|-------------------------------------------------------|---------------------|---------------------------------------------------------------| +| `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` | Java temp directory | Base directory for file storage | +| `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` | `us-east-1` | AWS region to mock | +| `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` | none | Comma-separated list of buckets to create on startup | +| `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` | `false` | Keep files after shutdown | +| `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` | none | Comma-separated KMS key ARNs (validation only, no encryption) | +| `COM_ADOBE_TESTING_S3MOCK_CONTROLLER_CONTEXT_PATH` | `""` | Base context path for all endpoints | +| `debug` | `false` | Enable Spring Boot debug logging | +| `trace` | `false` | Enable Spring Boot trace logging | -``` -///multiparts// -``` +## Important Limitations -The multiparts metadata file is created below the folder named with the `uploadId`: +- **Path-style access only**: S3Mock supports `http://localhost:9090/bucket/key`, not `http://bucket.localhost:9090/key` +- **Presigned URLs**: Accepted but not validated (expiration, signature, HTTP verb not checked) +- **Self-signed SSL**: Included certificate requires clients to trust it or ignore SSL errors +- **KMS**: Key validation only - no actual encryption performed +- **Not for production**: S3Mock is a testing tool and lacks security features required for production use -``` -///multiparts//multipartMetadata.json -``` +## File System Structure -Each part is stored in the parts folder with the `partNo` as name and `.part` as a suffix. +S3Mock stores data on disk with the following structure: ``` -///multiparts//.part +/ + / + bucketMetadata.json # Bucket metadata + / + binaryData # Object content + objectMetadata.json # Object metadata + -binaryData # Versioned object (if versioning enabled) + -objectMetadata.json + multiparts/ + / + multipartMetadata.json + .part ``` -Once the MultipartUpload is completed, the `uploadId` folder is deleted. +**Note:** The file system structure is an implementation detail and may change between releases. While files can be inspected during runtime, reusing persisted data across restarts is not officially supported. ## Build & Run -### Build - -#### With Docker - -To build this project, you need Docker, JDK 17 or higher, and Maven: +**Requirements**: Java 17+, Maven 3.9+, Docker (for Docker build and integration tests) +**Build:** ```shell +# Full build with Docker ./mvnw clean install -``` -#### Without Docker - -If you want to skip the Docker build, pass the optional parameter "skipDocker": - -```shell +# Skip Docker build ./mvnw clean install -DskipDocker ``` -### Run - -You can run the S3Mock from the sources by either of the following methods: - -#### Spring Application - -Run or Debug the class `com.adobe.testing.s3mock.S3MockApplication` in the IDE. - -#### Docker - -Use Docker: - +**Run from source:** ```shell +# As Spring Boot application +./mvnw spring-boot:run -pl server + +# As Docker container ./mvnw clean package -pl server -am -DskipTests -docker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock:latest +docker run -p 9090:9090 -p 9191:9191 adobe/s3mock:latest ``` -#### Docker Maven plugin - -Use the Docker Maven plugin: - +**Run integration tests:** ```shell -./mvnw clean package docker:start -pl server -am -DskipTests -Ddocker.follow -Dit.s3mock.port_http=9090 -Dit.s3mock.port_https=9191 +./mvnw verify -pl integration-tests ``` -(stop with `ctrl-c`) - -### Run integration tests - -Once the application is started using default ports, you can execute the `*IT` tests from your IDE. - -### Java - -This repo is built with Java 17, output is _currently_ bytecode compatible with Java 17. -### Kotlin +**Technology:** +- Built with Kotlin 2.3.0 and Spring Boot 4.0 +- Tests written in Kotlin +- Target JVM: 17 -The [Integration Tests](integration-tests) and unit tests are built in Kotlin. - -## Governance model - -The project owner and leads make all final decisions. See the `developers` section in the [pom.xml](pom.xml) for a list -of leads. - -## Vulnerability reports - -S3Mock uses GitHub actions to produce an SBOM and to check dependencies for vulnerabilities. All vulnerabilities are -evaluated and fixed if possible. -Vulnerabilities may also be reported through the GitHub issue tracker. +## Contributing -## Security +Contributions are welcome! See [Contributing Guide](.github/CONTRIBUTING.md). -S3Mock is not intended to be used in production environments. It is a mock server meant to be used in -development and testing environments only. It does not implement all security features of AWS S3 and should not be used -as a replacement for AWS S3 in production. -It is implemented using [Spring Boot](https://github.com/spring-projects/spring-boot), which is a Java framework that is -designed to be secure by default. +**Governance**: Project leads make final decisions - see `developers` in [pom.xml](pom.xml). -## Contributing +**Security**: Report vulnerabilities via GitHub issues. S3Mock uses GitHub Actions for SBOM and vulnerability scanning. -Contributions are welcome! Read the [Contributing Guide](./.github/CONTRIBUTING.md) for more information. +## License -## Licensing +Licensed under the Apache License 2.0 - see [LICENSE](LICENSE). -This project is licensed under the Apache V2 License. See [LICENSE](LICENSE) for more information. +--- -## Powered by -[![IntelliJ IDEA logo.](https://resources.jetbrains.com/storage/products/company/brand/logos/IntelliJ_IDEA.svg)](https://jb.gg/OpenSourceSupport) +[![IntelliJ IDEA](https://resources.jetbrains.com/storage/products/company/brand/logos/IntelliJ_IDEA.svg)](https://jb.gg/OpenSourceSupport) -## Star History -[![Star History Chart](https://api.star-history.com/svg?repos=adobe/S3Mock&type=Date)](https://www.star-history.com/#adobe/S3Mock&Date) +[![Star History](https://api.star-history.com/svg?repos=adobe/S3Mock&type=Date)](https://www.star-history.com/#adobe/S3Mock&Date) From c21c570484082c9b020738e7064afb239d51e10d Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sun, 22 Feb 2026 14:06:26 +0100 Subject: [PATCH 06/36] feat: let Claude condense AGENTS.md Asked Claude to make AGENTS.md files more concise to make sure that they help agents while keeping token usage low. Bloated AGENTS.md will burn way more tokens and may actually confuse agents more than helping them. --- AGENTS.md | 217 +++++-------------- integration-tests/AGENTS.md | 410 +++++------------------------------- server/AGENTS.md | 242 ++++----------------- testsupport/AGENTS.md | 315 ++++----------------------- 4 files changed, 183 insertions(+), 1001 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8df3c1d9c..c19bd092f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,221 +1,110 @@ # Agent Context for S3Mock -This file provides context-specific information for AI agents working on the S3Mock project. +Lightweight S3 API mock server for local integration testing. -## Project Overview - -**S3Mock** is a lightweight server implementing parts of the Amazon S3 API for local integration testing. - -- **Language**: Kotlin 2.3.0 (with Java interoperability, target JVM 17) -- **Framework**: Spring Boot 4.0.x -- **Build Tool**: Maven 3.9+ +## Tech Stack +- **Kotlin 2.3** (JVM 17), Spring Boot 4.0.x, Maven 3.9+ - **Testing**: JUnit 5, Mockito, AssertJ, Testcontainers -- **Container**: Docker (Alpine Linux base) - -## Project Structure +- **Container**: Docker/Alpine +## Structure ``` -/ -├── server/ # Core S3Mock implementation -│ ├── src/main/kotlin/ # Application code -│ └── src/test/ # Unit tests -├── integration-tests/ # Integration tests with AWS SDKs -├── testsupport/ # Test framework integrations -│ ├── junit4/ -│ ├── junit5/ -│ ├── testcontainers/ -│ └── testng/ -└── docker/ # Docker image build +server/ # Core implementation (Controller→Service→Store) +integration-tests/ # AWS SDK integration tests +testsupport/ # JUnit 4/5, Testcontainers, TestNG integrations ``` ## Architecture -S3Mock follows a layered architecture: +**Layered**: Controller (REST) → Service (logic) → Store (filesystem) -``` -Controller Layer (HTTP endpoints, REST API) - ↓ -Service Layer (Business logic, S3 operations) - ↓ -Store Layer (File system persistence, metadata) -``` - -### Key Packages - -- `controller/` - REST API endpoints, request/response handling -- `service/` - Business logic for S3 operations -- `store/` - File system storage and metadata management -- `dto/` - Data Transfer Objects with XML/JSON serialization +**Key packages**: `controller/`, `service/`, `store/`, `dto/` ## Code Style -### Language Conventions - -- **Primary**: Kotlin with idiomatic patterns -- **Naming**: PascalCase for classes, camelCase for functions/properties -- **Data Classes**: Use for DTOs and value objects -- **Null Safety**: Leverage Kotlin's type system -- **Expression Bodies**: Use for simple functions - -### Spring Conventions +**Kotlin idioms**: Data classes for DTOs, null safety, expression bodies, constructor injection -- Constructor injection (preferred over field injection) -- `@Service` for service layer -- `@RestController` for controllers -- `@Component` for other managed beans - -### Example +**Spring**: `@RestController`, `@Service`, `@Component`, constructor injection over field injection +**Example**: ```kotlin @RestController -@RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") -class ObjectController( - private val objectService: ObjectService -) { +class ObjectController(private val objectService: ObjectService) { @GetMapping("/{bucketName:.+}/{*key}") - fun getObject( - @PathVariable bucketName: String, - @PathVariable key: String - ): ResponseEntity { - val result = objectService.getObject(bucketName, key) - return ResponseEntity.ok() - .header("ETag", result.etag) - .body(result.data) - } + fun getObject(@PathVariable bucketName: String, @PathVariable key: String) = + objectService.getObject(bucketName, key).let { + ResponseEntity.ok().header("ETag", it.etag).body(it.data) + } } ``` ## XML Serialization -S3Mock uses Jackson XML for AWS S3 API compatibility. Follow AWS XML structure exactly. - -```kotlin -@JacksonXmlRootElement(localName = "ListBucketResult") -data class ListBucketResult( - @JacksonXmlProperty(localName = "Name") - val name: String, - - @JacksonXmlElementWrapper(useWrapping = false) - @JacksonXmlProperty(localName = "Contents") - val contents: List = emptyList() -) -``` - -## File System Structure +Jackson XML with AWS-compatible structure. Key annotations: +- `@JacksonXmlRootElement(localName = "...")` +- `@JacksonXmlProperty(localName = "...")` +- `@JacksonXmlElementWrapper(useWrapping = false)` for collections -S3Mock persists data to the file system: +## Storage +Filesystem layout: ``` -// - // - bucketMetadata.json - // - binaryData - objectMetadata.json - -# With versioning: - /-binaryData - /-objectMetadata.json - -# Multipart uploads: - /multiparts/ - // - multipartMetadata.json - /.part +//bucketMetadata.json +///binaryData + objectMetadata.json +///-binaryData # versioning +//multiparts//.part ``` ## Configuration -Environment variables (see `S3MockProperties.kt`): - -- `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` - Base directory for files -- `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` - Keep files after shutdown -- `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` - AWS region (default: us-east-1) -- `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` - Comma-separated bucket names -- `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` - Valid KMS key ARNs +Environment variables (prefix: `COM_ADOBE_TESTING_S3MOCK_STORE_`): +- `ROOT` - storage directory +- `RETAIN_FILES_ON_EXIT` - keep files after shutdown +- `REGION` - AWS region (default: us-east-1) +- `INITIAL_BUCKETS` - comma-separated bucket names +- `VALID_KMS_KEYS` - valid KMS ARNs ## Error Handling -Use S3-specific exceptions: +S3 exceptions: `NoSuchBucketException`, `NoSuchKeyException`, `BucketAlreadyExistsException` -```kotlin -throw NoSuchBucketException("Bucket does not exist: $bucketName") -throw NoSuchKeyException("Key does not exist: $key") -throw BucketAlreadyExistsException("Bucket already exists: $bucketName") -``` +HTTP codes: 200, 204, 404, 409, 500 -HTTP status codes: 200 OK, 204 No Content, 404 Not Found, 409 Conflict, 500 Internal Error +## Testing -## Testing Philosophy +- Unit tests: Mock dependencies, test in isolation, suffix `Test` +- Integration tests: Real AWS SDKs v1/v2, suffix `IT` +- Test independence: Each test self-contained -- **Unit Tests**: Test components in isolation, mock dependencies -- **Integration Tests**: Test with real AWS SDK clients against S3Mock -- **Test Naming**: End with `Test` (unit) or `IT` (integration) -- **Test Independence**: Each test should be self-contained - -## Build Commands +## Build ```bash -# Full build with tests and Docker -./mvnw clean install - -# Build without Docker -./mvnw clean install -DskipDocker - -# Run unit tests only -./mvnw test - -# Run integration tests +./mvnw clean install # Full build +./mvnw clean install -DskipDocker # Skip Docker ./mvnw verify -pl integration-tests - -# Format code ./mvnw ktlint:format - -# Check code style -./mvnw checkstyle:check ``` -## AWS S3 API Reference - -When implementing S3 operations, refer to: -- [AWS S3 API Documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/) -- README.md for list of supported operations - -## Important Constraints +## Constraints -- **Path-style access only** (no domain-style: `http://bucket.localhost`) -- **Presigned URLs**: Accepted but parameters ignored (no validation) -- **Self-signed SSL**: Included certificate, clients must trust it -- **Mock implementation**: Not for production use -- **KMS**: Validation only, no actual encryption +- Path-style URLs only (not `bucket.localhost`) +- Presigned URLs accepted but not validated +- Self-signed SSL certificate +- KMS validation only, no encryption +- Not for production ## Common Patterns -### ETag Generation ```kotlin +// ETag val etag = DigestUtils.md5Hex(data) -``` -### Response Headers -```kotlin +// Response ResponseEntity.ok() .header("ETag", "\"$etag\"") .header("Last-Modified", lastModified) - .header("Content-Type", contentType) .body(data) -``` -### Date Formatting -```kotlin -val formatted = Instant.now().toString() // ISO 8601 +// Dates +Instant.now().toString() // ISO 8601 ``` - -## Documentation - -- **KDoc**: Use for public APIs -- **README**: Update supported operations table when adding features -- **CHANGELOG**: Document changes in releases - -## Contributing - -See `.github/CONTRIBUTING.md` for contribution guidelines. diff --git a/integration-tests/AGENTS.md b/integration-tests/AGENTS.md index 6d00bc238..c78bcc5d7 100644 --- a/integration-tests/AGENTS.md +++ b/integration-tests/AGENTS.md @@ -1,411 +1,107 @@ # Agent Context for S3Mock Integration Tests -This module contains integration tests that verify S3Mock against real AWS SDK clients. +Integration tests verifying S3Mock with real AWS SDK clients (v1 and v2). -## Module Structure +## Structure ``` -integration-tests/ -├── src/test/kotlin/ -│ └── com/adobe/testing/s3mock/its/ -│ ├── S3TestBase.kt # Base class for all integration tests -│ ├── BucketIT.kt # Bucket operation tests -│ ├── ObjectIT.kt # Object operation tests -│ ├── MultipartUploadIT.kt # Multipart upload tests -│ └── *IT.kt # Other integration tests -└── pom.xml +integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/ +├── S3TestBase.kt # Base class with s3Client (v2), s3ClientV1 +├── BucketIT.kt # Bucket operations +├── ObjectIT.kt # Object operations +├── MultipartUploadIT.kt # Multipart uploads +└── *IT.kt # Other tests ``` -## Test Base Class +## Base Class -All integration tests extend `S3TestBase`: +Extend `S3TestBase` for access to: +- `s3Client` - AWS SDK v2 (default) +- `s3ClientV1` - AWS SDK v1 +- `serviceEndpoint`, `serviceEndpointHttp`, `serviceEndpointHttps` -```kotlin -abstract class S3TestBase { - companion object { - lateinit var s3Client: S3Client // AWS SDK v2 - lateinit var s3ClientV1: AmazonS3 // AWS SDK v1 - lateinit var serviceEndpoint: String // S3Mock endpoint - lateinit var serviceEndpointHttp: String // HTTP endpoint - lateinit var serviceEndpointHttps: String // HTTPS endpoint - } -} -``` - -### Provided Clients - -- `s3Client` - AWS SDK v2 client (default) -- `s3ClientV1` - AWS SDK v1 client (for v1 compatibility tests) -- `httpClient` - Configured HTTP client for direct HTTP requests +## Test Pattern -## Test Structure - -Follow the Arrange-Act-Assert pattern: +Arrange-Act-Assert with unique bucket names (`UUID.randomUUID()`): ```kotlin class MyFeatureIT : S3TestBase() { - @Test - fun `should perform operation successfully`() { - // Arrange - Setup test data - val bucketName = "test-bucket-${UUID.randomUUID()}" - val key = "test-key" - s3Client.createBucket { it.bucket(bucketName) } + fun `should perform operation`() { + // Arrange + val bucketName = givenBucket() - // Act - Perform the operation + // Act s3Client.putObject( - PutObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build(), - RequestBody.fromString("test content") + PutObjectRequest.builder().bucket(bucketName).key("key").build(), + RequestBody.fromString("content") ) - // Assert - Verify results + // Assert val response = s3Client.getObject( - GetObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - assertThat(response.response().contentLength()).isEqualTo(12L) - assertThat(response.readAllBytes().decodeToString()).isEqualTo("test content") - } -} -``` - -## Test Naming Conventions - -- Class names: End with `IT` (e.g., `BucketIT.kt`) -- Test methods: Use descriptive names with backticks - - `` `should perform action when condition` `` - - `` `should throw exception when invalid input` `` - -## Common Test Patterns - -### Testing Object Operations - -```kotlin -@Test -fun `should store and retrieve object`() { - val bucketName = givenBucket() - val key = "my-key" - val content = "test content" - - s3Client.putObject( - PutObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build(), - RequestBody.fromString(content) - ) - - val response = s3Client.getObject( - GetObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - assertThat(response.readAllBytes().decodeToString()).isEqualTo(content) -} -``` - -### Testing Bucket Operations - -```kotlin -@Test -fun `should create and list buckets`() { - val bucketName = "test-bucket-${UUID.randomUUID()}" - - s3Client.createBucket { it.bucket(bucketName) } - - val buckets = s3Client.listBuckets() - - assertThat(buckets.buckets()) - .anyMatch { it.name() == bucketName } -} -``` - -### Testing Multipart Uploads - -```kotlin -@Test -fun `should complete multipart upload`() { - val bucketName = givenBucket() - val key = "large-file" - - // Initiate - val uploadId = s3Client.createMultipartUpload( - CreateMultipartUploadRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ).uploadId() - - // Upload parts - val part1 = s3Client.uploadPart( - UploadPartRequest.builder() - .bucket(bucketName) - .key(key) - .uploadId(uploadId) - .partNumber(1) - .build(), - RequestBody.fromString("part1") - ) - - val part2 = s3Client.uploadPart( - UploadPartRequest.builder() - .bucket(bucketName) - .key(key) - .uploadId(uploadId) - .partNumber(2) - .build(), - RequestBody.fromString("part2") - ) - - // Complete - s3Client.completeMultipartUpload( - CompleteMultipartUploadRequest.builder() - .bucket(bucketName) - .key(key) - .uploadId(uploadId) - .multipartUpload( - CompletedMultipartUpload.builder() - .parts( - CompletedPart.builder().partNumber(1).eTag(part1.eTag()).build(), - CompletedPart.builder().partNumber(2).eTag(part2.eTag()).build() - ) - .build() - ) - .build() - ) - - // Verify - val response = s3Client.getObject( - GetObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - assertThat(response.readAllBytes().decodeToString()).isEqualTo("part1part2") -} -``` - -### Testing Error Conditions - -```kotlin -@Test -fun `should throw NoSuchBucket exception`() { - assertThrows { - s3Client.getObject( - GetObjectRequest.builder() - .bucket("non-existent-bucket") - .key("key") - .build() + GetObjectRequest.builder().bucket(bucketName).key("key").build() ) + assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") } } ``` -### Testing with Headers +## Common Patterns +**Helper**: ```kotlin -@Test -fun `should handle custom metadata`() { - val bucketName = givenBucket() - val key = "key-with-metadata" - - s3Client.putObject( - PutObjectRequest.builder() - .bucket(bucketName) - .key(key) - .metadata(mapOf("custom-key" to "custom-value")) - .build(), - RequestBody.fromString("content") - ) - - val response = s3Client.headObject( - HeadObjectRequest.builder() - .bucket(bucketName) - .key(key) - .build() - ) - - assertThat(response.metadata()).containsEntry("custom-key", "custom-value") +private fun givenBucket() = "test-bucket-${UUID.randomUUID()}".also { + s3Client.createBucket { it.bucket(it) } } ``` -## Helper Methods - -### Bucket Creation +**Multipart**: Initiate → Upload parts → Complete → Verify +**Errors**: ```kotlin -private fun givenBucket(): String { - val bucketName = "test-bucket-${UUID.randomUUID()}" - s3Client.createBucket { it.bucket(bucketName) } - return bucketName +assertThrows { + s3Client.getObject(...) } ``` -### Random Test Data - +**Metadata**: ```kotlin -private fun randomBytes(size: Int): ByteArray { - val bytes = ByteArray(size) - Random.nextBytes(bytes) - return bytes -} +s3Client.putObject( + PutObjectRequest.builder() + .metadata(mapOf("key" to "value")) + .build(), + RequestBody.fromString("content") +) ``` ## Assertions -Use AssertJ for fluent assertions: - +AssertJ: ```kotlin -// Basic assertions assertThat(result).isNotNull() -assertThat(result).isEqualTo(expected) -assertThat(list).hasSize(3) -assertThat(list).contains(item) - -// Collection assertions -assertThat(buckets.buckets()) - .hasSize(1) - .anyMatch { it.name() == bucketName } - -// Exception assertions -assertThrows { - s3Client.deleteBucket { it.bucket("non-existent") } -} +assertThat(list).hasSize(3).contains(item) +assertThat(buckets.buckets()).anyMatch { it.name() == bucketName } ``` -## XML Response Verification - -For testing XML responses directly: - -```kotlin -import org.xmlunit.assertj3.XmlAssert - -@Test -fun `should return valid XML`() { - val xml = """ - - test-bucket - - """.trimIndent() - - XmlAssert.assertThat(xml) - .nodesByXPath("//Name") - .exist() -} -``` - -## Test Execution +## Running ```bash -# Run all integration tests ./mvnw verify -pl integration-tests - -# Run specific test class ./mvnw verify -pl integration-tests -Dit.test=BucketIT - -# Run specific test method ./mvnw verify -pl integration-tests -Dit.test=BucketIT#shouldCreateBucket - -# Skip Docker build ./mvnw verify -pl integration-tests -DskipDocker ``` -## S3Mock Test Configuration - -S3Mock is started automatically via Docker Maven Plugin. Configuration in `pom.xml`: - -```xml - - io.fabric8 - docker-maven-plugin - - - - adobe/s3mock:latest - - - it.s3mock.port_http:9090 - it.s3mock.port_https:9191 - - - - - - -``` - -## Test Best Practices - -1. **Isolation**: Each test is independent, no shared state -2. **Cleanup**: Tests clean up automatically (or rely on fresh S3Mock instance) -3. **Naming**: Use unique bucket names with UUID to avoid conflicts -4. **Speed**: Keep tests fast, avoid unnecessary delays -5. **Coverage**: Test both AWS SDK v1 and v2 when applicable -6. **Real SDK**: Use actual AWS SDK clients, not mocks - -## Testing Both SDK Versions - -When testing features with both SDK versions: - -```kotlin -@Test -fun `should work with SDK v1`() { - val bucketName = givenBucket() - - // Use s3ClientV1 (AWS SDK v1) - s3ClientV1.putObject(bucketName, "key", "content") - - val content = s3ClientV1.getObjectAsString(bucketName, "key") - assertThat(content).isEqualTo("content") -} - -@Test -fun `should work with SDK v2`() { - val bucketName = givenBucket() - - // Use s3Client (AWS SDK v2) - s3Client.putObject( - PutObjectRequest.builder().bucket(bucketName).key("key").build(), - RequestBody.fromString("content") - ) - - val response = s3Client.getObject( - GetObjectRequest.builder().bucket(bucketName).key("key").build() - ) - - assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") -} -``` - -## Verifying S3Mock Behavior - -Integration tests verify: -- Correct HTTP status codes -- Proper response headers (ETag, Content-Type, etc.) -- Valid XML/JSON response bodies -- Error responses match AWS S3 -- Edge cases and boundary conditions - -## Debugging Failed Tests +## Best Practices -1. Check S3Mock logs (Docker container logs) -2. Verify endpoint connectivity -3. Check request/response with HTTP client debugging -4. Validate test data and assumptions -5. Compare with AWS S3 API documentation +1. Independent tests (no shared state) +2. Unique bucket names with UUID +3. Test both SDK v1 (`s3ClientV1`) and v2 (`s3Client`) when applicable +4. Verify: HTTP codes, headers (ETag, Content-Type), XML/JSON bodies, error responses +5. Use actual AWS SDK clients (not mocks) -## References +## Debugging -- Main project context: `/AGENTS.md` -- Server implementation: `/server/AGENTS.md` -- AWS SDK v2: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/ -- AWS SDK v1: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/ +- Check Docker container logs +- Verify endpoint connectivity +- Compare with AWS S3 API docs diff --git a/server/AGENTS.md b/server/AGENTS.md index 32eb566c8..64dcd06fc 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -1,119 +1,41 @@ # Agent Context for S3Mock Server Module -This module contains the core S3Mock server implementation. +Core S3Mock server implementation. -## Module Structure +## Structure ``` -server/ -├── src/main/kotlin/ -│ └── com/adobe/testing/s3mock/ -│ ├── S3MockApplication.kt # Spring Boot application entry point -│ ├── S3MockConfiguration.kt # Main configuration -│ ├── S3MockProperties.kt # Configuration properties -│ ├── controller/ # REST API endpoints -│ │ ├── BucketController.kt # Bucket operations -│ │ ├── ObjectController.kt # Object operations -│ │ └── *Converter.kt # Header/parameter converters -│ ├── dto/ # Data Transfer Objects -│ │ ├── *Result.kt # Response objects -│ │ └── *.kt # Request/response models -│ ├── service/ # Business logic -│ │ └── ObjectService.kt # Object operations service -│ └── store/ # Persistence layer -│ ├── BucketStore.kt # Bucket storage -│ ├── ObjectStore.kt # Object storage -│ ├── BucketMetadata.kt # Bucket metadata model -│ └── S3ObjectMetadata.kt # Object metadata model -└── src/test/ # Unit tests +server/src/main/kotlin/com/adobe/testing/s3mock/ +├── S3MockApplication.kt # Spring Boot entry +├── S3MockConfiguration.kt # Config +├── S3MockProperties.kt # Properties binding +├── controller/ # REST endpoints (BucketController, ObjectController) +├── dto/ # XML/JSON models (*Result, request/response) +├── service/ # Business logic (ObjectService, etc.) +└── store/ # Persistence (BucketStore, ObjectStore, metadata) ``` -## Implementation Workflow +## Implementation Flow -When implementing new S3 operations: - -1. **Define DTOs** (`dto/`) - - Create request/response data classes - - Add Jackson XML annotations - - Match AWS S3 XML structure exactly - -2. **Update Store Layer** (`store/`) - - Add persistence methods to `BucketStore` or `ObjectStore` - - Update metadata classes if needed - - Handle file system operations - -3. **Implement Service** (`service/`) - - Add business logic methods - - Validate inputs - - Use store layer for data access - - Handle errors with S3 exceptions - -4. **Add Controller Endpoint** (`controller/`) - - Map HTTP method and path - - Handle headers and parameters - - Call service layer - - Return proper response with headers - -## DTO Patterns - -All DTOs must serialize to XML matching AWS S3 API: +**Adding S3 operation**: DTO (Jackson XML) → Store (filesystem) → Service (validation, logic) → Controller (HTTP mapping) +### 1. DTOs (`dto/`) ```kotlin @JacksonXmlRootElement(localName = "CreateBucketConfiguration") data class CreateBucketConfiguration( @JacksonXmlProperty(localName = "LocationConstraint") - val locationConstraint: LocationConstraint? = null, - - @JacksonXmlProperty(localName = "Bucket") - val bucket: BucketInfo? = null + val locationConstraint: LocationConstraint? = null ) ``` -### Common Annotations - -- `@JacksonXmlRootElement` - Root element name -- `@JacksonXmlProperty` - Element/attribute name -- `@JacksonXmlElementWrapper(useWrapping = false)` - Unwrap collections -- Custom serializers/deserializers for special cases (Region, Instant, etc.) - -## Store Layer Patterns - -The store layer manages file system persistence: - -```kotlin -@Component -class ObjectStore( - private val storeProperties: StoreProperties -) { - fun storeObject(bucket: BucketMetadata, key: String, data: InputStream): S3Object { - val uuid = UUID.randomUUID().toString() - val objectPath = getObjectPath(bucket, uuid) - Files.createDirectories(objectPath) - - // Store binary data - val dataPath = objectPath.resolve("binaryData") - Files.copy(data, dataPath, StandardCopyOption.REPLACE_EXISTING) - - // Store metadata - val metadata = S3ObjectMetadata(...) - storeMetadata(objectPath, metadata) - - return S3Object(...) - } -} -``` - -### Key Responsibilities - -- File system path resolution -- Binary data storage -- Metadata serialization/deserialization -- Cleanup and deletion +Key annotations: `@JacksonXmlRootElement`, `@JacksonXmlProperty`, `@JacksonXmlElementWrapper(useWrapping = false)` -## Service Layer Patterns - -Business logic and S3 operation implementation: +### 2. Store (`store/`) +- Filesystem path resolution, binary storage, metadata JSON +- `BucketStore`, `ObjectStore` for CRUD operations +- `BucketMetadata`, `S3ObjectMetadata` models +### 3. Service (`service/`) ```kotlin @Service class ObjectService( @@ -121,105 +43,57 @@ class ObjectService( private val objectStore: ObjectStore ) { fun getObject(bucketName: String, key: String): S3Object { - // Validate bucket exists val bucket = bucketStore.getBucketMetadata(bucketName) ?: throw NoSuchBucketException(bucketName) - - // Get object - val s3Object = objectStore.getObject(bucket, key) + return objectStore.getObject(bucket, key) ?: throw NoSuchKeyException(key) - - return s3Object } } ``` -### Key Responsibilities - -- Input validation -- Error handling with S3 exceptions -- Coordinate store operations -- Implement S3 operation logic - -## Controller Layer Patterns - -REST endpoints implementing S3 API: +Responsibilities: Validation, S3 exceptions, coordinate stores +### 4. Controller (`controller/`) ```kotlin @RestController -@RequestMapping("\${com.adobe.testing.s3mock.controller.context-path:}") -class ObjectController( - private val objectService: ObjectService -) { - @GetMapping( - value = ["/{bucketName:.+}/{*key}"], - produces = [MediaType.APPLICATION_OCTET_STREAM_VALUE] - ) - fun getObject( - @PathVariable bucketName: String, - @PathVariable key: String, - @RequestHeader(required = false) headers: Map - ): ResponseEntity { - val s3Object = objectService.getObject(bucketName, key) - - return ResponseEntity - .ok() - .header("ETag", "\"${s3Object.etag}\"") - .header("Content-Type", s3Object.contentType) - .header("Last-Modified", s3Object.lastModified) - .body(s3Object.dataStream) - } +class ObjectController(private val objectService: ObjectService) { + @GetMapping("/{bucketName:.+}/{*key}") + fun getObject(@PathVariable bucketName: String, @PathVariable key: String) = + objectService.getObject(bucketName, key).let { + ResponseEntity.ok() + .header("ETag", "\"${it.etag}\"") + .body(it.dataStream) + } } ``` -### Key Responsibilities - -- HTTP request mapping -- Header and parameter parsing -- Response construction with proper headers -- Stream large responses when needed +Responsibilities: HTTP mapping, headers, streaming responses ## Testing -### Unit Tests - -Test individual components with mocked dependencies: - +Unit tests with Mockito: ```kotlin @ExtendWith(MockitoExtension::class) -internal class ObjectServiceTest { - @Mock - private lateinit var bucketStore: BucketStore - - @Mock - private lateinit var objectStore: ObjectStore - - @InjectMocks - private lateinit var objectService: ObjectService +class ObjectServiceTest { + @Mock lateinit var bucketStore: BucketStore + @Mock lateinit var objectStore: ObjectStore + @InjectMocks lateinit var objectService: ObjectService @Test fun `should get object`() { - val bucket = BucketMetadata(...) - val s3Object = S3Object(...) - whenever(bucketStore.getBucketMetadata("bucket")).thenReturn(bucket) whenever(objectStore.getObject(bucket, "key")).thenReturn(s3Object) - - val result = objectService.getObject("bucket", "key") - - assertThat(result).isEqualTo(s3Object) + assertThat(objectService.getObject("bucket", "key")).isEqualTo(s3Object) } } ``` ## Configuration -Configuration bound from environment variables: - ```kotlin @ConfigurationProperties(prefix = "com.adobe.testing.s3mock.store") data class StoreProperties( - val root: Path = Files.createTempDirectory("s3mock"), + val root: Path, val retainFilesOnExit: Boolean = false, val validKmsKeys: Set = emptySet(), val initialBuckets: Set = emptySet(), @@ -227,44 +101,10 @@ data class StoreProperties( ) ``` -## Running the Server +## Running ```bash -# As Spring Boot application ./mvnw spring-boot:run -pl server - -# Build executable JAR -./mvnw package -pl server -am -java -jar server/target/s3mock-*.jar - -# With Docker -./mvnw clean package -pl server -am -DskipTests +./mvnw package -pl server -am && java -jar server/target/s3mock-*.jar docker run -p 9090:9090 -p 9191:9191 adobe/s3mock:latest ``` - -## Common Tasks - -### Adding a New S3 Operation - -1. Check AWS S3 API documentation -2. Create request/response DTOs in `dto/` -3. Add store methods in `store/` if needed -4. Implement service method in `service/` -5. Add controller endpoint in `controller/` -6. Write unit tests -7. Test with integration tests (see `integration-tests/AGENTS.md`) - -### Updating Existing Operations - -1. Identify affected layers (controller, service, store) -2. Update DTOs if request/response changed -3. Modify service logic -4. Update store operations if persistence changed -5. Update tests -6. Verify with integration tests - -## References - -- Main project context: `/AGENTS.md` -- Integration testing: `/integration-tests/AGENTS.md` -- AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/ diff --git a/testsupport/AGENTS.md b/testsupport/AGENTS.md index 4b0d26297..5dea9217b 100644 --- a/testsupport/AGENTS.md +++ b/testsupport/AGENTS.md @@ -1,324 +1,81 @@ # Agent Context for S3Mock Test Support -This module provides test framework integrations for using S3Mock in various testing environments. +Test framework integrations for using S3Mock: JUnit 4/5, Testcontainers, TestNG. -## Module Structure +## Framework Selection -``` -testsupport/ -├── common/ # Shared test support utilities -├── junit4/ # JUnit 4 Rule -├── junit5/ # JUnit 5 Extension -├── testcontainers/ # Testcontainers support -└── testng/ # TestNG Listener -``` - -## Common Module - -Shared utilities used by all test framework integrations. - -### Key Classes - -- `S3MockStarter` - Base class for starting S3Mock programmatically -- Configuration helpers -- Common test utilities +- **JUnit 5** (`junit5/`) - Modern extension with parameter injection. Default for new projects. +- **JUnit 4** (`junit4/`) - Legacy Rule API for older projects. +- **Testcontainers** (`testcontainers/`) - Docker container isolation, works with any framework. +- **TestNG** (`testng/`) - Listener-based integration for TestNG projects. +- **Common** (`common/`) - Shared `S3MockStarter` base class and utilities. ## JUnit 5 Extension -Provides `S3MockExtension` for JUnit 5 tests. - -### Usage Patterns - -**Declarative with Injection**: - ```kotlin @ExtendWith(S3MockExtension::class) class MyTest { @Test fun test(s3Client: S3Client) { - // S3Client injected automatically s3Client.createBucket { it.bucket("test-bucket") } } } ``` -**Programmatic with Builder**: - +Builder alternative: ```kotlin -class MyTest { - @RegisterExtension - val s3Mock = S3MockExtension.builder() - .withInitialBuckets("bucket1", "bucket2") - .withValidKmsKeys("arn:aws:kms:us-east-1:1234567890:key/test-key") - .build() - - @Test - fun test() { - val s3Client = s3Mock.createS3ClientV2() - // Use client - } -} +@RegisterExtension +val s3Mock = S3MockExtension.builder() + .withInitialBuckets("bucket1") + .withValidKmsKeys("arn:aws:kms:...") + .build() ``` -### Implementation Notes - -- Starts S3Mock before all tests -- Stops S3Mock after all tests -- Provides parameter injection for S3Client -- Supports both HTTP and HTTPS endpoints - ## JUnit 4 Rule -Provides `S3MockRule` for JUnit 4 tests. - -### Usage - ```java -public class MyTest { - @Rule - public S3MockRule s3MockRule = S3MockRule.builder() - .withInitialBuckets("test-bucket") - .build(); - - @Test - public void test() { - AmazonS3 s3Client = s3MockRule.createS3Client(); - s3Client.createBucket("my-bucket"); - } -} -``` - -## Testcontainers Support - -Provides `S3MockContainer` - a Testcontainers implementation. - -### Usage - -**With JUnit 5**: - -```kotlin -@Testcontainers -class MyTest { - @Container - val s3Mock = S3MockContainer("latest") - .withInitialBuckets("test-bucket") - .withValidKmsKeys("test-key-id") - - @Test - fun test() { - val s3Client = S3Client.builder() - .endpointOverride(URI.create(s3Mock.httpEndpoint)) - .region(Region.US_EAST_1) - .credentialsProvider( - StaticCredentialsProvider.create( - AwsBasicCredentials.create("foo", "bar") - ) - ) - .build() - - s3Client.createBucket { it.bucket("my-bucket") } - } -} +@Rule +public S3MockRule s3MockRule = S3MockRule.builder() + .withInitialBuckets("test-bucket") + .build(); ``` -**Manual Management**: +## Testcontainers ```kotlin -class MyTest { - private lateinit var s3Mock: S3MockContainer +@Container +val s3Mock = S3MockContainer("latest") + .withInitialBuckets("test-bucket") + .withValidKmsKeys("key-id") - @BeforeEach - fun setup() { - s3Mock = S3MockContainer("latest") - .withInitialBuckets("test-bucket") - s3Mock.start() - } - - @AfterEach - fun teardown() { - s3Mock.stop() - } - - @Test - fun test() { - // Use s3Mock.httpEndpoint or s3Mock.httpsEndpoint - } -} +// Access via s3Mock.httpEndpoint or s3Mock.httpsEndpoint ``` -### Container Configuration - -```kotlin -S3MockContainer("latest") - .withInitialBuckets("bucket1", "bucket2") - .withValidKmsKeys("key1", "key2") - .withRetainFilesOnExit(true) - .withRoot("/custom/path") - .withRegion("us-west-2") -``` - -### Key Methods - -- `httpEndpoint` - Get HTTP endpoint URL -- `httpsEndpoint` - Get HTTPS endpoint URL -- `withInitialBuckets()` - Pre-create buckets -- `withValidKmsKeys()` - Configure valid KMS keys -- `withRetainFilesOnExit()` - Keep files after container stops +Configuration methods: `withInitialBuckets()`, `withValidKmsKeys()`, `withRetainFilesOnExit()`, `withRoot()`, `withRegion()`. ## TestNG Listener -Provides `S3MockListener` for TestNG tests. - -### Usage - Configure in `testng.xml`: - ```xml - - - - - - - - - - - - - + ``` -In test class: - +Access via system properties: ```kotlin -class MyTest { - @Test - fun test() { - val endpoint = System.getProperty("s3mock.httpEndpoint") - val s3Client = S3Client.builder() - .endpointOverride(URI.create(endpoint)) - // ... configure - .build() - - s3Client.createBucket { it.bucket("my-bucket") } - } -} +val endpoint = System.getProperty("s3mock.httpEndpoint") ``` -### Configuration Parameters - -- `s3mock.httpPort` - HTTP port (default: 9090) -- `s3mock.httpsPort` - HTTPS port (default: 9191) -- `s3mock.initialBuckets` - Comma-separated bucket names -- `s3mock.root` - Storage root directory - -## Writing Test Support Code - -When modifying or extending test support: - -### Design Principles +Parameters: `s3mock.httpPort` (9090), `s3mock.httpsPort` (9191), `s3mock.initialBuckets`, `s3mock.root`. -1. **Framework-agnostic core**: Keep common logic in `common` module -2. **Simple API**: Minimize configuration complexity -3. **Sensible defaults**: Work out of the box for common cases -4. **Lifecycle management**: Proper startup/shutdown -5. **Resource cleanup**: Clean up after tests +## Design Principles -### Example: Adding New Configuration - -```kotlin -// In common module -abstract class S3MockStarter { - var customOption: String? = null - - protected fun buildEnvironment(): Map { - val env = mutableMapOf() - customOption?.let { env["CUSTOM_ENV_VAR"] = it } - return env - } -} - -// In JUnit 5 extension -class S3MockExtension { - class Builder { - fun withCustomOption(value: String): Builder { - this.customOption = value - return this - } - } -} -``` - -## Testing Test Support - -Each module includes tests: - -```kotlin -// Test the extension itself -@ExtendWith(S3MockExtension::class) -class S3MockExtensionTest { - @Test - fun `extension should start S3Mock`(s3Client: S3Client) { - // Verify S3Mock is accessible - assertThat(s3Client.listBuckets()).isNotNull() - } - - @Test - fun `should create initial buckets`() { - // Test initial bucket creation - } -} -``` - -## Maven Dependencies - -Users add test support as test dependencies: - -**JUnit 5**: -```xml - - com.adobe.testing - s3mock-junit5 - ${s3mock.version} - test - -``` - -**Testcontainers**: -```xml - - com.adobe.testing - s3mock-testcontainers - ${s3mock.version} - test - -``` +1. Framework-agnostic core in `common` module +2. Simple API with sensible defaults +3. Proper lifecycle management (start before tests, stop after) +4. Resource cleanup after tests ## Common Issues -### S3Mock Not Starting - -- Check port availability (9090, 9191) -- Verify Docker is running (for Testcontainers) -- Check logs for startup errors - -### Connection Refused - -- Ensure S3Mock started before tests run -- Verify endpoint URL is correct -- Check firewall/network settings - -### Tests Interfere with Each Other - -- Use unique bucket names (UUID) -- Ensure proper test isolation -- Consider separate S3Mock instances per test class - -## References - -- Main project context: `/AGENTS.md` -- Integration tests: `/integration-tests/AGENTS.md` -- JUnit 5: https://junit.org/junit5/ -- Testcontainers: https://www.testcontainers.org/ -- TestNG: https://testng.org/ +- **Not starting**: Check ports 9090/9191 availability, Docker running (Testcontainers) +- **Connection refused**: Verify S3Mock started, correct endpoint URL +- **Test interference**: Use unique bucket names (UUID), ensure test isolation From 080ff03c8df93766630e6be05e9f1431f5154bbd Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sun, 22 Feb 2026 14:12:05 +0100 Subject: [PATCH 07/36] feat: expand README.md with "Architecture & Development" section Added a new section detailing project structure, architecture overview, and testing philosophy to provide clearer guidance for developers. --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 5bea70fe8..9ac646669 100755 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ * [Configuration](#configuration) * [Important Limitations](#important-limitations) * [File System Structure](#file-system-structure) + * [Architecture & Development](#architecture--development) * [Build & Run](#build--run) * [Contributing](#contributing) * [License](#license) @@ -346,6 +347,14 @@ S3Mock stores data on disk with the following structure: **Note:** The file system structure is an implementation detail and may change between releases. While files can be inspected during runtime, reusing persisted data across restarts is not officially supported. +## Architecture & Development + +**Module Documentation:** +- [Project Overview](AGENTS.md) - Architecture, code style, testing philosophy +- [Server Module](server/AGENTS.md) - Core implementation (Controller→Service→Store layers) +- [Integration Tests](integration-tests/AGENTS.md) - Testing with AWS SDK clients +- [Test Support](testsupport/AGENTS.md) - JUnit 4/5, Testcontainers, TestNG integrations + ## Build & Run **Requirements**: Java 17+, Maven 3.9+, Docker (for Docker build and integration tests) From 9482c4edb770fcec3dfb1521fd15df6f656f1d8d Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Sun, 22 Feb 2026 17:21:18 +0100 Subject: [PATCH 08/36] feat: improve README usability Updated `MADE with` section links to point to "Build & Run" and made the operations table collapsible for better readability. --- README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9ac646669..4e6ef2f28 100755 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ ![Maven Build](https://github.com/adobe/S3Mock/workflows/Maven%20Build/badge.svg) [![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/) [![Docker Pulls](https://img.shields.io/docker/pulls/adobe/s3mock)](https://hub.docker.com/r/adobe/s3mock) -[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#kotlin) -[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#java) +[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#build--run) +[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#build--run) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7673/badge)](https://bestpractices.coreinfrastructure.org/projects/7673) [![OpenSSF Scorecard](https://img.shields.io/ossf-scorecard/github.com/adobe/S3Mock?label=openssf%20scorecard&style=flat)](https://api.securityscorecards.dev/projects/github.com/adobe/S3Mock) [![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/) @@ -41,9 +41,10 @@ S3Mock is a lightweight server implementing parts of the [Amazon S3 API](https:/ ## Supported S3 Operations -See the [complete operations table](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) in AWS documentation. +See the [complete operations table](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) in AWS documentation. -Operations marked :white_check_mark: below are supported by S3Mock: +
+Click to expand operations table (operations marked :white_check_mark: are supported) | Operation | Support | Comment | |-----------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|------------------------| @@ -146,6 +147,8 @@ Operations marked :white_check_mark: below are supported by S3Mock: | [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) | :white_check_mark: | | | [WriteGetObjectResponse](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html) | :x: | | +
+ ## Usage ### Docker (Recommended) From eea69c62060fdc3eec4f520e445e7a55f1f93f04 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 12:15:40 +0100 Subject: [PATCH 09/36] chore: changelog for 5.0.0 --- CHANGELOG.md | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a8c8a6a3b..ea7c9b644 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # Changelog S3Mock follows [Semantic Versioning](https://semver.org/). It depends on lots of 3rd party libraries which are updated regularly. -Whenever a 3rd party library is updated, S3Mock will update it's MINOR version. +Whenever a 3rd party library is updated, S3Mock will update its MINOR version. * [Changelog](#changelog) @@ -115,31 +115,27 @@ Whenever a 3rd party library is updated, S3Mock will update it's MINOR version. # PLANNED - 6.x - RELEASE TBD -Version 6.x is JDK25 LTS bytecode compatible, with Docker integration. +Version 6.x will target JDK 25 LTS bytecode compatibility with Docker-only integration. -Will be released after Spring Boot 5.x, updating baselines etc. as Spring Boot 5.x requires. +Will be released after Spring Boot 5.x becomes available. -Any JUnit / direct Java usage support will most likely be dropped and only supported on a best-effort basis. -(i.e., the modules will be deleted from the code base and not released anymore. It *may* be possible to -still run S3Mock directly in Java.) -The S3Mock is a Spring Boot application and currently contains various workarounds to make it possible -to easily to run `S3MockApplication#start` from a static context. These workarounds will be deleted. +**Breaking Changes:** +- **JUnit/TestNG direct integration removed.** Test framework modules (junit4, junit5, testng) will be deleted from the codebase and no longer released. Running S3Mock embedded in the same JVM will no longer be officially supported. +- **TestContainers becomes the recommended testing approach.** Use [TestContainers](https://www.testcontainers.org/) to run S3Mock as a Docker container in your tests. +- **Standard Spring Boot application.** S3Mock will be refactored to a standard Spring Boot application, removing workarounds that enabled `S3MockApplication#start()` in static contexts. -Running S3Mock in unit tests is still supported by using [TestContainers](https://www.testcontainers.org/). - -**Once 6.x is released, 5.x may receive bug fixes and features. This will be best-effort only.** +**Support:** Once 6.x is released, version 5.x may receive bug fixes and features on a best-effort basis only. ## Planned changes -* Features and fixes +* **Features and fixes** * TBD -* Refactorings - * Looking to Remove unit test modules. This enables - * Refactoring S3Mock to a "standard" Spring Boot application. - * Removal of workarounds to use `S3MockApplication#start` from a static context -* Version updates - * Bump Spring Boot version to 5.0.0 - * Bump Spring Framework version to 8.0.0 +* **Refactorings** + * Remove unit test modules (junit4, junit5, testng) + * Refactor to standard Spring Boot application without static context workarounds +* **Version updates** + * Spring Boot 5.0.0 + * Spring Framework 8.0.0 # CURRENT - 5.x - THIS VERSION IS UNDER ACTIVE DEVELOPMENT Version 5.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration. @@ -183,13 +179,17 @@ Version 5.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Jav * Bump org.apache.maven.plugins:maven-release-plugin from 3.3.0 to 3.3.1 * Bump org.codehaus.mojo:exec-maven-plugin from 3.6.2 to 3.6.3 * Bump org.apache.maven.plugins:maven-compiler-plugin from 3.14.1 to 3.15.0 + * Bump org.apache.maven.plugins:maven-dependency-plugin from 3.9.0 to 3.10.0 + * Bump io.fabric8:docker-maven-plugin from 0.48.0 to 0.48.1 * Bump digital.pragmatech.testing:spring-test-profiler from 0.0.14 to 0.0.15 * Bump com.puppycrawl.tools:checkstyle from 12.2.0 to 13.2.0 - * Bump actions/upload-artifact from 5.0.0 to 6.0.0 * Bump actions/checkout from 6.0.1 to 6.0.2 - * Bump github/codeql-action from 4.31.6 to 4.32.2 + * Bump actions/dependency-review-action from 4.8.2 to 4.8.3 * Bump actions/setup-java from 5.0.0 to 5.2.0 - * Bump step-security/harden-runner from 2.13.3 to 2.14.1 + * Bump actions/stale from 10.1.1 to 10.2.0 + * Bump actions/upload-artifact from 5.0.0 to 6.0.0 + * Bump github/codeql-action from 4.31.6 to 4.32.4 + * Bump step-security/harden-runner from 2.13.3 to 2.14.2 # DEPRECATED - 4.x Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration. From 5eec9fdc93394e1f2578dd03a89cb0f9cf49918d Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:07:35 +0100 Subject: [PATCH 10/36] feat: add security policy --- .github/SECURITY.md | 48 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 .github/SECURITY.md diff --git a/.github/SECURITY.md b/.github/SECURITY.md new file mode 100644 index 000000000..ced54cc9f --- /dev/null +++ b/.github/SECURITY.md @@ -0,0 +1,48 @@ +# Security Policy + +## Important Notice + +S3Mock is a **testing tool** designed for local integration testing. It is **not intended for production use** and lacks the security features required for production environments. Do not expose S3Mock to untrusted networks or use it to store sensitive data. + +## Supported Versions + +| Version | Supported | +|---------|--------------------| +| 5.x | :white_check_mark: | +| 4.x | :x: | +| 3.x | :x: | +| < 3.0 | :x: | + +## Reporting a Vulnerability + +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, please report vulnerabilities using one of the following methods: + +1. **GitHub Security Advisories** (preferred): Use the [private vulnerability reporting](https://github.com/adobe/S3Mock/security/advisories/new) feature to submit a report directly through GitHub. + +2. **Email**: Send a report to [Grp-opensourceoffice@adobe.com](mailto:Grp-opensourceoffice@adobe.com). + +### What to Include + +- Type of vulnerability +- Affected version(s) +- Steps to reproduce +- Impact assessment +- Suggested fix (if any) + +### Response Timeline + +- **Acknowledgment**: Within 5 business days +- **Initial assessment**: Within 10 business days +- **Resolution**: Depending on severity and complexity + +## Security Measures + +S3Mock uses the following automated security tools: + +- **CodeQL**: Static analysis for security vulnerabilities (via GitHub Actions) +- **SBOM**: Software Bill of Materials generation for dependency tracking +- **OpenSSF Scorecard**: Security health assessment +- **Dependabot**: Automated dependency updates for Maven, Docker, and GitHub Actions +- **Dependency Review**: Automated review of dependency changes in PRs From 79c94b06150e336671b4f912804a4fb911884ef4 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:15:25 +0100 Subject: [PATCH 11/36] feat: add quick start and troubleshooting sections --- README.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4e6ef2f28..781cde780 100755 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ * [S3Mock](#s3mock) + * [Quick Start](#quick-start) * [Changelog](#changelog) * [Supported S3 Operations](#supported-s3-operations) * [Usage](#usage) @@ -21,6 +22,7 @@ * [cURL](#curl) * [Configuration](#configuration) * [Important Limitations](#important-limitations) + * [Troubleshooting](#troubleshooting) * [File System Structure](#file-system-structure) * [Architecture & Development](#architecture--development) * [Build & Run](#build--run) @@ -34,6 +36,26 @@ S3Mock is a lightweight server implementing parts of the [Amazon S3 API](https:/ **Recommended usage**: Run S3Mock as a Docker container or with Testcontainers to avoid classpath conflicts. +## Quick Start + +Get up and running in 30 seconds: + +```shell +# 1. Start S3Mock +docker run -p 9090:9090 adobe/s3mock + +# 2. Create a bucket +aws s3api create-bucket --bucket my-bucket --endpoint-url http://localhost:9090 + +# 3. Upload a file +aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url http://localhost:9090 + +# 4. Download the file +aws s3api get-object --bucket my-bucket --key my-file --endpoint-url http://localhost:9090 output-file +``` + +For programmatic testing, see [Testcontainers](#testcontainers) or [JUnit 5 Extension](#junit-5-extension) below. + ## Changelog - [GitHub Releases](https://github.com/adobe/S3Mock/releases) @@ -329,6 +351,49 @@ Configure S3Mock using environment variables: - **KMS**: Key validation only - no actual encryption performed - **Not for production**: S3Mock is a testing tool and lacks security features required for production use +## Troubleshooting + +
+Click to expand troubleshooting guide + +**Port already in use (`Address already in use`)** +- Ports `9090` (HTTP) and `9191` (HTTPS) must be free +- Check with: `lsof -i :9090` (macOS/Linux) or `netstat -ano | findstr :9090` (Windows) +- Stop conflicting processes or map to different ports: `docker run -p 9091:9090 adobe/s3mock` + +**Connection refused** +- Verify S3Mock is running: `docker ps | grep s3mock` +- Ensure you're using the correct endpoint URL (e.g., `http://localhost:9090`) +- Wait for startup to complete — check logs with `docker logs ` + +**SSL certificate errors** +- S3Mock uses a self-signed certificate on the HTTPS port (9191) +- AWS CLI: Add `--no-verify-ssl` flag +- cURL: Add `--insecure` flag +- Java/Kotlin: Configure the SDK to trust the S3Mock certificate or disable SSL verification + +**Docker not running (Testcontainers)** +- Testcontainers and integration tests require a running Docker daemon +- Start Docker Desktop or the Docker service before running tests +- Check with: `docker info` + +**Classpath conflicts (JUnit 5 Extension / embedded mode)** +- S3Mock's embedded Spring Boot server may conflict with your application's dependencies +- **Recommended**: Use [Testcontainers](#testcontainers) or [Docker](#docker-recommended) instead to run S3Mock in isolation +- If using embedded mode, ensure compatible Spring Boot versions + +**AWS SDK endpoint configuration** +- AWS SDK v2: Use `.endpointOverride(URI.create("http://localhost:9090"))` +- Credentials: Use any dummy credentials (e.g., `AwsBasicCredentials.create("foo", "bar")`) +- Region: Use any region (S3Mock defaults to `us-east-1`) + +**Objects not found / wrong bucket** +- S3Mock supports **path-style access only**: `http://localhost:9090/bucket/key` +- Virtual-hosted style (`http://bucket.localhost:9090/key`) is **not supported** +- Verify your SDK client is configured for path-style access + +
+ ## File System Structure S3Mock stores data on disk with the following structure: @@ -397,7 +462,7 @@ Contributions are welcome! See [Contributing Guide](.github/CONTRIBUTING.md). **Governance**: Project leads make final decisions - see `developers` in [pom.xml](pom.xml). -**Security**: Report vulnerabilities via GitHub issues. S3Mock uses GitHub Actions for SBOM and vulnerability scanning. +**Security**: See [Security Policy](.github/SECURITY.md) for reporting vulnerabilities. S3Mock uses GitHub Actions for SBOM and vulnerability scanning. ## License From bf00a7bf069cda936b8c00dc6faffc8166a5ff24 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:39:33 +0100 Subject: [PATCH 12/36] feat: DO/DON'T lists, clarifications for agents --- .claude/skills/document/SKILL.md | 88 +++++++++++------ .claude/skills/implement/SKILL.md | 107 ++++++++++++--------- .claude/skills/test/SKILL.md | 153 +++++++++++++++++++++++------- AGENTS.md | 72 +++++++++++++- server/AGENTS.md | 14 +-- 5 files changed, 316 insertions(+), 118 deletions(-) diff --git a/.claude/skills/document/SKILL.md b/.claude/skills/document/SKILL.md index 35a91122a..06fc2bc00 100644 --- a/.claude/skills/document/SKILL.md +++ b/.claude/skills/document/SKILL.md @@ -3,42 +3,76 @@ name: document description: Generate or update project documentation. Use when asked to document code, create docs, or explain features. --- -# Documentation Skill +# Documentation Skill — S3Mock -Generate and maintain project documentation. +Generate and maintain documentation for the S3Mock project. ## When to Use -- Creating or updating README files -- Generating API documentation -- Documenting features and usage -- Writing guides and examples +- Creating or updating README.md +- Updating the CHANGELOG.md for new features/fixes +- Updating the S3 operations support table +- Documenting new configuration options +- Updating AGENTS.md files when architecture changes -## Instructions +## Pre-Flight Checklist -1. **Understand Context** - - Review the code or feature to document - - Check existing documentation style - - Identify target audience +- [ ] Read existing `AGENTS.md` for project conventions and structure +- [ ] Review the documentation being updated for current style and tone +- [ ] Identify target audience (end users for README, agents for AGENTS.md, contributors for CONTRIBUTING.md) -2. **Gather Information** - - Read source code and comments - - Review test files for examples - - Check external references (APIs, specs) +## S3Mock Documentation Structure -3. **Write Clear Documentation** - - Use concise, clear language - - Include code examples - - Add links to related docs - - Follow project conventions (see AGENTS.md) +| File | Audience | Purpose | +|------|----------|---------| +| `README.md` | End users & contributors | Usage, configuration, quick start | +| `CHANGELOG.md` | End users | Version history, breaking changes, migration notes | +| `AGENTS.md` (root) | AI agents & contributors | Architecture, code style, DO/DON'T guardrails | +| `server/AGENTS.md` | AI agents | Server module implementation details | +| `integration-tests/AGENTS.md` | AI agents | Integration test patterns and helpers | +| `testsupport/AGENTS.md` | AI agents | Test framework integration details | +| `.github/CONTRIBUTING.md` | Contributors | How to contribute, CLA, code reviews | -4. **Quality Check** - - Technical accuracy verified - - Examples work correctly - - Links are valid - - Consistent style - - Proper formatting +## Documentation Tasks + +### When a New S3 Operation is Implemented +1. Update the **operations table** in `README.md` — change `:x:` to `:white_check_mark:` for the operation +2. Add a CHANGELOG entry under the current version section in `CHANGELOG.md` +3. Update `server/AGENTS.md` if the implementation introduces new patterns + +### When Configuration Changes +1. Update the **Configuration table** in `README.md` +2. Update the **Configuration section** in `AGENTS.md` +3. Add a CHANGELOG entry + +### When Architecture Changes +1. Update the relevant module's `AGENTS.md` +2. Update the root `AGENTS.md` if the change affects the overall structure +3. Add a CHANGELOG entry for breaking changes + +### CHANGELOG Format +Follow the existing pattern in `CHANGELOG.md`: +- Group changes under the current version heading (e.g., `## 5.0.0`) +- Use clear, user-facing language +- Note breaking changes prominently +- Reference related GitHub issues or PRs + +## Writing Style + +- **Concise**: Short sentences, active voice +- **Code examples**: Include runnable examples where possible (Kotlin for API, shell for CLI) +- **Links**: Reference AWS S3 API docs for operations, link to source files for implementations +- **Consistent**: Match existing formatting — Markdown headings, table alignment, badge style + +## Post-Flight Checklist + +- [ ] Technical accuracy verified against source code +- [ ] Code examples compile/run correctly +- [ ] Links are valid (internal file paths, external URLs) +- [ ] Consistent style with surrounding documentation +- [ ] Markdown renders correctly (tables, code blocks, badges) +- [ ] No outdated version numbers or deprecated references ## Output -Provide documentation ready to integrate into project files. +Provide documentation ready to integrate into the appropriate project files, matching existing conventions. diff --git a/.claude/skills/implement/SKILL.md b/.claude/skills/implement/SKILL.md index addf077c7..234f2341b 100644 --- a/.claude/skills/implement/SKILL.md +++ b/.claude/skills/implement/SKILL.md @@ -3,56 +3,71 @@ name: implement description: Implement features, fix bugs, or refactor code. Use when asked to add functionality, modify code, or improve structure. --- -# Implementation Skill +# Implementation Skill — S3Mock -Implement features and fix bugs following best practices. +Implement features and fix bugs in the S3Mock project (Kotlin 2.3, Spring Boot 4.0.x, Maven). ## When to Use -- Adding new features -- Fixing bugs -- Refactoring code -- Optimizing performance -- Updating dependencies - -## Instructions - -1. **Understand the Requirement** - - Review specifications or issue description - - Check existing similar implementations - - Understand expected behavior and edge cases - - Review project architecture (see AGENTS.md) - -2. **Plan the Implementation** - - Identify affected components - - Consider architectural patterns - - Plan for error handling - - Think about testing - -3. **Write Clean Code** - - Follow project style guide (see AGENTS.md) - - Use appropriate naming conventions - - Add necessary comments - - Handle errors properly - - Consider edge cases - -4. **Follow Project Standards** - - Match existing code style - - Use project utilities and patterns - - Follow architectural guidelines (see AGENTS.md) - - Maintain consistency - -5. **Verify Implementation** - - Code compiles/builds - - Tests pass - - Linting passes - - No regressions - -6. **Document Changes** - - Add/update code comments - - Update relevant documentation - - Document breaking changes +- Adding new S3 API operations +- Fixing bugs in existing operations +- Refactoring server, service, or store layers +- Updating DTOs or XML serialization + +## Pre-Flight Checklist + +- [ ] Read the root `AGENTS.md` — especially the DO/DON'T section +- [ ] Read the module-specific `AGENTS.md` (`server/AGENTS.md`, etc.) +- [ ] Check `CHANGELOG.md` for planned changes or deprecations +- [ ] Identify which S3 API operation is being implemented (check [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)) +- [ ] Review existing similar implementations in the codebase + +## Implementation Flow for New S3 Operations + +Follow the **DTO → Store → Service → Controller** layered architecture: + +### 1. DTO (`server/src/main/kotlin/com/adobe/testing/s3mock/dto/`) +- Create request/response data classes with Jackson XML annotations +- Use `@JacksonXmlRootElement(localName = "...")` matching the AWS API element name exactly +- Use `@JacksonXmlProperty(localName = "...")` for properties +- Use `@JacksonXmlElementWrapper(useWrapping = false)` for collections +- Verify naming against [AWS S3 API docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) + +### 2. Store (`server/src/main/kotlin/com/adobe/testing/s3mock/store/`) +- Add filesystem operations to `BucketStore` or `ObjectStore` +- Follow existing patterns for metadata JSON and binary data storage +- Handle file I/O with proper error handling + +### 3. Service (`server/src/main/kotlin/com/adobe/testing/s3mock/service/`) +- Add business logic, validation, and store coordination +- Throw S3 exceptions (`NoSuchBucketException`, `NoSuchKeyException`, etc.) +- Use constructor injection for dependencies + +### 4. Controller (`server/src/main/kotlin/com/adobe/testing/s3mock/controller/`) +- Add HTTP endpoint mapping (`@GetMapping`, `@PutMapping`, etc.) +- Controllers only map HTTP — delegate all logic to services +- Return proper HTTP status codes and headers (ETag, Content-Type, etc.) + +## Code Standards + +- **Language**: Kotlin 2.3, JVM target 17 +- **DI**: Constructor injection only — never `@Autowired` or field injection +- **DTOs**: Data classes with `val` properties +- **Null safety**: Use `?`, `?.`, `?:` — avoid `!!` +- **Functions**: Expression bodies for simple functions +- **Dependencies**: Prefer Kotlin stdlib over third-party libraries +- **Versions**: All dependency versions in root `pom.xml` only + +## Post-Flight Checklist + +- [ ] Run `./mvnw ktlint:format` to fix code style +- [ ] Run `./mvnw clean install` to verify build +- [ ] Verify no checkstyle violations +- [ ] Add/update unit tests (`*Test.kt`) for new service/store logic +- [ ] Add/update integration tests (`*IT.kt`) for new endpoints +- [ ] Update `CHANGELOG.md` under the current version section +- [ ] Update the operations table in `README.md` if a new S3 operation was added ## Output -Provide clean, tested, well-documented code following project conventions. +Provide clean, well-structured Kotlin code following the layered architecture and project conventions defined in AGENTS.md. diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md index cd94da828..d31c88d5d 100644 --- a/.claude/skills/test/SKILL.md +++ b/.claude/skills/test/SKILL.md @@ -3,48 +3,129 @@ name: test description: Write, update, or fix tests. Use when asked to test code, create test cases, or debug failing tests. --- -# Test Skill +# Test Skill — S3Mock -Create and maintain comprehensive tests. +Create and maintain tests for S3Mock (JUnit 5, Mockito, AssertJ, AWS SDK v2). ## When to Use - Writing new unit or integration tests - Fixing failing tests -- Adding test coverage -- Creating test fixtures - -## Instructions - -1. **Identify Test Type** - - Unit tests: Test components in isolation - - Integration tests: Test system interactions - - End-to-end tests: Test full workflows - -2. **Understand Test Context** - - Locate source code to test - - Review existing test patterns - - Identify test framework and conventions (see AGENTS.md) - -3. **Write Tests** - - Use Arrange-Act-Assert pattern - - Test both success and failure cases - - Use descriptive test names - - Keep tests focused and independent - -4. **Follow Project Standards** - - Match existing test style - - Use project test utilities - - Follow naming conventions (see AGENTS.md) - - Set up proper test fixtures - -5. **Quality Check** - - Tests pass locally - - Good coverage of edge cases - - Clear assertions - - No flaky tests - - Fast execution +- Adding test coverage for S3 operations +- Creating test fixtures or helpers + +## Pre-Flight Checklist + +- [ ] Read `AGENTS.md` — especially DO/DON'T and Testing sections +- [ ] Identify test type needed (unit vs. integration — see below) +- [ ] Review existing test patterns in the target module +- [ ] For integration tests, review `S3TestBase` for available helpers + +## Test Types in S3Mock + +### Unit Tests (`*Test.kt`) +- **Location**: `server/src/test/kotlin/com/adobe/testing/s3mock/` +- **Framework**: JUnit 5 + `@SpringBootTest` with `@MockitoBean` for mocking +- **Assertions**: AssertJ (`assertThat(...)`) +- **Purpose**: Test services and stores in isolation with Spring context and mocked dependencies +- **Pattern**: +```kotlin +@SpringBootTest(classes = [ServiceConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) +@MockitoBean(types = [BucketService::class, MultipartService::class, MultipartStore::class]) +internal class ObjectServiceTest : ServiceTestBase() { + @Autowired + private lateinit var iut: ObjectService + + @Test + fun `should delete object`() { + val bucketName = "bucket" + val key = "key" + givenBucketWithContents(bucketName, "", listOf(givenS3Object(key))) + + iut.deleteObject(bucketName, key) + + assertThat(iut.getObject(bucketName, key)).isNull() + } +} +``` + +### Integration Tests (`*IT.kt`) +- **Location**: `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/` +- **Base class**: Extend `S3TestBase` for access to pre-configured `s3Client` (AWS SDK v2) +- **Assertions**: AssertJ +- **Purpose**: Test S3Mock end-to-end with real AWS SDK clients against the Docker container +- **Pattern**: +```kotlin +class MyFeatureIT : S3TestBase() { + @Test + fun `should perform operation`(testInfo: TestInfo) { + // Arrange — always use unique bucket names + val bucketName = givenBucket(testInfo) + + // Act + s3Client.putObject( + PutObjectRequest.builder().bucket(bucketName).key("key").build(), + RequestBody.fromString("content") + ) + + // Assert + val response = s3Client.getObject( + GetObjectRequest.builder().bucket(bucketName).key("key").build() + ) + assertThat(response.readAllBytes().decodeToString()).isEqualTo("content") + } +} +``` + +## Test Standards + +- **Naming**: Use backtick names with descriptive sentences: `` fun `should create bucket successfully`() `` + - Legacy `testSomething` camelCase naming exists in older tests — refactor to backtick style when touching those tests +- **Independence**: Each test creates its own resources — never share state between tests +- **Bucket names**: Always use `UUID.randomUUID()` for unique bucket names in integration tests +- **Arrange-Act-Assert**: Follow this pattern consistently +- **Both paths**: Test success cases AND error/exception cases +- **SDK version**: Use AWS SDK v2 (`s3Client`) only — SDK v1 has been removed +- **No JUnit 4**: Use JUnit 5 exclusively (`@Test` from `org.junit.jupiter.api`) +- **No MockitoExtension**: Use `@SpringBootTest` with `@MockitoBean` for mocking — never `@ExtendWith(MockitoExtension::class)` +- **Mocking**: Use `@MockitoBean` (class-level `types` or field-level) instead of `@Mock` / `@InjectMocks` +- **Injection**: Use `@Autowired` for the class under test in Spring Boot tests + +## Running Tests + +```bash +# Unit tests (server module) +./mvnw test -pl server + +# All integration tests +./mvnw verify -pl integration-tests + +# Specific integration test class +./mvnw verify -pl integration-tests -Dit.test=BucketIT + +# Specific test method +./mvnw verify -pl integration-tests -Dit.test=BucketIT#shouldCreateBucket + +# Skip Docker (for unit tests only) +./mvnw test -pl server -DskipDocker +``` + +## Post-Flight Checklist + +- [ ] Tests pass locally (`./mvnw test` or `./mvnw verify`) +- [ ] Tests are independent (can run in any order) +- [ ] Both success and failure cases covered +- [ ] Assertions are specific (not just `isNotNull()`) +- [ ] No hardcoded bucket names (use UUID) +- [ ] Code style passes (`./mvnw ktlint:format`) + +## Troubleshooting Failing Tests + +- **Docker not running**: Integration tests require Docker — start Docker Desktop +- **Port conflict**: Ports 9090/9191 may be in use — check with `lsof -i :9090` +- **Flaky test**: Ensure test independence — check for shared state or ordering dependencies +- **Compilation error**: Run `./mvnw clean install -DskipDocker -DskipTests` first ## Output -Provide complete, runnable tests following project conventions. +Provide complete, runnable Kotlin tests following S3Mock conventions and the Arrange-Act-Assert pattern. diff --git a/AGENTS.md b/AGENTS.md index c19bd092f..9b9eef663 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,9 @@ Lightweight S3 API mock server for local integration testing. ``` server/ # Core implementation (Controller→Service→Store) integration-tests/ # AWS SDK integration tests -testsupport/ # JUnit 4/5, Testcontainers, TestNG integrations +testsupport/ # JUnit 5, Testcontainers, TestNG integrations +build-config/ # Shared build configuration +docker/ # Docker image build ``` ## Architecture @@ -20,6 +22,38 @@ testsupport/ # JUnit 4/5, Testcontainers, TestNG integrations **Key packages**: `controller/`, `service/`, `store/`, `dto/` +## DO / DON'T + +### DO +- Use **constructor injection** for all Spring beans (in production code) +- Use **data classes** for DTOs with Jackson XML annotations +- Use **Kotlin stdlib** and built-in language features over third-party utilities +- Use **AWS SDK v2** for all new integration tests +- Use **JUnit 5** for all new tests +- Use **`@SpringBootTest`** with **`@MockitoBean`** for unit tests — this is the project's standard mocking approach +- Use **expression bodies** for simple functions +- Use **null safety** (`?`, `?.`, `?:`) instead of null checks +- Match **AWS S3 API naming exactly** in Jackson XML annotations (`localName = "..."`) +- Keep tests **independent** — each test creates its own resources (UUID bucket names) +- Use **backtick test names** with descriptive sentences: `` fun `should create bucket successfully`() `` +- **Refactor** legacy `testSomething` camelCase names to backtick style when touching existing tests +- Validate XML serialization against [AWS S3 API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) + +### DON'T +- DON'T use `@Autowired` or field injection in production code — always use constructor injection +- DON'T use `var` for public API properties — prefer `val` (immutability) +- DON'T use AWS SDK v1 — it has been removed in 5.x +- DON'T use JUnit 4 — it has been removed in 5.x +- DON'T use `@ExtendWith(MockitoExtension::class)` or `@Mock` / `@InjectMocks` — use `@SpringBootTest` with `@MockitoBean` instead +- DON'T add Apache Commons dependencies — use Kotlin stdlib equivalents +- DON'T put business logic in controllers — controllers only map HTTP, delegate to services +- DON'T return raw strings from controllers — use typed DTOs for XML/JSON responses +- DON'T declare dependency versions in sub-module POMs — all versions are managed in root `pom.xml` +- DON'T share mutable state between tests — each test must be self-contained +- DON'T hardcode bucket names in tests — use `UUID.randomUUID()` for uniqueness +- DON'T use legacy `testSomething` camelCase naming for new tests — use backtick names instead +- DON'T add new functionality to deprecated modules (`junit4/`) + ## Code Style **Kotlin idioms**: Data classes for DTOs, null safety, expression bodies, constructor injection @@ -45,6 +79,10 @@ Jackson XML with AWS-compatible structure. Key annotations: - `@JacksonXmlProperty(localName = "...")` - `@JacksonXmlElementWrapper(useWrapping = false)` for collections +**Important**: XML element and attribute names must match the AWS S3 API specification exactly. +Verify against [AWS API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) +and existing integration tests. + ## Storage Filesystem layout: @@ -73,7 +111,7 @@ HTTP codes: 200, 204, 404, 409, 500 ## Testing - Unit tests: Mock dependencies, test in isolation, suffix `Test` -- Integration tests: Real AWS SDKs v1/v2, suffix `IT` +- Integration tests: Real AWS SDK v2, suffix `IT` - Test independence: Each test self-contained ## Build @@ -85,6 +123,36 @@ HTTP codes: 200, 204, 404, 409, 500 ./mvnw ktlint:format ``` +## CI/CD Pipeline + +All PRs and pushes are validated by the `maven-ci-and-prb.yml` GitHub Actions workflow. + +**Required gates** (all must pass before merge): +1. Compilation and build (`./mvnw clean install`) +2. Unit tests (`*Test.kt` in each module) +3. Integration tests (`*IT.kt` against Docker container) +4. ktlint (Kotlin code style) +5. Checkstyle (Java/XML code style, config in `etc/checkstyle.xml`) +6. Docker image build (unless `-DskipDocker`) + +**Additional workflows**: CodeQL (security scanning), SBOM (dependency tracking), OpenSSF Scorecard, Dependabot (automated dependency updates), Stale issue management. + +## Dependency Management + +- **All versions** are declared in the root `pom.xml` `` section +- Sub-modules inherit versions — never declare versions in sub-module POMs +- **BOMs** are preferred for multi-artifact dependencies (Kotlin BOM, Spring Boot BOM, AWS SDK BOM) +- Prefer Kotlin stdlib / JDK APIs over adding new third-party libraries +- Dependabot manages automated version updates for Maven, Docker, and GitHub Actions + +## PR & Commit Conventions + +- PRs should target `main` (active development) or version maintenance branches +- Reference related GitHub issues in PR description +- Update `CHANGELOG.md` under the current version section for user-facing changes +- Ensure all CI gates pass before requesting review +- See [PR template](.github/PULL_REQUEST_TEMPLATE.md) and [Contributing Guide](.github/CONTRIBUTING.md) + ## Constraints - Path-style URLs only (not `bucket.localhost`) diff --git a/server/AGENTS.md b/server/AGENTS.md index 64dcd06fc..49c784ae2 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -71,19 +71,19 @@ Responsibilities: HTTP mapping, headers, streaming responses ## Testing -Unit tests with Mockito: +Spring Boot tests with `@MockitoBean`: ```kotlin -@ExtendWith(MockitoExtension::class) -class ObjectServiceTest { - @Mock lateinit var bucketStore: BucketStore - @Mock lateinit var objectStore: ObjectStore - @InjectMocks lateinit var objectService: ObjectService +@SpringBootTest(classes = [ServiceConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE) +@MockitoBean(types = [BucketService::class, MultipartService::class, MultipartStore::class]) +internal class ObjectServiceTest : ServiceTestBase() { + @Autowired + private lateinit var iut: ObjectService @Test fun `should get object`() { whenever(bucketStore.getBucketMetadata("bucket")).thenReturn(bucket) whenever(objectStore.getObject(bucket, "key")).thenReturn(s3Object) - assertThat(objectService.getObject("bucket", "key")).isEqualTo(s3Object) + assertThat(iut.getObject("bucket", "key")).isEqualTo(s3Object) } } ``` From dd6c8f68a61fe64ac8b27508a5cfeea2ee743db0 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:47:16 +0100 Subject: [PATCH 13/36] feat: sub-module AGENTS.md guardrails --- integration-tests/AGENTS.md | 26 +++++++++++++++++++++----- server/AGENTS.md | 16 ++++++++++++++++ testsupport/AGENTS.md | 36 ++++++++++++++++++++++-------------- 3 files changed, 59 insertions(+), 19 deletions(-) diff --git a/integration-tests/AGENTS.md b/integration-tests/AGENTS.md index c78bcc5d7..ce0f0b227 100644 --- a/integration-tests/AGENTS.md +++ b/integration-tests/AGENTS.md @@ -1,12 +1,12 @@ # Agent Context for S3Mock Integration Tests -Integration tests verifying S3Mock with real AWS SDK clients (v1 and v2). +Integration tests verifying S3Mock with real AWS SDK v2 clients. ## Structure ``` integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/ -├── S3TestBase.kt # Base class with s3Client (v2), s3ClientV1 +├── S3TestBase.kt # Base class with s3Client (v2) ├── BucketIT.kt # Bucket operations ├── ObjectIT.kt # Object operations ├── MultipartUploadIT.kt # Multipart uploads @@ -16,10 +16,26 @@ integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/ ## Base Class Extend `S3TestBase` for access to: -- `s3Client` - AWS SDK v2 (default) -- `s3ClientV1` - AWS SDK v1 +- `s3Client` - AWS SDK v2 - `serviceEndpoint`, `serviceEndpointHttp`, `serviceEndpointHttps` +## DO / DON'T + +### DO +- Extend `S3TestBase` for all integration tests +- Use **backtick test names**: `` fun `should create bucket and upload object`(testInfo: TestInfo) `` +- Use **unique bucket names** via `givenBucket(testInfo)` or `UUID.randomUUID()` +- Use **Arrange-Act-Assert** pattern consistently +- Accept `testInfo: TestInfo` parameter in test methods for unique resource naming +- Verify HTTP codes, headers (ETag, Content-Type), XML/JSON bodies, and error responses +- Refactor legacy `testSomething` camelCase names to backtick style when touching tests + +### DON'T +- DON'T use AWS SDK v1 — it has been removed in 5.x +- DON'T share state between tests — each test must be self-contained +- DON'T hardcode bucket names — use `UUID.randomUUID()` for uniqueness +- DON'T mock AWS SDK clients — use actual SDK clients against S3Mock + ## Test Pattern Arrange-Act-Assert with unique bucket names (`UUID.randomUUID()`): @@ -96,7 +112,7 @@ assertThat(buckets.buckets()).anyMatch { it.name() == bucketName } 1. Independent tests (no shared state) 2. Unique bucket names with UUID -3. Test both SDK v1 (`s3ClientV1`) and v2 (`s3Client`) when applicable +3. Use AWS SDK v2 (`s3Client`) — SDK v1 has been removed 4. Verify: HTTP codes, headers (ETag, Content-Type), XML/JSON bodies, error responses 5. Use actual AWS SDK clients (not mocks) diff --git a/server/AGENTS.md b/server/AGENTS.md index 49c784ae2..5d195b0ba 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -69,6 +69,22 @@ class ObjectController(private val objectService: ObjectService) { Responsibilities: HTTP mapping, headers, streaming responses +## DO / DON'T + +### DO +- Follow the **DTO → Store → Service → Controller** flow when adding new S3 operations +- Add **Jackson XML annotations** matching the AWS API naming exactly (verify against [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)) +- Use **`@SpringBootTest`** with **`@MockitoBean`** for all tests +- Use **backtick test names**: `` fun `should return object with correct etag`() `` +- Throw **S3 exceptions** (`NoSuchBucketException`, etc.) from the Service layer + +### DON'T +- DON'T put business logic in controllers — controllers only map HTTP requests and delegate to services +- DON'T use `@Autowired` in production code — use constructor injection +- DON'T return raw strings — use typed DTOs for XML/JSON responses +- DON'T use `@ExtendWith(MockitoExtension::class)`, `@Mock`, or `@InjectMocks` — use `@SpringBootTest` with `@MockitoBean` +- DON'T use legacy `testSomething` naming — refactor to backtick style when touching tests + ## Testing Spring Boot tests with `@MockitoBean`: diff --git a/testsupport/AGENTS.md b/testsupport/AGENTS.md index 5dea9217b..7ad6cc948 100644 --- a/testsupport/AGENTS.md +++ b/testsupport/AGENTS.md @@ -1,15 +1,32 @@ # Agent Context for S3Mock Test Support -Test framework integrations for using S3Mock: JUnit 4/5, Testcontainers, TestNG. +Test framework integrations for using S3Mock: JUnit 5, Testcontainers, TestNG. + +> **Deprecation Notice (6.x):** The JUnit 5, TestNG, and all direct-integration modules will be +> removed in S3Mock 6.x. Testcontainers will become the only officially supported testing approach. +> See [CHANGELOG.md](../CHANGELOG.md) for details. ## Framework Selection -- **JUnit 5** (`junit5/`) - Modern extension with parameter injection. Default for new projects. -- **JUnit 4** (`junit4/`) - Legacy Rule API for older projects. -- **Testcontainers** (`testcontainers/`) - Docker container isolation, works with any framework. -- **TestNG** (`testng/`) - Listener-based integration for TestNG projects. +- **Testcontainers** (`testcontainers/`) - Docker container isolation, works with any framework. **Recommended.** +- **JUnit 5** (`junit5/`) - Extension with parameter injection. Will be removed in 6.x. +- **TestNG** (`testng/`) - Listener-based integration. Will be removed in 6.x. - **Common** (`common/`) - Shared `S3MockStarter` base class and utilities. +> **Note:** JUnit 4 support (`junit4/`) was removed in 5.x. + +## DO / DON'T + +### DO +- Prefer **Testcontainers** for new test setups — it provides the best isolation +- Keep framework modules **thin** — delegate to `common/` for shared logic +- Maintain backward compatibility for existing public APIs + +### DON'T +- DON'T add new features to `junit4/` — it has been removed in 5.x +- DON'T invest heavily in `junit5/` or `testng/` — they will be removed in 6.x +- DON'T add framework-specific logic that belongs in `common/` + ## JUnit 5 Extension ```kotlin @@ -31,15 +48,6 @@ val s3Mock = S3MockExtension.builder() .build() ``` -## JUnit 4 Rule - -```java -@Rule -public S3MockRule s3MockRule = S3MockRule.builder() - .withInitialBuckets("test-bucket") - .build(); -``` - ## Testcontainers ```kotlin From dad56021c84c1887f8391cf77e6b04ef4a60c9a1 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:55:09 +0100 Subject: [PATCH 14/36] feat: version compatibility, migrations, architecture diagram --- README.md | 47 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 781cde780..fd63b36fb 100755 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ * [S3Mock](#s3mock) * [Quick Start](#quick-start) * [Changelog](#changelog) + * [Version Compatibility](#version-compatibility) + * [Migration Guides](#migration-guides) + * [4.x to 5.x (Current)](#4x-to-5x-current) + * [3.x to 4.x](#3x-to-4x) * [Supported S3 Operations](#supported-s3-operations) * [Usage](#usage) * [Docker (Recommended)](#docker-recommended) @@ -61,6 +65,33 @@ For programmatic testing, see [Testcontainers](#testcontainers) or [JUnit 5 Exte - [GitHub Releases](https://github.com/adobe/S3Mock/releases) - [Detailed Changelog](CHANGELOG.md) +## Version Compatibility + +| S3Mock | Status | Spring Boot | Kotlin | Java (target) | Java (compile) | AWS SDK v2 | Testcontainers | +|--------|-------------|-------------|---------|---------------|----------------|------------|----------------| +| 5.x | **Active** | 4.0.x | 2.3 | 17 | 25 | 2.x | 2.x | +| 4.x | Deprecated | 3.x | 2.1-2.2 | 17 | 17 | 2.x | 1.x | +| 3.x | Deprecated | 2.x | 1.x-2.0 | 17 | 17 | 2.x | 1.x | +| 2.x | End of Life | 2.x | - | 11 | 11 | 1.x/2.x | - | + +## Migration Guides + +### 4.x to 5.x (Current) +- **Jackson 3**: XML annotations updated to Jackson 3 (`tools.jackson` packages) +- **AWS SDK v1 removed**: All v1 client support has been dropped +- **JUnit 4 removed**: The `s3mock-junit4` module no longer exists +- **Controller package moved**: `com.adobe.testing.s3mock` to `com.adobe.testing.s3mock.controller` +- **Legacy properties removed**: Old-style configuration properties have been removed +- **Apache Commons removed**: `commons-compress`, `commons-codec`, `commons-lang3` replaced by Kotlin/Java stdlib +- **Owner DisplayName removed**: AWS APIs stopped returning `DisplayName` - this is a file system breaking change for existing data + +### 3.x to 4.x +- **Tomcat replaces Jetty**: Application container changed from Jetty to Tomcat +- **Versioning API**: Basic support for S3 versioning added +- **If-(Un)modified-Since**: Conditional request handling implemented + +For full details, see the [Changelog](CHANGELOG.md). + ## Supported S3 Operations See the [complete operations table](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) in AWS documentation. @@ -417,11 +448,21 @@ S3Mock stores data on disk with the following structure: ## Architecture & Development +```mermaid +graph LR + Client["AWS SDK / CLI / cURL"] -->|HTTP :9090 / HTTPS :9191| Controller + subgraph S3Mock + Controller["Controller
(REST endpoints)"] -->|delegates| Service["Service
(business logic)"] + Service -->|coordinates| Store["Store
(persistence)"] + Store -->|read/write| FS["Filesystem"] + end +``` + **Module Documentation:** -- [Project Overview](AGENTS.md) - Architecture, code style, testing philosophy +- [Project Overview](AGENTS.md) - Architecture, code style, DO/DON'T guardrails - [Server Module](server/AGENTS.md) - Core implementation (Controller→Service→Store layers) -- [Integration Tests](integration-tests/AGENTS.md) - Testing with AWS SDK clients -- [Test Support](testsupport/AGENTS.md) - JUnit 4/5, Testcontainers, TestNG integrations +- [Integration Tests](integration-tests/AGENTS.md) - Testing with AWS SDK v2 clients +- [Test Support](testsupport/AGENTS.md) - Testcontainers, JUnit 5, TestNG integrations ## Build & Run From c94e4d2f5ce375b0476fd2e32650a9648a412a9a Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:55:44 +0100 Subject: [PATCH 15/36] feat: add CODEOWNERS Created with @afranken as default owner, plus specific paths for server, integration-tests, testsupport, docker, .github, build-config. --- .github/CODEOWNERS | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 000000000..f06a7ed77 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,20 @@ +# Default owner for everything in the repo +* @afranken + +# Core server implementation +/server/ @afranken + +# Integration tests +/integration-tests/ @afranken + +# Test support modules +/testsupport/ @afranken + +# Docker configuration +/docker/ @afranken + +# CI/CD and GitHub configuration +/.github/ @afranken + +# Build configuration +/build-config/ @afranken From d68f6e119498e4420fbf53540d5cb0c305e7e30c Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:56:22 +0100 Subject: [PATCH 16/36] feat: enhance CONTRIBUTING with clarifications and how-tos --- .github/CONTRIBUTING.md | 61 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 3 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 706fca46e..327ee7f34 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -12,11 +12,66 @@ This project adheres to the Adobe [code of conduct](../CODE_OF_CONDUCT.md). By p All third-party contributions to this project must be accompanied by a signed contributor license. This gives Adobe permission to redistribute your contributions as part of the project. Sign our [CLA](http://adobe.github.io/cla.html). You only need to submit an Adobe CLA one time, so if you have submitted one previously, you are good to go! +## Development Setup + +**Prerequisites:** +- Java 25 (compile version; targets JVM 17) +- Maven 3.9+ (use the included `./mvnw` wrapper) +- Docker (for Docker build and integration tests) + +**Build and verify:** +```shell +# Full build with Docker +./mvnw clean install + +# Skip Docker (faster, for unit tests only) +./mvnw clean install -DskipDocker + +# Run integration tests +./mvnw verify -pl integration-tests + +# Format Kotlin code +./mvnw ktlint:format +``` + +## Architecture + +S3Mock follows a **Controller - Service - Store** layered architecture. For detailed architecture documentation, code style guidelines, and project conventions, see the [AGENTS.md](../AGENTS.md) in the project root. + +Module-specific documentation: +- [Server Module](../server/AGENTS.md) - core implementation +- [Integration Tests](../integration-tests/AGENTS.md) - test patterns +- [Test Support](../testsupport/AGENTS.md) - framework integrations + +## Code Style + +- **Kotlin**: Enforced by ktlint - run `./mvnw ktlint:format` before submitting +- **XML/Java**: Enforced by Checkstyle - configuration in [`etc/checkstyle.xml`](../etc/checkstyle.xml) +- **Key conventions**: Constructor injection, data classes for DTOs, backtick test names, `val` over `var` +- See the DO / DON'T section in [AGENTS.md](../AGENTS.md) for the full list + ## Code Reviews All submissions should come in the form of pull requests and need to be reviewed by project committers. Read [GitHub's pull request documentation](https://help.github.com/articles/about-pull-requests/) for more information on sending pull requests. -All submissions must include unit tests for any new functionality or bug fixes. If you are adding a new feature, please include a test that demonstrates the feature. -S3Mock uses Unit tests for function coverage, Spring Boot tests for component coverage, and integration tests against the Docker container artefact for end-to-end coverage. Please ensure that your code is covered by at least one of these test types. +## Testing + +All submissions must include tests for any new functionality or bug fixes. + +S3Mock uses three test levels: +1. **Unit tests** (`*Test.kt`) - Spring Boot tests with `@MockitoBean` for mocking, in `server/src/test/` +2. **Spring Boot tests** - component-level coverage with Spring context +3. **Integration tests** (`*IT.kt`) - end-to-end tests against the Docker container using real AWS SDK v2 clients, in `integration-tests/src/test/` + +Please ensure that your code is covered by at least one of these test types. + +## Submitting Changes + +1. Ensure all CI gates pass (build, tests, ktlint, checkstyle, Docker) +2. Update [CHANGELOG.md](../CHANGELOG.md) under the current version section for user-facing changes +3. Update documentation if applicable (README.md, AGENTS.md) +4. Follow the [pull request template](PULL_REQUEST_TEMPLATE.md) when submitting + +## Security -Lastly, please follow the [pull request template](PULL_REQUEST_TEMPLATE.md) when submitting a pull request! +To report security vulnerabilities, see the [Security Policy](SECURITY.md). From 74897a908c5617ee24da779a1ab091ef02931d72 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:57:20 +0100 Subject: [PATCH 17/36] feat: add Github issue forms to help triage issues --- .github/ISSUE_TEMPLATE.md | 0 .github/ISSUE_TEMPLATE/bug_report.yml | 87 ++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 8 ++ .github/ISSUE_TEMPLATE/feature_request.yml | 62 +++++++++++++++ 4 files changed, 157 insertions(+) delete mode 100644 .github/ISSUE_TEMPLATE.md create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 000000000..f8b3b47af --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,87 @@ +name: Bug Report +description: Report a bug or unexpected behavior in S3Mock +title: "[Bug]: " +labels: ["bug"] +assignees: + - afranken +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to report a bug! Please fill out the information below to help us investigate. + - type: input + id: version + attributes: + label: S3Mock Version + description: Which version of S3Mock are you using? + placeholder: "e.g., 5.0.0" + validations: + required: true + - type: dropdown + id: usage-mode + attributes: + label: Usage Mode + description: How are you running S3Mock? + options: + - Docker + - Testcontainers + - JUnit 5 Extension + - TestNG Listener + - Other + validations: + required: true + - type: textarea + id: description + attributes: + label: Description + description: A clear description of the bug. + placeholder: Describe what happened and what you expected to happen. + validations: + required: true + - type: textarea + id: steps + attributes: + label: Steps to Reproduce + description: How can we reproduce this issue? + placeholder: | + 1. Start S3Mock with ... + 2. Call API ... + 3. Observe ... + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What did you expect to happen? + validations: + required: true + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened? + validations: + required: true + - type: dropdown + id: sdk + attributes: + label: AWS SDK + description: Which AWS SDK are you using? + options: + - AWS SDK v2 (Java/Kotlin) + - AWS SDK v2 (Python/boto3) + - AWS SDK v2 (Other language) + - AWS CLI + - Other / Not applicable + - type: textarea + id: logs + attributes: + label: Relevant Log Output + description: Paste any relevant log output or error messages. + render: shell + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other context, screenshots, or configuration details. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 000000000..e9486e756 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: true +contact_links: + - name: S3Mock Documentation + url: https://github.com/adobe/S3Mock#readme + about: Check the README for usage instructions, configuration, and troubleshooting. + - name: AWS S3 API Reference + url: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html + about: Reference for S3 API operations and expected behavior. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 000000000..0390f3095 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,62 @@ +name: Feature Request +description: Suggest a new feature or enhancement for S3Mock +title: "[Feature]: " +labels: ["enhancement"] +assignees: + - afranken +body: + - type: markdown + attributes: + value: | + Thanks for suggesting an improvement! Please describe the feature you'd like to see. + - type: dropdown + id: category + attributes: + label: Feature Category + description: What area of S3Mock does this relate to? + options: + - New S3 API operation + - Improve existing S3 API operation + - Configuration / Setup + - Testcontainers integration + - JUnit 5 Extension + - Docker image + - Documentation + - Other + validations: + required: true + - type: input + id: s3-operation + attributes: + label: S3 API Operation + description: If this is about an S3 API operation, which one? + placeholder: "e.g., GetBucketEncryption, PutBucketTagging" + - type: textarea + id: description + attributes: + label: Description + description: A clear description of the feature you'd like. + validations: + required: true + - type: textarea + id: use-case + attributes: + label: Use Case + description: Describe the use case or problem this feature would solve. + validations: + required: true + - type: textarea + id: proposed-solution + attributes: + label: Proposed Solution + description: If you have ideas on how this could be implemented, describe them here. + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Any alternative solutions or workarounds you've considered. + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other context, references, or screenshots. From 832c4543bd3be5aa1472674d47941ab7b83feee1 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:57:59 +0100 Subject: [PATCH 18/36] feat: add Github issue forms to help triage issues --- .github/dependabot.yml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 730288265..c60aceaae 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -17,6 +17,22 @@ updates: # Check for updates once per day, Github defaults to random time every day. schedule: interval: daily + # Group related dependencies to reduce PR noise + groups: + aws-sdk: + patterns: + - "software.amazon.awssdk*" + - "aws.sdk.kotlin*" + spring: + patterns: + - "org.springframework*" + maven-plugins: + patterns: + - "org.apache.maven.plugins*" + kotlin: + patterns: + - "org.jetbrains.kotlin*" + - "org.jetbrains.kotlinx*" # Enable version updates for Docker - package-ecosystem: docker # Look for `Dockerfile` in the `/docker/` directory, that's where all versions are managed @@ -41,3 +57,10 @@ updates: # Check for updates once per day, Github defaults to random time every day. schedule: interval: daily + # Group related actions to reduce PR noise + groups: + github-actions: + patterns: + - "actions/*" + - "github/*" + - "step-security/*" From a9d78a1ff0b34c4474ddb9c9f70b25e0d590c681 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 13:58:40 +0100 Subject: [PATCH 19/36] feat: update tasks in PR template --- .github/PULL_REQUEST_TEMPLATE.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1f3f65ade..ef574418d 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -9,3 +9,6 @@ - [ ] I have signed the [CLA](http://adobe.github.io/cla.html). - [ ] I have written tests and verified that they fail without my change. +- [ ] I have run `./mvnw ktlint:format` to fix code style. +- [ ] I have updated `CHANGELOG.md` (if applicable). +- [ ] I have updated documentation (if applicable). From fb3756482081344f4f190c527adc614283064a1c Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:23:48 +0100 Subject: [PATCH 20/36] feat: add version and branch strategy --- AGENTS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 9b9eef663..8a23490b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -153,6 +153,13 @@ All PRs and pushes are validated by the `maven-ci-and-prb.yml` GitHub Actions wo - Ensure all CI gates pass before requesting review - See [PR template](.github/PULL_REQUEST_TEMPLATE.md) and [Contributing Guide](.github/CONTRIBUTING.md) +## Version & Branch Strategy + +- **`main`** — active development for the current major version (5.x) +- **Version branches** (`s3mock-v2`, `s3mock-v3`, `s3mock-v4`) — maintenance for previous major versions +- **Tags** follow semver: `5.0.0`, `4.11.0`, etc. +- **6.x** is planned after Spring Boot 5.x — will remove JUnit/TestNG modules and target JDK 25 LTS bytecode + ## Constraints - Path-style URLs only (not `bucket.localhost`) From 8b7751f5accd268704369d7d9a3d454852f8329d Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:26:54 +0100 Subject: [PATCH 21/36] feat: add performance and resources section --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index fd63b36fb..25ab41f39 100755 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ * [Important Limitations](#important-limitations) * [Troubleshooting](#troubleshooting) * [File System Structure](#file-system-structure) + * [Performance & Resources](#performance--resources) * [Architecture & Development](#architecture--development) * [Build & Run](#build--run) * [Contributing](#contributing) @@ -446,6 +447,15 @@ S3Mock stores data on disk with the following structure: **Note:** The file system structure is an implementation detail and may change between releases. While files can be inspected during runtime, reusing persisted data across restarts is not officially supported. +## Performance & Resources + +S3Mock is designed for testing, not production workloads. Keep the following in mind: + +- **Disk**: All objects are stored on the local filesystem — disk usage grows with stored data +- **Memory**: Scales with concurrent multipart uploads and in-flight requests +- **CI environments**: Consider setting Docker resource limits (e.g., `--memory=512m`) to avoid contention with other services +- **Cleanup**: By default, S3Mock deletes all stored data on shutdown. Use `RETAIN_FILES_ON_EXIT=true` only when needed + ## Architecture & Development ```mermaid From 935f85fdcd166ab3f386cd2ab6f1eff57cc92702 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:28:50 +0100 Subject: [PATCH 22/36] feat: let users know to prefer testcontainers over TestNG / JUnit5 --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 25ab41f39..d5fcc213e 100755 --- a/README.md +++ b/README.md @@ -289,6 +289,8 @@ class MyTest { ### JUnit 5 Extension +> **Note:** This module may be removed in S3Mock 6.x. Consider using [Testcontainers](#testcontainers) instead. + **Maven dependency:** ```xml @@ -314,6 +316,8 @@ See examples: [Declarative](testsupport/junit5/src/test/kotlin/com/adobe/testing ### TestNG Listener +> **Note:** This module may be removed in S3Mock 6.x. Consider using [Testcontainers](#testcontainers) instead. + **Maven dependency:** ```xml From 3487e2a701c7b42dc93dda1cc771ca65181f04b4 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:30:22 +0100 Subject: [PATCH 23/36] feat: add troubleshooting section to SKILL.md --- .claude/skills/implement/SKILL.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.claude/skills/implement/SKILL.md b/.claude/skills/implement/SKILL.md index 234f2341b..295f67c89 100644 --- a/.claude/skills/implement/SKILL.md +++ b/.claude/skills/implement/SKILL.md @@ -68,6 +68,13 @@ Follow the **DTO → Store → Service → Controller** layered architecture: - [ ] Update `CHANGELOG.md` under the current version section - [ ] Update the operations table in `README.md` if a new S3 operation was added +## Troubleshooting + +- **Build fails**: Check Java version (`java -version` — needs 25), run `./mvnw ktlint:format` +- **Checkstyle fails**: Review rules in `etc/checkstyle.xml` — common issues are import ordering and missing Javadoc +- **Tests fail after changes**: Ensure XML serialization matches AWS API exactly — compare element names against [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) +- **Docker build fails**: Try `./mvnw clean install -DskipDocker` first to isolate the issue + ## Output Provide clean, well-structured Kotlin code following the layered architecture and project conventions defined in AGENTS.md. From b1d0aee151657fe998b6c45481115db4541f4891 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:31:19 +0100 Subject: [PATCH 24/36] feat: clarify verification process for AWS S3 XML output in AGENTS.md --- AGENTS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 8a23490b3..f1a96fd9d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -81,7 +81,8 @@ Jackson XML with AWS-compatible structure. Key annotations: **Important**: XML element and attribute names must match the AWS S3 API specification exactly. Verify against [AWS API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) -and existing integration tests. +and existing integration tests. A compile-check is not sufficient — always verify that serialized +XML output matches the expected AWS response format by running integration tests. ## Storage From 5ccae8b18663ee3aad2ccb0eb136dd387d9bf3f2 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:32:12 +0100 Subject: [PATCH 25/36] feat: update badge from Java to JVM in README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d5fcc213e..c5a3328ee 100755 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/) [![Docker Pulls](https://img.shields.io/docker/pulls/adobe/s3mock)](https://hub.docker.com/r/adobe/s3mock) [![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#build--run) -[![Java](https://img.shields.io/badge/MADE%20with-Java-RED.svg)](#build--run) +[![JVM](https://img.shields.io/badge/runs%20on-JVM-RED.svg)](#build--run) [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7673/badge)](https://bestpractices.coreinfrastructure.org/projects/7673) [![OpenSSF Scorecard](https://img.shields.io/ossf-scorecard/github.com/adobe/S3Mock?label=openssf%20scorecard&style=flat)](https://api.securityscorecards.dev/projects/github.com/adobe/S3Mock) [![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/) From 5e21dc6471405e0ca0badd6713090dddb3003575 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:33:09 +0100 Subject: [PATCH 26/36] feat: mark test classes as internal in AGENTS.md --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index f1a96fd9d..6d4c4bb8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,6 +36,7 @@ docker/ # Docker image build - Match **AWS S3 API naming exactly** in Jackson XML annotations (`localName = "..."`) - Keep tests **independent** — each test creates its own resources (UUID bucket names) - Use **backtick test names** with descriptive sentences: `` fun `should create bucket successfully`() `` +- Mark test classes as **`internal`**: `internal class ObjectServiceTest` - **Refactor** legacy `testSomething` camelCase names to backtick style when touching existing tests - Validate XML serialization against [AWS S3 API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) From b69e62b78b8fd48317e1e2837555ea70e6a2b586 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:36:28 +0100 Subject: [PATCH 27/36] feat: update testing guidelines in AGENTS.md --- AGENTS.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6d4c4bb8e..e86aa2fa0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -112,9 +112,15 @@ HTTP codes: 200, 204, 404, 409, 500 ## Testing -- Unit tests: Mock dependencies, test in isolation, suffix `Test` -- Integration tests: Real AWS SDK v2, suffix `IT` +- Unit tests: `@SpringBootTest` with `@MockitoBean`, suffix `Test` +- Integration tests: Real AWS SDK v2 against Docker container, suffix `IT` - Test independence: Each test self-contained +- Name the class under test **`iut`** (implementation under test): `private lateinit var iut: ObjectService` +- **Base classes** — always extend the appropriate base: + - `ServiceTestBase` for service tests + - `StoreTestBase` for store tests + - `BaseControllerTest` for controller tests + - `S3TestBase` for integration tests ## Build From 9a9d8b36616e91eb26f6c3b03f692341a31a39ae Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:48:44 +0100 Subject: [PATCH 28/36] feat: add guideline to name 'it' parameter in nested lambdas --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index e86aa2fa0..df0cdba64 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,7 @@ docker/ # Docker image build - Use **`@SpringBootTest`** with **`@MockitoBean`** for unit tests — this is the project's standard mocking approach - Use **expression bodies** for simple functions - Use **null safety** (`?`, `?.`, `?:`) instead of null checks +- **Name the `it` parameter** in nested lambdas, loops, and scope functions to avoid shadowing: `.map { part -> ... }` instead of `.map { it.name }` - Match **AWS S3 API naming exactly** in Jackson XML annotations (`localName = "..."`) - Keep tests **independent** — each test creates its own resources (UUID bucket names) - Use **backtick test names** with descriptive sentences: `` fun `should create bucket successfully`() `` From 4f1974402052f2bf12c4be56ac93dd2a8db6ad05 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:50:38 +0100 Subject: [PATCH 29/36] feat: add error handling guidelines for S3 exceptions in AGENTS.md --- server/AGENTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/server/AGENTS.md b/server/AGENTS.md index 5d195b0ba..738fba2fb 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -69,6 +69,14 @@ class ObjectController(private val objectService: ObjectService) { Responsibilities: HTTP mapping, headers, streaming responses +## Error Handling + +- Services throw **`S3Exception`** constants (e.g., `S3Exception.NO_SUCH_KEY`, `S3Exception.INVALID_BUCKET_NAME`) +- Controllers never catch exceptions — Spring's `@ExceptionHandler` in `ControllerConfiguration` handles them +- `S3MockExceptionHandler` converts `S3Exception` → XML `ErrorResponse` with the correct HTTP status +- `IllegalStateExceptionHandler` converts unexpected errors → `500 InternalError` +- DON'T create new exception classes — add new constants to `S3Exception` + ## DO / DON'T ### DO From b1de2c9d2b515841d07a0af0f86da69864dd7767 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:52:48 +0100 Subject: [PATCH 30/36] feat: update S3 exception handling to use S3Exception constants --- AGENTS.md | 2 +- server/AGENTS.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df0cdba64..ee991d26d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,7 +107,7 @@ Environment variables (prefix: `COM_ADOBE_TESTING_S3MOCK_STORE_`): ## Error Handling -S3 exceptions: `NoSuchBucketException`, `NoSuchKeyException`, `BucketAlreadyExistsException` +`S3Exception` constants: `NO_SUCH_BUCKET`, `NO_SUCH_KEY`, `BUCKET_ALREADY_OWNED_BY_YOU`, `INVALID_BUCKET_NAME`, etc. HTTP codes: 200, 204, 404, 409, 500 diff --git a/server/AGENTS.md b/server/AGENTS.md index 738fba2fb..6a9b514a5 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -44,9 +44,9 @@ class ObjectService( ) { fun getObject(bucketName: String, key: String): S3Object { val bucket = bucketStore.getBucketMetadata(bucketName) - ?: throw NoSuchBucketException(bucketName) + ?: throw S3Exception.NO_SUCH_BUCKET return objectStore.getObject(bucket, key) - ?: throw NoSuchKeyException(key) + ?: throw S3Exception.NO_SUCH_KEY } } ``` @@ -84,7 +84,7 @@ Responsibilities: HTTP mapping, headers, streaming responses - Add **Jackson XML annotations** matching the AWS API naming exactly (verify against [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)) - Use **`@SpringBootTest`** with **`@MockitoBean`** for all tests - Use **backtick test names**: `` fun `should return object with correct etag`() `` -- Throw **S3 exceptions** (`NoSuchBucketException`, etc.) from the Service layer +- Throw **`S3Exception` constants** (e.g., `S3Exception.NO_SUCH_BUCKET`, `S3Exception.NO_SUCH_KEY`) from the Service layer ### DON'T - DON'T put business logic in controllers — controllers only map HTTP requests and delegate to services From 04593c34d31e74a42a0b2ddc7902b8ebcb7c7687 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 14:59:54 +0100 Subject: [PATCH 31/36] feat: update configuration section with default values and binding --- server/AGENTS.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/server/AGENTS.md b/server/AGENTS.md index 6a9b514a5..2f8117846 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -114,14 +114,23 @@ internal class ObjectServiceTest : ServiceTestBase() { ## Configuration +Three `@ConfigurationProperties` classes bind environment variables to typed properties: +- `StoreProperties` (`com.adobe.testing.s3mock.store.*`) — storage root, buckets, KMS, region +- `ControllerProperties` (`com.adobe.testing.s3mock.controller.*`) — context path +- `S3MockProperties` (`com.adobe.testing.s3mock.*`) — top-level settings + +Spring Boot relaxed binding maps properties to environment variables automatically: +`com.adobe.testing.s3mock.store.initial-buckets` → `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` + ```kotlin -@ConfigurationProperties(prefix = "com.adobe.testing.s3mock.store") +@JvmRecord +@ConfigurationProperties("com.adobe.testing.s3mock.store") data class StoreProperties( - val root: Path, - val retainFilesOnExit: Boolean = false, - val validKmsKeys: Set = emptySet(), - val initialBuckets: Set = emptySet(), - val region: Region = Region.US_EAST_1 + @param:DefaultValue("false") val retainFilesOnExit: Boolean, + @param:DefaultValue("") val root: String, + @param:DefaultValue("") val validKmsKeys: Set, + @param:DefaultValue("") val initialBuckets: List, + @param:DefaultValue("us-east-1") val region: Region ) ``` From 65e12aa71faf4c2bab67c22e45ee1776a57774d5 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 15:05:56 +0100 Subject: [PATCH 32/36] feat: instructions for unique bucket names and error handling guidelines --- integration-tests/AGENTS.md | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/integration-tests/AGENTS.md b/integration-tests/AGENTS.md index ce0f0b227..002947f63 100644 --- a/integration-tests/AGENTS.md +++ b/integration-tests/AGENTS.md @@ -38,14 +38,14 @@ Extend `S3TestBase` for access to: ## Test Pattern -Arrange-Act-Assert with unique bucket names (`UUID.randomUUID()`): +Arrange-Act-Assert with `testInfo` for unique bucket names: ```kotlin -class MyFeatureIT : S3TestBase() { +internal class MyFeatureIT : S3TestBase() { @Test - fun `should perform operation`() { + fun `should perform operation`(testInfo: TestInfo) { // Arrange - val bucketName = givenBucket() + val bucketName = givenBucket(testInfo) // Act s3Client.putObject( @@ -64,20 +64,15 @@ class MyFeatureIT : S3TestBase() { ## Common Patterns -**Helper**: -```kotlin -private fun givenBucket() = "test-bucket-${UUID.randomUUID()}".also { - s3Client.createBucket { it.bucket(it) } -} -``` +**Bucket creation**: Use `givenBucket(testInfo)` from `S3TestBase` — don't create your own helper. **Multipart**: Initiate → Upload parts → Complete → Verify -**Errors**: +**Errors** (use AssertJ, not JUnit `assertThrows`): ```kotlin -assertThrows { - s3Client.getObject(...) -} +assertThatThrownBy { s3Client.deleteBucket { it.bucket(bucketName) } } + .isInstanceOf(AwsServiceException::class.java) + .hasMessageContaining("Status Code: 409") ``` **Metadata**: From 8c4bb798f8b1269411f4ba49366fdc2e30b54996 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 15:08:45 +0100 Subject: [PATCH 33/36] feat: update ETag handling instructions --- AGENTS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ee991d26d..15f5da4b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -180,12 +180,12 @@ All PRs and pushes are validated by the `maven-ci-and-prb.yml` GitHub Actions wo ## Common Patterns ```kotlin -// ETag -val etag = DigestUtils.md5Hex(data) +// ETag (using project's own DigestUtil, not Apache Commons) +val etag = DigestUtil.hexDigest(data) // Response ResponseEntity.ok() - .header("ETag", "\"$etag\"") + .eTag(normalizeEtag(s3ObjectMetadata.etag)) .header("Last-Modified", lastModified) .body(data) From 372cb58b09693061013c20b058f92ee56b985f9b Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 15:21:45 +0100 Subject: [PATCH 34/36] feat: update test guidelines for unique bucket names and visibility --- .claude/skills/test/SKILL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md index d31c88d5d..1cb20c841 100644 --- a/.claude/skills/test/SKILL.md +++ b/.claude/skills/test/SKILL.md @@ -56,7 +56,7 @@ internal class ObjectServiceTest : ServiceTestBase() { - **Purpose**: Test S3Mock end-to-end with real AWS SDK clients against the Docker container - **Pattern**: ```kotlin -class MyFeatureIT : S3TestBase() { +internal class MyFeatureIT : S3TestBase() { @Test fun `should perform operation`(testInfo: TestInfo) { // Arrange — always use unique bucket names @@ -82,11 +82,13 @@ class MyFeatureIT : S3TestBase() { - **Naming**: Use backtick names with descriptive sentences: `` fun `should create bucket successfully`() `` - Legacy `testSomething` camelCase naming exists in older tests — refactor to backtick style when touching those tests - **Independence**: Each test creates its own resources — never share state between tests -- **Bucket names**: Always use `UUID.randomUUID()` for unique bucket names in integration tests +- **Bucket names**: Use `givenBucket(testInfo)` from `S3TestBase` in integration tests for unique names - **Arrange-Act-Assert**: Follow this pattern consistently - **Both paths**: Test success cases AND error/exception cases +- **Error assertions**: Use `assertThatThrownBy { ... }.isInstanceOf(AwsServiceException::class.java)` (AssertJ), not `assertThrows` - **SDK version**: Use AWS SDK v2 (`s3Client`) only — SDK v1 has been removed - **No JUnit 4**: Use JUnit 5 exclusively (`@Test` from `org.junit.jupiter.api`) +- **Visibility**: Mark test classes as `internal` - **No MockitoExtension**: Use `@SpringBootTest` with `@MockitoBean` for mocking — never `@ExtendWith(MockitoExtension::class)` - **Mocking**: Use `@MockitoBean` (class-level `types` or field-level) instead of `@Mock` / `@InjectMocks` - **Injection**: Use `@Autowired` for the class under test in Spring Boot tests From bc2a5f2743c242c7f5e426ea66752daac335abb4 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 15:26:27 +0100 Subject: [PATCH 35/36] feat: Add security and issue templates to the documentation skill --- .claude/skills/document/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.claude/skills/document/SKILL.md b/.claude/skills/document/SKILL.md index 06fc2bc00..e32dbac49 100644 --- a/.claude/skills/document/SKILL.md +++ b/.claude/skills/document/SKILL.md @@ -31,7 +31,10 @@ Generate and maintain documentation for the S3Mock project. | `server/AGENTS.md` | AI agents | Server module implementation details | | `integration-tests/AGENTS.md` | AI agents | Integration test patterns and helpers | | `testsupport/AGENTS.md` | AI agents | Test framework integration details | -| `.github/CONTRIBUTING.md` | Contributors | How to contribute, CLA, code reviews | +| `.github/CONTRIBUTING.md` | Contributors | How to contribute, dev setup, code reviews | +| `.github/SECURITY.md` | Security researchers | Vulnerability reporting, supported versions | +| `.github/CODEOWNERS` | GitHub | Automatic PR review assignment | +| `.github/ISSUE_TEMPLATE/*.yml` | Users & contributors | Structured bug reports and feature requests | ## Documentation Tasks From 29d656c33621271fc3a1c9dc4ce29ecd5d8c2de0 Mon Sep 17 00:00:00 2001 From: Arne Franken Date: Mon, 23 Feb 2026 15:29:34 +0100 Subject: [PATCH 36/36] feat: add detailed structure and component descriptions --- server/AGENTS.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/server/AGENTS.md b/server/AGENTS.md index 2f8117846..ac96740f6 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -7,12 +7,25 @@ Core S3Mock server implementation. ``` server/src/main/kotlin/com/adobe/testing/s3mock/ ├── S3MockApplication.kt # Spring Boot entry -├── S3MockConfiguration.kt # Config +├── S3MockConfiguration.kt # Top-level config ├── S3MockProperties.kt # Properties binding -├── controller/ # REST endpoints (BucketController, ObjectController) +├── controller/ +│ ├── *Controller.kt # REST endpoints +│ ├── ControllerConfiguration.kt # Controller beans + exception handlers +│ └── ControllerProperties.kt ├── dto/ # XML/JSON models (*Result, request/response) -├── service/ # Business logic (ObjectService, etc.) -└── store/ # Persistence (BucketStore, ObjectStore, metadata) +├── service/ +│ ├── *Service.kt # Business logic +│ └── ServiceConfiguration.kt # Service beans (used in @SpringBootTest) +├── store/ +│ ├── *Store.kt # Persistence +│ ├── StoreConfiguration.kt # Store beans (used in @SpringBootTest) +│ └── StoreProperties.kt +└── util/ + ├── DigestUtil.kt # ETag/checksum computation (replaces Apache Commons) + ├── EtagUtil.kt # ETag normalization + ├── HeaderUtil.kt # HTTP header helpers + └── AwsHttpHeaders.kt # AWS-specific header constants ``` ## Implementation Flow