Skip to content

Commit f2d6992

Browse files
afrankenafranken
authored
Merge pull request #2557 from adobe/junie-evaluation
junie-evaluation
2 parents c2206f7 + a2f2d16 commit f2d6992

File tree

14 files changed

+652
-83
lines changed

14 files changed

+652
-83
lines changed

‎.junie/guidelines.md‎

Lines changed: 360 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,360 @@
1+
# S3Mock Development Guidelines (Concise)
2+
3+
Essential info for working on S3Mock. This top section is a quick, no‑frills guide. Details follow below.
4+
5+
Quickstart TL;DR
6+
- Build (fast): ./mvnw clean install -DskipDocker
7+
- Build (full): ./mvnw clean install
8+
- Server tests only: ./mvnw -pl server test
9+
- All tests incl. ITs (requires Docker): ./mvnw verify
10+
- One server test: ./mvnw -pl server test -Dtest=ObjectStoreTest
11+
- One IT: ./mvnw -pl integration-tests -am verify -Dit.test=BucketIT
12+
13+
Requirements
14+
- JDK 17+
15+
- Docker (only for integration tests and Docker image build)
16+
17+
Common config (env vars)
18+
- COM_ADOBE_TESTING_S3MOCK_STORE_REGION (default: us-east-1)
19+
- COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS
20+
- COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT (default: false)
21+
- debug / trace (Spring Boot flags)
22+
23+
Tips
24+
- IDE runs: use server tests; ITs need Docker via Maven lifecycle.
25+
- Debug server on 9090/9191; trust self‑signed cert or use HTTP.
26+
- Before declaring done: run a full Maven build successfully.
27+
28+
Troubleshooting
29+
- Connection refused: ensure S3Mock is running on expected ports.
30+
- SSL errors: trust self‑signed cert or switch to HTTP.
31+
- Docker errors: ensure Docker is running and you have permissions.
32+
33+
—
34+
35+
## Build and Configuration Instructions
36+
37+
### Building the Project
38+
39+
S3Mock uses Maven for building. The project includes a Maven wrapper (`mvnw`) so you don't need to install Maven separately.
40+
41+
#### Basic Build Commands
42+
43+
```bash
44+
# Build the entire project
45+
./mvnw clean install
46+
47+
# Build without running Docker (useful for quick builds)
48+
./mvnw clean install -DskipDocker
49+
50+
# Build a specific module
51+
./mvnw clean install -pl server -am
52+
```
53+
54+
#### Build Requirements
55+
56+
- JDK 17 or higher
57+
- Docker (for building the Docker image and running integration tests)
58+
59+
### Configuration Options
60+
61+
S3Mock can be configured with the following environment variables:
62+
63+
| Environment Variable | Legacy Name | Description | Default |
64+
|-------------------------------------------------------|-----------------------------------|-------------------------------------------------------|---------------------|
65+
| `COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS` | `validKmsKeys` | List of KMS Key-Refs that are treated as valid | none |
66+
| `COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS` | `initialBuckets` | List of bucket names that will be available initially | none |
67+
| `COM_ADOBE_TESTING_S3MOCK_STORE_REGION` | `COM_ADOBE_TESTING_S3MOCK_REGION` | The region the S3Mock is supposed to mock | `us-east-1` |
68+
| `COM_ADOBE_TESTING_S3MOCK_STORE_ROOT` | `root` | Base directory for temporary files | Java temp directory |
69+
| `COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT` | `retainFilesOnExit` | Set to `true` to keep files after shutdown | `false` |
70+
| `debug` | - | Enable Spring Boot's debug output | `false` |
71+
| `trace` | - | Enable Spring Boot's trace output | `false` |
72+
73+
## Testing Information
74+
75+
### Test Structure
76+
77+
The project uses JUnit 5 for testing. There are two main types of tests in the project:
78+
79+
1. **Integration Tests**: Located in the `integration-tests` module and written in Kotlin. These tests verify the end-to-end functionality of S3Mock by starting a Docker container and making actual HTTP requests to it.
80+
81+
2. **Server Module Tests**: Located in the `server` module and primarily written in Kotlin. These include:
82+
- Unit tests for utility classes and DTOs
83+
- Spring Boot tests for controllers, services, and stores
84+
- XML serialization/deserialization tests
85+
86+
#### Integration Tests
87+
88+
The main test base class for integration tests is `S3TestBase` which provides utility methods for:
89+
- Creating S3 clients
90+
- Managing buckets and objects
91+
- Handling SSL certificates
92+
- Generating test data
93+
94+
#### Server Module Tests
95+
96+
The server module contains several types of tests:
97+
98+
1. **Controller Tests**: Use `@SpringBootTest` with `WebEnvironment.RANDOM_PORT` and `TestRestTemplate` to test HTTP endpoints. These tests mock the service layer using `@MockBean`.
99+
100+
2. **Store Tests**: Use `@SpringBootTest` with `WebEnvironment.NONE` to test the data storage layer. These tests often use `@Autowired` to inject the component under test.
101+
102+
3. **Service Tests**: Test the service layer with mocked dependencies.
103+
104+
4. **Unit Tests**: Simple tests for utility classes and DTOs without Spring context.
105+
106+
Common base classes and utilities:
107+
- `BaseControllerTest`: Sets up XML serialization/deserialization for controller tests
108+
- `StoreTestBase`: Common setup for store tests
109+
- `ServiceTestBase`: Common setup for service tests
110+
111+
### Running Tests
112+
113+
#### Running Server Module Tests
114+
115+
The server module tests can be run without Docker:
116+
117+
```bash
118+
# Run all server module tests
119+
./mvnw -pl server test
120+
121+
# Run a specific test class
122+
./mvnw -pl server test -Dtest=ObjectStoreTest
123+
124+
# Run a specific test method
125+
./mvnw -pl server test -Dtest=ObjectStoreTest#testStoreObject
126+
```
127+
128+
The server module uses Spring Test Profiler to analyze test performance. Test execution times are reported in the `target/spring-test-profiler` directory.
129+
130+
#### Running Integration Tests
131+
132+
Integration tests require Docker to be running as they start an S3Mock Docker container. The integration tests can only be executed through Maven because they need the S3Mock to run. The process works as follows:
133+
134+
1. The Docker Maven plugin (io.fabric8:docker-maven-plugin) starts the S3Mock Docker container in the pre-integration-test phase
135+
2. The Maven Failsafe plugin runs the integration tests against the running container
136+
3. The Docker Maven plugin stops the container in the post-integration-test phase
137+
138+
```bash
139+
# Run all integration tests
140+
./mvnw verify
141+
142+
# Run a specific integration test
143+
./mvnw -pl integration-tests -am verify -Dit.test=BucketIT
144+
145+
# Run tests without Docker (will skip integration tests)
146+
./mvnw verify -DskipDocker
147+
```
148+
149+
Note that attempting to run integration tests directly from your IDE without starting the S3Mock Docker container will result in connection errors, as the tests expect S3Mock to be running on specific ports.
150+
151+
### Writing New Tests
152+
153+
#### Writing Integration Tests
154+
155+
To create a new integration test:
156+
157+
1. Create a new Kotlin class in the `integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its` directory
158+
2. Extend the `S3TestBase` class to inherit utility methods
159+
3. Name your test class with an `IT` suffix to be recognized as an integration test
160+
4. Use the provided S3 client methods to interact with S3Mock
161+
162+
Example integration test:
163+
164+
```kotlin
165+
// ExampleIT.kt
166+
internal class ExampleIT : S3TestBase() {
167+
private val s3Client = createS3Client()
168+
169+
@Test
170+
fun testPutAndGetObject(testInfo: TestInfo) {
171+
// Create a bucket
172+
val bucketName = givenBucket(testInfo)
173+
174+
// Create test content
175+
val content = "This is a test file content"
176+
val contentBytes = content.toByteArray(StandardCharsets.UTF_8)
177+
178+
// Put an object into the bucket
179+
val key = "test-object.txt"
180+
val putObjectResponse = s3Client.putObject(
181+
PutObjectRequest.builder()
182+
.bucket(bucketName)
183+
.key(key)
184+
.build(),
185+
RequestBody.fromBytes(contentBytes)
186+
)
187+
188+
// Verify the object was uploaded successfully
189+
assertThat(putObjectResponse.eTag()).isNotBlank()
190+
191+
// Get the object back
192+
val getObjectResponse = s3Client.getObject(
193+
GetObjectRequest.builder()
194+
.bucket(bucketName)
195+
.key(key)
196+
.build()
197+
)
198+
199+
// Read the content and verify it matches what we uploaded
200+
val retrievedContent = getObjectResponse.readAllBytes()
201+
assertThat(retrievedContent).isEqualTo(contentBytes)
202+
203+
// Clean up
204+
s3Client.deleteObject { it.bucket(bucketName).key(key) }
205+
}
206+
}
207+
```
208+
209+
#### Writing Server Module Tests
210+
211+
The server module uses different testing approaches depending on what's being tested:
212+
213+
1. **Controller Tests**:
214+
- Extend `BaseControllerTest` to inherit XML serialization setup
215+
- Use `@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)`
216+
- Use `@MockBean` to mock service dependencies
217+
- Inject `TestRestTemplate` to make HTTP requests to the controller
218+
219+
Example controller test:
220+
221+
```kotlin
222+
// BucketControllerTest.kt
223+
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
224+
@MockBean(classes = [BucketService::class, ObjectService::class, MultipartService::class])
225+
internal class BucketControllerTest : BaseControllerTest() {
226+
@Autowired
227+
private lateinit var restTemplate: TestRestTemplate
228+
229+
@MockBean
230+
private lateinit var bucketService: BucketService
231+
232+
@Test
233+
fun testListBuckets() {
234+
// Mock service response
235+
whenever(bucketService.listBuckets()).thenReturn(givenBuckets(2))
236+
237+
// Make HTTP request
238+
val response = restTemplate.getForEntity("/", ListAllMyBucketsResult::class.java)
239+
240+
// Verify response
241+
assertThat(response.statusCode).isEqualTo(HttpStatus.OK)
242+
assertThat(response.body.buckets.bucket).hasSize(2)
243+
}
244+
}
245+
```
246+
247+
2. **Store Tests**:
248+
- Extend `StoreTestBase` for common setup
249+
- Use `@SpringBootTest(webEnvironment = WebEnvironment.NONE)`
250+
- Use `@Autowired` to inject the component under test
251+
252+
Example store test:
253+
254+
```kotlin
255+
// ObjectStoreTest.kt
256+
@SpringBootTest(classes = [StoreConfiguration::class], webEnvironment = SpringBootTest.WebEnvironment.NONE)
257+
@MockBean(classes = [KmsKeyStore::class, BucketStore::class])
258+
internal class ObjectStoreTest : StoreTestBase() {
259+
@Autowired
260+
private lateinit var objectStore: ObjectStore
261+
262+
@Test
263+
fun testStoreAndGetObject() {
264+
// Test storing and retrieving an object
265+
val sourceFile = File(TEST_FILE_PATH)
266+
val id = UUID.randomUUID().toString()
267+
val name = sourceFile.name
268+
269+
objectStore.storeS3ObjectMetadata(
270+
metadataFrom("test-bucket"), id, name, "text/plain",
271+
emptyMap(), sourceFile.toPath(),
272+
emptyMap(), emptyMap(), null, emptyList(),
273+
null, null, Owner.DEFAULT_OWNER, StorageClass.STANDARD
274+
)
275+
276+
val metadata = objectStore.getS3ObjectMetadata(metadataFrom("test-bucket"), id, null)
277+
278+
assertThat(metadata.key).isEqualTo(name)
279+
assertThat(metadata.contentType).isEqualTo("text/plain")
280+
}
281+
}
282+
```
283+
284+
3. **Unit Tests**:
285+
- Use standard JUnit 5 tests without Spring context
286+
- Focus on testing a single class or method in isolation
287+
288+
Example unit test:
289+
290+
```kotlin
291+
// DigestUtilTest.kt
292+
internal class DigestUtilTest {
293+
@Test
294+
fun testHexDigest() {
295+
val input = "test data".toByteArray()
296+
val expected = DigestUtils.md5Hex(input)
297+
298+
assertThat(DigestUtil.hexDigest(input)).isEqualTo(expected)
299+
}
300+
}
301+
```
302+
303+
## Additional Development Information
304+
305+
### Project Structure
306+
307+
S3Mock is a multi-module Maven project:
308+
309+
- `build-config`: Build configuration files
310+
- `docker`: Docker image build module
311+
- `integration-tests`: Integration tests
312+
- `server`: Core S3Mock implementation
313+
- `testsupport`: Test support modules for different testing frameworks
314+
315+
### Code Style
316+
317+
The project uses Checkstyle for Java code style checking. The configuration is in `build-config/checkstyle.xml`.
318+
319+
### License Headers
320+
321+
S3Mock uses the Apache License 2.0 and enforces proper license headers in all source files through the `license-maven-plugin`. Important rules:
322+
323+
- All source files must include the proper license header
324+
- When modifying a file, the copyright year range in the header must be updated to include the current year
325+
- The format is: `Copyright 2017-YYYY Adobe.` where YYYY is the current year
326+
- The license check runs automatically during the build process
327+
- To fix license headers, run: `./mvnw license:format`
328+
- To check license headers without modifying files: `./mvnw license:check`
329+
330+
### Debugging
331+
332+
To debug the S3Mock server:
333+
334+
1. Run the S3MockApplication class in your IDE with debug mode
335+
2. The server will start on ports 9090 (HTTP) and 9191 (HTTPS)
336+
3. Configure your S3 client to connect to these ports
337+
338+
Alternatively, you can run the Docker container with debug enabled:
339+
340+
```bash
341+
docker run -p 9090:9090 -p 9191:9191 -e debug=true -t adobe/s3mock
342+
```
343+
344+
### Common Issues
345+
346+
1. **Connection refused errors**: Ensure the S3Mock server is running and the ports are correctly configured.
347+
348+
2. **SSL certificate errors**: S3Mock uses a self-signed certificate. Configure your client to trust all certificates or use HTTP instead.
349+
350+
3. **Docker-related errors**: Make sure Docker is running, and you have permissions to create containers.
351+
352+
### Recommended Development Workflow
353+
354+
1. Make changes to the code
355+
2. Run unit tests frequently to verify basic functionality
356+
- Unit tests should be run very frequently during development
357+
- No task can be declared as done without running a full Maven build successfully
358+
3. Run integration tests to verify end-to-end functionality
359+
4. Build the Docker image to verify packaging
360+
5. Test with your application to verify real-world usage

‎CHANGELOG.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Jav
148148

149149
**The current major version 4 will receive new features, dependency updates and bug fixes on a continuous basis.**
150150

151-
## 4.8.0 - PLANNED
151+
## 4.9.0 - PLANNED
152152
Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration.
153153

154154
* Features and fixes
@@ -160,6 +160,26 @@ Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Jav
160160
* Version updates (build dependencies)
161161
* TBD
162162

163+
## 4.8.0 - PLANNED
164+
Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration.
165+
166+
* Features and fixes
167+
* TBD
168+
* Refactorings
169+
* UploadId is always a UUID. Use UUID type in S3Mock instead of String.
170+
* Validate that partNumbers to be positive integers.
171+
* Force convergence on the newest available transitive dependency versions.
172+
* Optimize file storage for large objects by using buffered streams.
173+
* Version updates (deliverable dependencies)
174+
*
175+
* Bump org.apache.commons:commons-compress from 1.27.1 to 1.28.0
176+
* Version updates (build dependencies)
177+
* Bump kotlin.version from 2.2.0 to 2.2.10
178+
* Bump digital.pragmatech.testing:spring-test-profiler from 0.0.5 to 0.0.11
179+
* Bump com.puppycrawl.tools:checkstyle from 10.26.1 to 11.0.0
180+
* Bump github/codeql-action from 3.29.4 to 3.29.9
181+
* Bump actions/checkout from 4.2.2 to 5.0.0
182+
163183
## 4.7.0
164184
Version 4.x is JDK17 LTS bytecode compatible, with Docker and JUnit / direct Java integration.
165185

0 commit comments

Comments
 (0)