Skip to content

Commit 0ff91e6

Browse files
committed
Add multipart upload API and object version listing
Implements S3-compatible multipart upload endpoints with a new MultipartHandler, service, and router integration. Adds ListObjectVersions API and XML response types to object handler. Updates README and MEMORY_BANK.md to reflect completed versioning phase and multipart upload as high priority. Refactors main.go for multipart service wiring and updates dependencies for testing.
1 parent 117bf5b commit 0ff91e6

File tree

11 files changed

+3014
-151
lines changed

11 files changed

+3014
-151
lines changed

MEMORY_BANK.md

Lines changed: 110 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -141,38 +141,59 @@
141141
- [x] HTTP Router - `internal/handler/router.go`
142142
- [x] Server Integration - `cmd/alexander-server/main.go`
143143

144-
### Phase 4: Object Operations (Non-Versioned) - Current
145-
- [ ] PutObject (with CAS deduplication)
146-
- [ ] GetObject
147-
- [ ] HeadObject
148-
- [ ] DeleteObject
149-
- [ ] ListObjects (v1)
150-
- [ ] ListObjectsV2
151-
- [ ] CopyObject
152-
153-
### Phase 5: Versioning
154-
- [ ] PutObject with version creation
155-
- [ ] GetObject with versionId
156-
- [ ] DeleteObject (create delete marker)
157-
- [ ] DeleteObject with versionId
158-
- [ ] ListObjectVersions
159-
160-
### Phase 6: Multipart Upload
144+
### Phase 4: Object Operations ✅ COMPLETED
145+
- [x] PutObject (with CAS deduplication) - `internal/service/object_service.go`, `internal/handler/object_handler.go`
146+
- [x] GetObject - `internal/service/object_service.go`, `internal/handler/object_handler.go`
147+
- [x] HeadObject - `internal/service/object_service.go`, `internal/handler/object_handler.go`
148+
- [x] DeleteObject - `internal/service/object_service.go`, `internal/handler/object_handler.go`
149+
- [x] ListObjects (v1) - `internal/service/object_service.go`, `internal/handler/object_handler.go`
150+
- [x] ListObjectsV2 - `internal/service/object_service.go`, `internal/handler/object_handler.go`
151+
- [x] CopyObject - `internal/service/object_service.go`, `internal/handler/object_handler.go`
152+
- [x] Router Integration - `internal/handler/router.go`
153+
- [x] Server Integration - `cmd/alexander-server/main.go`
154+
155+
### Phase 5: Versioning ✅ COMPLETED
156+
- [x] PutObject with version creation - Versioning enabled bucket'larda yeni version oluşturulur
157+
- [x] GetObject with versionId - `?versionId=xxx` query parametresi ile belirli version getirme
158+
- [x] DeleteObject (create delete marker) - Versioning enabled bucket'larda delete marker oluşturma
159+
- [x] DeleteObject with versionId - Belirli version'ı permanent silme
160+
- [x] ListObjectVersions - `GET /{bucket}?versions` endpoint'i
161+
162+
### Phase 6: Multipart Upload ⚠️ HIGH PRIORITY - Current
163+
> **Community Feedback**: "Without multipart uploads, large files can't be uploaded reliably. This is critical for S3 compatibility."
164+
161165
- [ ] InitiateMultipartUpload
162166
- [ ] UploadPart
163167
- [ ] CompleteMultipartUpload
164168
- [ ] AbortMultipartUpload
165169
- [ ] ListMultipartUploads
166170
- [ ] ListParts
167171

172+
**API Design Preview:**
173+
```
174+
POST /bucket/key?uploads → InitiateMultipartUpload
175+
PUT /bucket/key?partNumber=N&uploadId=X → UploadPart
176+
POST /bucket/key?uploadId=X → CompleteMultipartUpload
177+
DELETE /bucket/key?uploadId=X → AbortMultipartUpload
178+
GET /bucket?uploads → ListMultipartUploads
179+
GET /bucket/key?uploadId=X → ListParts
180+
```
181+
168182
### Phase 7: Operations & Observability
169183
- [ ] Garbage collection for orphan blobs
170184
- [ ] Prometheus metrics
171185
- [ ] Health check endpoints
172186
- [ ] Request tracing
173187
- [ ] Rate limiting
174188

175-
### Phase 8: Advanced Features (Future)
189+
### Phase 8: Architecture Improvements (Community Requested)
190+
> **Community Feedback**: "PostgreSQL + Redis is overkill for single-node deployments."
191+
192+
- [ ] Embedded database support (SQLite or BadgerDB)
193+
- [ ] Memory-based locking for single-node mode (eliminate Redis dependency)
194+
- [ ] Single binary deployment mode
195+
196+
### Phase 9: Advanced Features (Future)
176197
- [ ] Bucket policies
177198
- [ ] Object lifecycle rules
178199
- [ ] Cross-region replication
@@ -330,55 +351,93 @@ Path: /data/ab/cd/abcdef1234567890...
330351

331352
---
332353

354+
### Decision 8: Optional Redis for Single-Node Deployments
355+
356+
**Date**: 2025-12-04
357+
**Status**: ✅ Approved (Documentation Updated)
358+
359+
**Context**: Community feedback that PostgreSQL + Redis is "overkill" for simple homelab deployments.
360+
361+
**Decision**: Make Redis optional. Single-node deployments use in-memory locking.
362+
363+
**Rationale**:
364+
- **Simpler Deployment**: `docker run` with just PostgreSQL
365+
- **Lower Resource Usage**: No Redis process for small deployments
366+
- **Scalability Path**: Redis enabled for cluster/HA deployments
367+
368+
**Implementation**:
369+
- `internal/lock` interface abstraction
370+
- `MemoryLocker` for single-node (sync.Mutex)
371+
- `RedisLocker` for distributed deployments
372+
- Config flag: `storage.distributed_mode: true|false`
373+
374+
---
375+
376+
### Decision 9: Future Embedded Database Support
377+
378+
**Date**: 2025-12-04
379+
**Status**: 🔜 Planned
380+
381+
**Context**: Community feedback requesting "zero-dependency" single-binary deployment.
382+
383+
**Decision**: Add SQLite or BadgerDB as alternative metadata backend (Phase 8).
384+
385+
**Rationale**:
386+
- **True Zero-Dependency**: Single binary, no external services
387+
- **Homelab Friendly**: `./alexander-server` just works
388+
- **Edge Deployments**: IoT, embedded systems, air-gapped networks
389+
390+
**Implementation Plan**:
391+
- Repository interface already supports this (abstraction exists)
392+
- Add `internal/repository/sqlite/` or `internal/repository/badger/`
393+
- Config: `database.driver: postgres|sqlite|badger`
394+
395+
**Trade-offs**:
396+
- SQLite: Limited concurrent writes, but excellent for read-heavy archival
397+
- BadgerDB: Better write performance, Go-native, but less tooling
398+
399+
---
400+
333401
## Section 4: Current Context
334402

335403
### Active Development Phase
336-
**Phase 3: Bucket Operations**
404+
**Phase 6: Multipart Upload**
337405

338406
### Current Task
339-
Implementing bucket service and handlers for S3-compatible bucket operations
407+
Implementing S3-compatible multipart upload for large files
340408

341409
### Last Updated
342410
2025-12-04
343411

412+
### Completed Phases
413+
- ✅ Phase 1: Core Infrastructure
414+
- ✅ Phase 2: IAM & Authentication
415+
- ✅ Phase 3: Bucket Operations
416+
- ✅ Phase 4: Object Operations
417+
- ✅ Phase 5: Versioning
418+
344419
### Files Modified This Session
345-
- `go.mod` - Project module definition
346-
- `Makefile` - Build and development commands
347-
- `Dockerfile` - Container build definition
348-
- `.gitignore` - Git ignore rules
349-
- `configs/config.yaml.example` - Configuration template
350-
- `configs/docker-compose.yaml` - Local development stack
351-
- `cmd/alexander-server/main.go` - Server entry point
352-
- `cmd/alexander-admin/main.go` - Admin CLI entry point
353-
- `cmd/alexander-migrate/main.go` - Migration tool entry point
354-
- `MEMORY_BANK.md` - This file
355-
- `migrations/postgres/000001_init.up.sql` - Database schema
356-
- `migrations/postgres/000001_init.down.sql` - Schema rollback
357-
- `internal/domain/*.go` - Domain models (7 files)
358-
- `internal/pkg/crypto/*.go` - Crypto utilities (3 files)
359-
- `internal/storage/*.go` - Storage interfaces (3 files)
360-
- `internal/storage/filesystem/storage.go` - Filesystem storage implementation
361-
- `internal/repository/*.go` - Repository interfaces (3 files)
362-
- `internal/repository/postgres/*.go` - PostgreSQL repositories (8 files)
363-
- `internal/cache/redis/*.go` - Redis cache implementation (2 files)
364-
- `internal/config/config.go` - Configuration loading with Viper
365-
- `internal/auth/*.go` - Auth middleware and AWS v4 signature (6 files)
366-
- `internal/service/errors.go` - Service layer error definitions
367-
- `internal/service/user_service.go` - User management service
368-
- `internal/service/iam_service.go` - IAM/access key management service
369-
- `internal/service/presign_service.go` - Presigned URL generation service
420+
- `internal/service/object_service.go` - Added ListObjectVersions, fixed versioning logic
421+
- `internal/service/object_service_test.go` - Added versioning unit tests
422+
- `internal/handler/object_handler.go` - Added ListObjectVersions handler and XML types
423+
- `internal/handler/router.go` - Added `?versions` route
424+
- `MEMORY_BANK.md` - Updated with Phase 5 completion
370425

371426
### Pending Tasks
372-
1. Implement BucketService (`internal/service/bucket_service.go`)
373-
2. Create HTTP handlers for bucket operations
374-
3. Wire services with chi router
375-
4. Add integration tests for bucket operations
427+
1. Multipart upload support (Phase 6 - HIGH PRIORITY)
428+
2. Add embedded database option (Phase 8)
376429

377430
### Known Issues
378431
None currently.
379432

380-
### Technical Debt
381-
None currently.
433+
### Community Feedback Addressed
434+
- [x] Added "Best for: Archival, Backups, Homelabs" to README
435+
- [x] Added Mermaid architecture diagrams
436+
- [x] Clarified io.TeeReader streaming hash in docs
437+
- [x] Marked Multipart Upload as HIGH PRIORITY
438+
- [x] Documented Redis as optional for single-node
439+
- [x] Added future SQLite/BadgerDB support to roadmap
440+
- [x] Added benchmark section placeholder
382441

383442
---
384443

0 commit comments

Comments
 (0)