Store Logs in object storeage instead of database

[6543]

optional config (false by default) to store log streams in object storage

---

Bounty: 100$

[qwerty287]

#3568 has been merged, it's possible now to use files to store the logs instead of the DB.
This dicussion says "object storage" however: What were the ideas originally? Simple file storing or something like S3?
Also maybe @go-gitea members know more because I think the bounty was created by them?

[xoxys]

I would prefer object store (s3) storage for logs as well. The main reason is it works on k3s and allows pretty simple retention policies. Pipeline logs are flooding the DB and It's pretty hard to clean them up on DB level.

[Levy-Tal]

I want to implement this S3 log storage feature.

Background

In Woodpecker, agents send log chunks of around 1MB of data. Unfortunately, S3 doesn't support appending to an object.
Also, with S3 you pay for storage/network traffic/API requests, so there are several implementation options, each with tradeoffs:

Implementation Options for the Interface

https://github.com/woodpecker-ci/woodpecker/blob/main/server/services/log/service.go#L5

The examples below are for a large log file (1GB) with 1000 append calls of 1MB each.

S3 Pricing Reference (2025 US East):

• PUT, COPY, POST, LIST requests: $0.005 per 1,000 requests
• GET and all other requests: $0.0004 per 1,000 requests
• Data transfer OUT: $0.09 per GB (first 1GB/month free)
• Data transfer IN: Free

1. Simple Approach (Single Object)

LogAppend: Fetch the existing log file, add the new content, and push back to S3

• Memory: 1GB + 1MB chunk size (grows with each append)
• Network: ~499.5GB download + 500.5GB upload (uploud data is free) - very wasteful
• API calls: 2000 total (1000 GETs + 1000 PUTs)
• Cost per 1GB log: ~$44.965 ($44.955 data transfer on download + $0.004 GET + $0.005 PUT)

LogFind: Fetch the entire file from S3 to memory, then unmarshal and return the parsed log

• Memory: 2GB peak (1GB for file + 1GB for unmarshaling)
• Network: 1GB (can be faster with multipart download, but increases API calls)
• API calls: 1 GET (or more with multipart download)
• Cost per read: ~$0.09

Note: This is extremely wasteful for LogAppend and can lead to OOM errors, but fast for LogFind

2. Chunked Approach

LogAppend: Every append creates a new object under a folder with the stepID (from stepId.json to stepId/chunk-1.json...)

• Memory: ~1MB constant
• Network: 1GB total
• API calls: ~2000 (1000 LIST operations + 1000 PUT operations)
• Cost per 1GB log: $0.01 ($0.005 LIST + $0.005 PUT)

LogFind: Get the number of chunks in a step, then fetch them one by one and return the full log

• Memory: ~1GB total (streaming chunks)
• Network: 1GB (slower than multipart download)
• API calls: ~1001 (1 LIST + 1000 GETs)
• Cost per read: $0.0904 ($0.09 data transfer + $0.0004 GET)

Note: Woodpecker is read-intensive - after a step completes, it's mostly LogFind operations. Chunked approach has faster uploads but slower reads.

3. Optimized Variations

Simple approach optimization:

• LogFind: Write file to disk instead of keeping in memory: 1GB disk + reduced memory usage to prevent OOM

Chunked optimization (Database metadata):
Save metadata in the database (in the log_entries table that's empty when using S3) - this eliminates expensive S3 LIST operations:

LogAppend:

• Memory: ~1MB constant
• Network: 1GB total
• API calls: 1000 PUTs (1 DB query replaces 1000 LIST operations)
• Cost per 1GB log: $0.005 ($0.005 PUT)

LogFind:

• Memory: ~1GB total
• Network: 1GB total
• API calls: 1000 GETs (1 DB query replaces 1 LIST operations)
• Cost per read: $0.0904 ($0.09 data transfer + $0.0004 GET)

Hybrid approach (Stream-to-disk):
LogAppend saves temp file to disk, and when the log is finished (indicated by NULL data in the last entry), upload to S3 and delete temp file:

LogAppend:

• Memory: ~1MB constant
• Network: 1GB total (efficient multipart upload)
• API calls: 1 PUT operation
• Disk: 1GB temporary file
• Cost per 1GB log: $0.000005 (single PUT request)

LogFind:

• Memory: ~1GB total
• Network: 1GB total (efficient multipart download)
• API calls: 1 GET operation
• Disk: 0
• Cost per read: $0.09 ($0.09 data transfer + ~$0 1 GET)

Recommended Solution: Hybrid Approach

I think the best option is the hybrid approach, as it provides:

• Best cost efficiency: Minimal API calls mean almost zero API costs
• Best performance: Single PUT/GET operations with multipart efficiency
• Best resource usage: Balanced for Woodpecker's read-heavy workload

Would love to hear the maintainers thoughts on this approach.

[lafriks]

I think the best approach would be to store in temp file and save to s3 once step is finished

[Levy-Tal]

There is other option - instead of temp file, to store the temp log in the database. The same as in the default log store. And when the step is over, upload it to S3 and delete it from the database.