Skip to content

Commit 74a0d0f

Browse files
steelbrainsteelbrain
committed
v5: Tunnel file I/O over TCP — no shared filesystem required
Architecture: - Replace shared filesystem (NFS/SMB) with file I/O tunneling over TCP - Patch ffmpeg source to replace 10 POSIX file ops (open, read, write, close, lseek, fstat, ftruncate, unlink, rename, mkdir) with fio_* equivalents - All file I/O is tunneled back to the client — files are never stored on the server Core: - Go client binary that acts as a drop-in ffmpeg replacement - Go server binary that launches patched ffmpeg and relays fio messages - Binary wire protocol with HMAC-SHA256 authentication - Session multiplexer handling TCP, stdout/stderr, and fio loopback - Concurrent client support — each session gets its own ffmpeg process ffmpeg: - Vendor jellyfin-ffmpeg 7.1.3 as git submodule for 94 HW acceleration patches - Pre-built ffmpeg/ffprobe binaries with NVENC, QSV, VAAPI, AMF, VideoToolbox support - 4 patches applied on top: fio makefile, file protocol, HLS delete, mkdir Platform support: - Linux x86_64 and arm64 (native builds) - macOS arm64 (native) and x86_64 (cross-compiled) - Windows x86_64 (MSYS2/MinGW) - Windows arm64 client only Configuration: - JSONC config with // and /* */ comments, trailing commas - 8-location config file search path - TCP and Unix domain socket support - Configurable logging with $TMPDIR/$HOME/$USER/$PWD interpolation - Server-side codec rewrites (plain string replacement on ffmpeg args) - Debug mode logging original and rewritten args CI/CD: - CI workflow: Go tests, fio unit tests (46), static analysis - Integration workflow: build patched ffmpeg + 9 integration test scripts - Release workflow: 12 Go binaries + 7 ffmpeg builds across all platforms Documentation: - Quick start, configuration reference, Docker integration, troubleshooting - v4 to v5 upgrade guide with migration checklist - Split license: fio/ and patches/ are GPL v3, everything else is MIT
1 parent 43c446f commit 74a0d0f

File tree

77 files changed

+15476
-1843
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

77 files changed

+15476
-1843
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 67 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -2,46 +2,84 @@ name: CI
22

33
on:
44
push:
5-
branches: [ main ]
5+
branches: [main, 'steelbrain/**']
66
pull_request:
77

8+
env:
9+
GO_VERSION: "1.24"
10+
811
jobs:
9-
build-and-test:
10-
name: Build and Test
12+
go-tests:
13+
name: Go Tests
1114
runs-on: ubuntu-latest
12-
1315
steps:
14-
- name: Checkout code
15-
uses: actions/checkout@v4
16-
17-
- name: Set up Go
18-
uses: actions/setup-go@v5
19-
with:
20-
go-version: '>=1.18'
21-
check-latest: true
16+
- uses: actions/checkout@v4
2217

23-
- name: Format Check
24-
id: fmt-check
18+
- name: Install build dependencies
2519
run: |
26-
go fmt ./...
27-
if [ -n "$(git status --porcelain)" ]; then
28-
echo "Code is not properly formatted. Run 'go fmt ./...' locally and commit the changes."
29-
git status --porcelain
30-
exit 1
31-
fi
20+
sudo apt-get update
21+
sudo apt-get install -y build-essential
22+
23+
- uses: actions/setup-go@v5
24+
with:
25+
go-version: ${{ env.GO_VERSION }}
3226

3327
- name: Build
34-
run: make build
28+
run: go build ./...
3529

3630
- name: Test
37-
run: make test
31+
run: go test ./internal/... -race -count=1 -timeout 120s
3832

39-
- name: Build for all platforms
40-
run: make cross-platform
33+
fio-unit-tests:
34+
name: fio Unit Tests
35+
runs-on: ubuntu-latest
36+
steps:
37+
- uses: actions/checkout@v4
38+
39+
- name: Install build dependencies
40+
run: |
41+
sudo apt-get update
42+
sudo apt-get install -y build-essential
43+
44+
- name: Build and run fio tests
45+
run: |
46+
cd fio
47+
cc -Wall -Wextra -Werror -std=c11 -DFIO_TESTING -D_DEFAULT_SOURCE \
48+
-o fio_test fio.c fio_test.c -lpthread
49+
./fio_test
4150
42-
- name: Upload cross-platform binaries as artifacts
43-
uses: actions/upload-artifact@v4
51+
- name: Build and run fio tests (sanitizers)
52+
run: |
53+
cd fio
54+
cc -Wall -Wextra -Werror -std=c11 -DFIO_TESTING -D_DEFAULT_SOURCE \
55+
-fsanitize=address,undefined -fno-omit-frame-pointer -g \
56+
-o fio_test_sanitize fio.c fio_test.c -lpthread
57+
./fio_test_sanitize
58+
59+
static-analysis:
60+
name: Static Analysis
61+
runs-on: ubuntu-latest
62+
steps:
63+
- uses: actions/checkout@v4
4464
with:
45-
name: ffmpeg-over-ip-cross-platform
46-
path: release/
47-
if-no-files-found: error
65+
submodules: true
66+
67+
- name: Install build dependencies
68+
run: |
69+
sudo apt-get update
70+
sudo apt-get install -y patch
71+
72+
- name: Patch ffmpeg source
73+
run: |
74+
cp fio/fio.c fio/fio.h third_party/jellyfin-ffmpeg/libavformat/
75+
cd third_party/jellyfin-ffmpeg
76+
for patch in ../../patches/*.patch; do
77+
patch -p1 < "$patch"
78+
done
79+
80+
- uses: actions/setup-python@v5
81+
with:
82+
python-version: '3.12'
83+
84+
- name: Scan for raw syscalls
85+
run: python3 tests/static-analysis/scan-raw-syscalls.py third_party/jellyfin-ffmpeg

‎.github/workflows/integration.yml‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: Integration
2+
3+
on:
4+
push:
5+
branches: [main, 'steelbrain/**']
6+
pull_request:
7+
8+
env:
9+
GO_VERSION: "1.24"
10+
11+
jobs:
12+
build-ffmpeg:
13+
name: Build Patched ffmpeg
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
with:
18+
submodules: true
19+
20+
- name: Cache ffmpeg build
21+
id: cache-ffmpeg
22+
uses: actions/cache@v4
23+
with:
24+
path: build/ffmpeg
25+
key: ffmpeg-${{ hashFiles('third_party/jellyfin-ffmpeg/.git', 'patches/*.patch', 'fio/fio.c', 'fio/fio.h') }}-minimal
26+
27+
- name: Build ffmpeg (minimal)
28+
if: steps.cache-ffmpeg.outputs.cache-hit != 'true'
29+
run: |
30+
sudo apt-get update
31+
sudo apt-get install -y build-essential nasm yasm patch
32+
bash scripts/build-ffmpeg.sh --minimal
33+
34+
- name: Upload ffmpeg binaries
35+
uses: actions/upload-artifact@v4
36+
with:
37+
name: ffmpeg-binaries
38+
path: build/ffmpeg/bin/
39+
retention-days: 7
40+
41+
integration-tests:
42+
name: Integration Tests
43+
runs-on: ubuntu-latest
44+
needs: [build-ffmpeg]
45+
steps:
46+
- uses: actions/checkout@v4
47+
48+
- uses: actions/setup-go@v5
49+
with:
50+
go-version: ${{ env.GO_VERSION }}
51+
52+
- name: Download ffmpeg binaries
53+
uses: actions/download-artifact@v4
54+
with:
55+
name: ffmpeg-binaries
56+
path: build/ffmpeg/bin/
57+
58+
- name: Install build dependencies
59+
run: |
60+
sudo apt-get update
61+
sudo apt-get install -y build-essential
62+
63+
- name: Make ffmpeg executable
64+
run: chmod +x build/ffmpeg/bin/*
65+
66+
- name: Build fio test binaries
67+
run: |
68+
cd tests/fio-bins
69+
cc -Wall -Wextra -Werror -std=c11 -DFIO_TESTING -D_DEFAULT_SOURCE \
70+
-I../../fio -o fio_copy ../../fio/fio.c fio_copy.c -lpthread
71+
cc -Wall -Wextra -Werror -std=c11 -DFIO_TESTING -D_DEFAULT_SOURCE \
72+
-I../../fio -o fio_ops ../../fio/fio.c fio_ops.c -lpthread
73+
74+
- name: Run integration tests
75+
run: |
76+
FAILED=0
77+
for test in tests/integration/test-*.sh; do
78+
echo ""
79+
echo "========================================"
80+
echo "Running: $(basename $test)"
81+
echo "========================================"
82+
if bash "$test"; then
83+
echo "PASSED: $(basename $test)"
84+
else
85+
echo "FAILED: $(basename $test)"
86+
FAILED=1
87+
fi
88+
done
89+
if [ "$FAILED" -ne 0 ]; then
90+
echo ""
91+
echo "Some integration tests FAILED"
92+
exit 1
93+
fi
94+
echo ""
95+
echo "All integration tests PASSED"

‎.github/workflows/release.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
branches: [main, 'steelbrain/**']
6+
pull_request:
7+
8+
permissions:
9+
contents: read
10+
11+
env:
12+
GO_VERSION: "1.24"
13+
14+
jobs:
15+
release:
16+
name: ${{ matrix.platform }}
17+
runs-on: ubuntu-latest
18+
strategy:
19+
matrix:
20+
include:
21+
- platform: linux-amd64
22+
goos: linux
23+
goarch: amd64
24+
ffmpeg_target: linux64
25+
ext: ""
26+
- platform: linux-arm64
27+
goos: linux
28+
goarch: arm64
29+
ffmpeg_target: linuxarm64
30+
ext: ""
31+
- platform: windows-amd64
32+
goos: windows
33+
goarch: amd64
34+
ffmpeg_target: win64
35+
ext: ".exe"
36+
- platform: macos-arm64
37+
goos: darwin
38+
goarch: arm64
39+
ffmpeg_target: macarm64
40+
ext: ""
41+
- platform: macos-amd64
42+
goos: darwin
43+
goarch: amd64
44+
ffmpeg_target: mac64
45+
ext: ""
46+
steps:
47+
- uses: actions/checkout@v4
48+
49+
- uses: actions/setup-go@v5
50+
with:
51+
go-version: ${{ env.GO_VERSION }}
52+
53+
- name: Build Go binaries
54+
env:
55+
GOOS: ${{ matrix.goos }}
56+
GOARCH: ${{ matrix.goarch }}
57+
CGO_ENABLED: "0"
58+
run: |
59+
go build -trimpath -ldflags="-s -w" \
60+
-o ffmpeg-over-ip-client${{ matrix.ext }} ./cmd/client
61+
go build -trimpath -ldflags="-s -w" \
62+
-o ffmpeg-over-ip-server${{ matrix.ext }} ./cmd/server
63+
64+
- name: Extract ffmpeg binaries
65+
run: |
66+
mkdir -p ffmpeg-bin
67+
ARCHIVE=$(ls artifacts/*_${{ matrix.ffmpeg_target }}-gpl.*)
68+
case "$ARCHIVE" in
69+
*.tar.xz) tar xf "$ARCHIVE" -C ffmpeg-bin ;;
70+
*.zip) unzip -q "$ARCHIVE" -d ffmpeg-bin ;;
71+
esac
72+
73+
- name: Upload client
74+
uses: actions/upload-artifact@v4
75+
with:
76+
name: ${{ matrix.platform }}-ffmpeg-over-ip-client
77+
path: ffmpeg-over-ip-client${{ matrix.ext }}
78+
retention-days: 30
79+
80+
- name: Prepare server package
81+
run: |
82+
cp ffmpeg-bin/ffmpeg${{ matrix.ext }} ffmpeg${{ matrix.ext }}
83+
cp ffmpeg-bin/ffprobe${{ matrix.ext }} ffprobe${{ matrix.ext }}
84+
85+
- name: Upload server
86+
uses: actions/upload-artifact@v4
87+
with:
88+
name: ${{ matrix.platform }}-ffmpeg-over-ip-server
89+
path: |
90+
ffmpeg-over-ip-server${{ matrix.ext }}
91+
ffmpeg${{ matrix.ext }}
92+
ffprobe${{ matrix.ext }}
93+
retention-days: 30

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,13 @@ node_modules
66
# Testing configs
77
ffmpeg-over-ip.client.jsonc
88
ffmpeg-over-ip.server.jsonc
9+
10+
# Build artifacts
11+
build/
12+
fio/fio_test
13+
fio/fio_test_sanitize
14+
fio/*.plist
15+
fio/*.dSYM/
16+
tests/fio-bins/fio_copy
17+
tests/fio-bins/fio_ops
18+
tests/integration/harness

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "third_party/jellyfin-ffmpeg"]
2+
path = third_party/jellyfin-ffmpeg
3+
url = https://github.com/jellyfin/jellyfin-ffmpeg.git

‎CONTRIBUTING.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Contributing
2+
3+
## Building from Source
4+
5+
### Prerequisites
6+
7+
- Go 1.24+
8+
- A C compiler (gcc or clang)
9+
- Make
10+
11+
### Client
12+
13+
The client has no dependencies beyond the Go standard library.
14+
15+
```bash
16+
go build -o ffmpeg-over-ip-client ./cmd/client
17+
```
18+
19+
### Server (with patched ffmpeg)
20+
21+
```bash
22+
# Build the patched ffmpeg (minimal, for testing)
23+
bash scripts/build-ffmpeg.sh --minimal
24+
25+
# Build the server
26+
go build -o build/ffmpeg-over-ip-server ./cmd/server
27+
28+
# Copy ffmpeg/ffprobe next to the server binary
29+
cp build/ffmpeg/bin/ffmpeg build/
30+
cp build/ffmpeg/bin/ffprobe build/
31+
```
32+
33+
34+
## Running Tests
35+
36+
```bash
37+
# Go unit tests
38+
go test ./internal/... -race
39+
40+
# C fio unit tests
41+
cd fio && make test
42+
43+
# Integration tests (requires patched ffmpeg)
44+
bash tests/integration/test-client-server.sh
45+
46+
# All integration tests
47+
for t in tests/integration/test-*.sh; do bash "$t"; done
48+
49+
# Static analysis (scan for raw syscalls in patched ffmpeg)
50+
python3 tests/static-analysis/scan-raw-syscalls.py
51+
```
52+
53+
## How It Works (in detail)
54+
55+
The patched ffmpeg replaces 10 POSIX file operations (open, read, write, close, lseek, fstat, ftruncate, unlink, rename, mkdir) with `fio_*` equivalents. When ffmpeg runs on the server, every file operation is tunneled back to the client over the TCP connection. The client performs the actual I/O on its local filesystem and sends results back.
56+
57+
58+
## Project Structure
59+
60+
- `cmd/client/` — client binary (drop-in ffmpeg replacement)
61+
- `cmd/server/` — server binary (launches patched ffmpeg)
62+
- `internal/` — shared Go packages (protocol, session, filehandler, config)
63+
- `fio/` — C tunneling layer patched into ffmpeg (GPL v3)
64+
- `patches/` — patches applied to jellyfin-ffmpeg source (GPL v3)
65+
- `third_party/jellyfin-ffmpeg/` — jellyfin-ffmpeg submodule
66+
- `scripts/` — build scripts
67+
- `tests/` — integration tests and static analysis
68+
69+
## License
70+
71+
Split license — see [LICENSE.md](LICENSE.md). Code in `fio/` and `patches/` is GPL v3. Everything else is MIT.

0 commit comments

Comments
 (0)