Removed python code, fixed issues with tires

Author: tess1o

README.md

@@ -32,10 +32,10 @@ A web app for watching Formula 1 sessions with real timing data, car positions o
 ## Architecture
 
 - **Frontend**: React + Vite with Tailwind CSS, served by Nginx
-- **Backend**: Go web service - serves pre-computed data from local storage
-- **Data Source**: [FastF1](https://github.com/theOehrly/Fast-F1) (used during data processing only)
+- **Backend**: Go web service - serves pre-computed data from SQLite (`/data/f1.db`)
+- **Data Source**: Official F1 timing endpoints (`livetiming.formula1.com`) ingested directly by the Go backend
 
-Session data is processed once and stored locally (or in R2 for remote access). You can either pre-compute data in bulk ahead of time, or let the app process sessions on demand when you select them.
+Session data is processed once and stored in SQLite (`/data/f1.db`). You can queue downloads in the UI, or let the app process sessions on demand when you select them.
 
 ## Self-Hosting Guide
 
@@ -53,7 +53,6 @@ services:
       - DATA_DIR=/data
     volumes:
       - f1data:/data
-      - f1cache:/data/fastf1-cache
 
   frontend:
     image: ghcr.io/adn8naiagent/f1replaytiming-frontend:latest
@@ -64,7 +63,6 @@ services:
 
 volumes:
   f1data:
-  f1cache:
 ```
 
 Then run:
@@ -132,34 +130,24 @@ frontend:
 - `REPLAY_CACHE_MAX_MB` - maximum RAM budget for cached replay sessions in the Go backend (default `256`)
 - `REPLAY_CACHE_TTL_SECONDS` - how long an inactive replay session stays cached after last client disconnect (default `300`)
 - `REPLAY_SAMPLE_INTERVAL_SECONDS` - replay frame sampling interval during precompute (default `0.5`; higher values reduce CPU/RAM during downloads)
+- `SQLITE_PATH` - SQLite database path (default `/data/f1.db`)
+- `SQLITE_BUSY_TIMEOUT_MS` - SQLite lock wait timeout in milliseconds (default `5000`)
+- `REPLAY_CHUNK_FRAMES` - replay protobuf chunk frame count (default `256`)
+- `TELEMETRY_CHUNK_SAMPLES` - telemetry sampling target per lap (default `512`)
+- `PROCESS_CHUNK_CODEC` - chunk codec for replay/telemetry (`protobuf` or `protobuf+zstd`, default `protobuf`)
+- `PROCESS_RAW_MIN_DT_SECONDS` - minimum accepted delta between raw stream samples per driver during ingest (default `0.10`)
 - `GOMEMLIMIT` / `GOGC` - Go runtime GC tuning knobs for tighter memory limits (for example `GOMEMLIMIT=256MiB`, `GOGC=50`)
 
 #### Data
 
 Session data is persisted in a Docker volume, so it survives restarts.
-
-To pre-process session data in bulk (instead of on demand), use the precompute script:
-
-```bash
-# Process a specific race weekend
-docker compose exec backend python data-fetcher/precompute.py 2026 --round 1
-
-# Process only the race session (skip practice/qualifying)
-docker compose exec backend python data-fetcher/precompute.py 2026 --round 1 --session R
-
-# Process an entire season (will take several hours)
-docker compose exec backend python data-fetcher/precompute.py 2025 --skip-existing
-
-# Process multiple years
-docker compose exec backend python data-fetcher/precompute.py 2024 2025 --skip-existing
-```
+Use the Downloads page in the UI to enqueue single sessions, weekends, or full-season ranges for background processing.
 
 ### Option C: Manual setup
 
 #### Prerequisites
 
 - Go 1.26+
-- Python 3.10+
 - Node.js 18+
 
 #### 1. Clone the repository
@@ -175,8 +163,7 @@ cd F1timing
 ```
 PORT=8000
 DATA_DIR=./data
-PY_WORKER_PATH=../data-fetcher/worker_bridge.py
-# PYTHON_BIN=python3
+SQLITE_PATH=./data/f1.db
 
 # Optional - restrict access with a passphrase
 AUTH_ENABLED=false
@@ -200,9 +187,6 @@ AUTH_PASSPHRASE=
 #### 3. Install dependencies and start
 
 ```bash
-# Python data-fetcher dependencies (required for on-demand session processing)
-python3 -m pip install -r data-fetcher/requirements-worker.txt
-
 # Backend (Go API)
 cd backend
 go run .
@@ -221,31 +205,11 @@ There are two ways to get session data into the app:
 
 #### Option A: On-demand processing (recommended for getting started)
 
-Simply select any past session from the homepage. If the data hasn't been processed yet, the app will automatically fetch and process it using FastF1 and start the replay. The first load of a session takes **1-3 minutes**. After that, it's instant.
+Select any past session from the homepage. If the data has not been processed yet, the backend will fetch and process it directly in Go, then store it in SQLite. The first load typically takes **1-3 minutes**.
 
-#### Option B: Bulk pre-compute (recommended for preparing a full season)
+#### Option B: Queue background downloads (recommended for preparing a full season)
 
-Use the CLI script to process sessions ahead of time. This is useful if you want all data ready before you start using the app.
-
-```bash
-# Data fetcher (Python)
-cd data-fetcher
-python -m venv venv
-source venv/bin/activate
-pip install -r requirements-worker.txt
-
-# Process a specific race weekend
-python precompute.py 2026 --round 1
-
-# Process only the race session (skip practice/qualifying)
-python precompute.py 2026 --round 1 --session R
-
-# Process an entire season (will take several hours)
-python precompute.py 2025 --skip-existing
-
-# Process multiple years
-python precompute.py 2024 2025 --skip-existing
-```
+Use the Downloads page to enqueue sessions ahead of time (single session, weekend, or season scopes). The queue runs automatically in the backend.
 
 **Timing estimates:**
 - A single session (e.g. one race) takes **1-3 minutes**
@@ -256,7 +220,7 @@ The app also includes a background task that automatically checks for and proces
 
 ## Acknowledgements
 
-This project is powered by [FastF1](https://github.com/theOehrly/Fast-F1), an open-source Python library for accessing Formula 1 timing and telemetry data. FastF1 is the original inspiration and data source for this project - without it, none of this would be possible.
+Thanks to the broader motorsport telemetry/open-source community and tools that informed earlier iterations of this project.
 
 ## License
 

backend/Dockerfile

@@ -13,7 +13,12 @@ COPY --from=builder /out/f1-go-backend /f1-go-backend
 
 ENV PORT=8000
 ENV DATA_DIR=/data
-ENV PROCESSOR_MODE=go
+ENV SQLITE_PATH=/data/f1.db
+ENV SQLITE_BUSY_TIMEOUT_MS=5000
+ENV REPLAY_CHUNK_FRAMES=256
+ENV TELEMETRY_CHUNK_SAMPLES=512
+ENV PROCESS_CHUNK_CODEC=protobuf
+ENV PROCESS_RAW_MIN_DT_SECONDS=0.10
 
 EXPOSE 8000
 

backend/downloads.go

@@ -1,16 +1,13 @@
 package main
 
 import (
-	"bufio"
 	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
 	"log"
 	"net/http"
 	"os"
-	"os/exec"
-	"path/filepath"
 	"sort"
 	"strconv"
 	"strings"
@@ -112,7 +109,6 @@ type queueSnapshot struct {
 
 type downloadManager struct {
 	app         *app
-	statePath   string
 	timeout     time.Duration
 	maxAttempts int
 	retryBase   time.Duration
@@ -131,7 +127,6 @@ func newDownloadManager(app *app, dataDir string) *downloadManager {
 
 	m := &downloadManager{
 		app:         app,
-		statePath:   filepath.Join(dataDir, "downloads", "state.json"),
 		timeout:     time.Duration(timeoutMinutes) * time.Minute,
 		maxAttempts: maxAttempts,
 		retryBase:   time.Duration(retryBaseSec) * time.Second,
@@ -343,7 +338,10 @@ func cloneJob(job *downloadJob) *downloadJob {
 }
 
 func (m *downloadManager) loadState() {
-	b, err := os.ReadFile(m.statePath)
+	if m.app.store == nil {
+		return
+	}
+	b, err := m.app.store.LoadDownloadStateBlob(context.Background())
 	if err != nil {
 		return
 	}
@@ -410,24 +408,16 @@ func (m *downloadManager) persistLocked() {
 		Active: m.active,
 		Recent: m.recent,
 	}
-	b, err := json.MarshalIndent(state, "", "  ")
+	b, err := json.Marshal(state)
 	if err != nil {
 		log.Printf("downloads: persist marshal failed: %v", err)
 		return
 	}
-
-	dir := filepath.Dir(m.statePath)
-	if err := os.MkdirAll(dir, 0o755); err != nil {
-		log.Printf("downloads: persist mkdir failed: %v", err)
+	if m.app.store == nil {
 		return
 	}
-	tmp := m.statePath + ".tmp"
-	if err := os.WriteFile(tmp, b, 0o644); err != nil {
-		log.Printf("downloads: persist write failed: %v", err)
-		return
-	}
-	if err := os.Rename(tmp, m.statePath); err != nil {
-		log.Printf("downloads: persist rename failed: %v", err)
+	if err := m.app.store.SaveDownloadStateBlob(context.Background(), b); err != nil {
+		log.Printf("downloads: persist sqlite failed: %v", err)
 	}
 }
 
@@ -653,106 +643,29 @@ func (m *downloadManager) failedKeysByYear(year int) []sessionKey {
 }
 
 func (a *app) runProcessSessionWorker(ctx context.Context, year, round int, sessionType string, onStatus func(string)) error {
-	if a.processor != nil {
-		return a.processor.ProcessSession(ctx, year, round, sessionType, onStatus)
-	}
-	args := []string{
-		a.workerPath,
-		"process-session",
-		"--year", strconv.Itoa(year),
-		"--round", strconv.Itoa(round),
-		"--type", strings.ToUpper(strings.TrimSpace(sessionType)),
-	}
-	cmd := exec.CommandContext(ctx, a.pythonBin, args...)
-	cmd.Env = os.Environ()
-
-	stdout, err := cmd.StdoutPipe()
-	if err != nil {
-		return err
+	if a.processor == nil {
+		return errors.New("session processor is not initialized")
 	}
-	stderr, err := cmd.StderrPipe()
-	if err != nil {
-		return err
-	}
-
-	if err := cmd.Start(); err != nil {
-		return err
-	}
-
-	var wg sync.WaitGroup
-	var mu sync.Mutex
-	var statusErr error
-
-	wg.Add(1)
-	go func() {
-		defer wg.Done()
-		s := bufio.NewScanner(stderr)
-		buf := make([]byte, 0, 64*1024)
-		s.Buffer(buf, 1024*1024)
-		for s.Scan() {
-			log.Printf("worker stderr: %s", s.Text())
-		}
-	}()
-
-	wg.Add(1)
-	go func() {
-		defer wg.Done()
-		s := bufio.NewScanner(stdout)
-		buf := make([]byte, 0, 64*1024)
-		s.Buffer(buf, 1024*1024)
-		for s.Scan() {
-			line := s.Bytes()
-			var evt map[string]any
-			if err := json.Unmarshal(line, &evt); err != nil {
-				log.Printf("worker out: %s", string(line))
-				continue
-			}
-			typ := asString(evt["type"])
-			if typ == "status" && onStatus != nil {
-				onStatus(asString(evt["message"]))
-			}
-			if typ == "error" {
-				mu.Lock()
-				statusErr = errors.New(defaultString(asString(evt["message"]), "worker failed"))
-				mu.Unlock()
-			}
-		}
-	}()
-
-	waitErr := cmd.Wait()
-	wg.Wait()
-
-	if errors.Is(ctx.Err(), context.DeadlineExceeded) {
-		return ctx.Err()
-	}
-
-	mu.Lock()
-	errCopy := statusErr
-	mu.Unlock()
-	if errCopy != nil {
-		return errCopy
-	}
-	if waitErr != nil {
-		if ctx.Err() != nil {
-			return ctx.Err()
-		}
-		return waitErr
-	}
-	return nil
+	return a.processor.ProcessSession(ctx, year, round, sessionType, onStatus)
 }
 
 func (a *app) isSessionDownloaded(year, round int, sessionType string) bool {
-	base := filepath.Join("sessions", strconv.Itoa(year), strconv.Itoa(round), strings.ToUpper(strings.TrimSpace(sessionType)))
-	return a.fileExists(filepath.Join(base, "replay.json")) && a.fileExists(filepath.Join(base, "info.json"))
+	if a.store == nil {
+		return false
+	}
+	ready, _, err := a.store.SessionReady(context.Background(), year, round, strings.ToUpper(strings.TrimSpace(sessionType)))
+	return err == nil && ready
 }
 
 func (a *app) sessionDataUpdatedAt(year, round int, sessionType string) (time.Time, bool) {
-	p := filepath.Join(a.dataDir, "sessions", strconv.Itoa(year), strconv.Itoa(round), strings.ToUpper(strings.TrimSpace(sessionType)), "replay.json")
-	st, err := os.Stat(p)
-	if err != nil {
+	if a.store == nil {
+		return time.Time{}, false
+	}
+	_, updated, err := a.store.SessionReady(context.Background(), year, round, strings.ToUpper(strings.TrimSpace(sessionType)))
+	if err != nil || updated.IsZero() {
 		return time.Time{}, false
 	}
-	return st.ModTime(), true
+	return updated, true
 }
 
 func (a *app) sessionDownloadStatus(year, round int, sessionType string) sessionDownloadStatus {

backend/go.mod

@@ -3,3 +3,19 @@ module f1replaytiming/backend
 go 1.26.0
 
 require github.com/gorilla/websocket v1.5.3
+
+require (
+	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/klauspost/compress v1.18.5 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/ncruces/go-strftime v1.0.0 // indirect
+	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	golang.org/x/sys v0.42.0 // indirect
+	google.golang.org/protobuf v1.33.0 // indirect
+	modernc.org/libc v1.70.0 // indirect
+	modernc.org/mathutil v1.7.1 // indirect
+	modernc.org/memory v1.11.0 // indirect
+	modernc.org/sqlite v1.48.0 // indirect
+)

backend/go.sum

@@ -1,2 +1,29 @@
+github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
+github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
 github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
+github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
+github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
+golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
+google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
+modernc.org/libc v1.70.0 h1:U58NawXqXbgpZ/dcdS9kMshu08aiA6b7gusEusqzNkw=
+modernc.org/libc v1.70.0/go.mod h1:OVmxFGP1CI/Z4L3E0Q3Mf1PDE0BucwMkcXjjLntvHJo=
+modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
+modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
+modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
+modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
+modernc.org/sqlite v1.48.0 h1:ElZyLop3Q2mHYk5IFPPXADejZrlHu7APbpB0sF78bq4=
+modernc.org/sqlite v1.48.0/go.mod h1:hWjRO6Tj/5Ik8ieqxQybiEOUXy0NJFNp2tpvVpKlvig=