Skip to content

SELFHOST.md

Latest commit

 

History

History
219 lines (166 loc) · 9.86 KB

SELFHOST.md

File metadata and controls

219 lines (166 loc) · 9.86 KB
Raw

Self-hosting Sandchest

Sandchest has three production processes: a Next.js control plane, a durable Postgres queue consumer, and a separate warm NVIDIA Parakeet inference service. Private S3-compatible storage holds retained media. Postgres stores tenant ownership, hashed API keys, durable jobs, transcripts, and usage.

Hardware

  • Apple Silicon development runs Parakeet with MLX and unified memory.
  • NVIDIA production requires a CUDA-capable GPU with sufficient VRAM.
  • The example AWS instance is g5.xlarge; check regional capacity and pricing.
  • FFmpeg and FFprobe provide format decoding and measured media duration.
  • Parakeet TDT v2 is English-only; automatic and non-English requests use a separately warmed Parakeet TDT v3 model.

Apple Silicon throughput does not predict CUDA production performance. Benchmark the actual model, GPU, concurrency, and representative audio before sizing.

Native Apple Silicon

Install Bun 1.4, Rust (edition 2024), Xcode command-line tools, and FFmpeg:

bun install
bun run db:migrate
bun run worker:inference

In another terminal:

bun run dev

Without DATABASE_URL, embedded Postgres is persisted in .sandchest/postgres and local objects in .sandchest/storage. Both are ignored by Git. Development claims queued jobs inside the web process to avoid opening embedded Postgres from two separate processes.

When DATABASE_URL points to external Postgres, inline processing defaults to disabled. Start the queue consumer in a third terminal, using the same database, storage, inference URL, and token settings as the web process:

bun run worker

All three processes must run: web, queue consumer, and native inference. Without the queue consumer, the API accepts jobs but does not process them. Docker Compose already includes a separate queue service. Do not run this extra consumer with the embedded database described above.

Model settings:

SANDCHEST_MODEL=nvidia/parakeet-tdt-0.6b-v2
SANDCHEST_MULTILINGUAL_MODEL=nvidia/parakeet-tdt-0.6b-v3
SANDCHEST_DEVICE=metal
SANDCHEST_INFERENCE_URL=http://127.0.0.1:8765

Startup prepares the checksummed native SDK and pinned model weights without Python. Both models are always warmed. Worker health is available at /health on port 8765; /ready returns HTTP 503 until the model worker is usable. Complete application health is available at /api/health on port 3000. The inference-independent /api/health/live endpoint keeps the dashboard and login reachable while models warm up or recover. Worker health reports both genuinely loaded model identifiers so multilingual readiness can be checked before use.

Docker Compose with an NVIDIA GPU

Install Docker, Docker Compose, current NVIDIA drivers, and NVIDIA Container Toolkit. Verify GPU passthrough, then put strong independent secrets in a git-ignored .env file or export them into the environment:

BETTER_AUTH_SECRET=at-least-32-random-characters
SANDCHEST_API_KEY_PEPPER=a-different-32-or-more-random-characters
SANDCHEST_INFERENCE_TOKEN=a-third-independent-random-secret
POSTGRES_PASSWORD=a-strong-private-database-password
MINIO_ROOT_PASSWORD=a-strong-private-object-store-password
RESEND_API_KEY=your-existing-resend-key
EMAIL_FROM=Sandchest <auth@your-verified-domain.example>
NEXT_PUBLIC_APP_URL=https://speech.your-domain.example
SANDCHEST_SELF_HOSTED=true

Start the complete stack:

docker compose config --quiet
docker compose up --build

Compose includes Postgres 17, private MinIO storage, bucket initialization, a native Rust/MLX CUDA Parakeet worker, the Next.js control plane, and an independent queue consumer. Model artifacts persist in a dedicated volume.

Compose refuses to start when any required signing, database, object-storage, or inference secret is missing. The application binds to localhost by default; set SANDCHEST_BIND_ADDRESS only when an intentional public bind is protected by a properly configured HTTPS reverse proxy.

Docker on Apple Silicon cannot provide an NVIDIA GPU; use native MLX on macOS. The CUDA image targets A10/SM86. Its runtime, memory use, and throughput still require validation on NVIDIA hardware before a production rollout. Explicit self-hosted mode permits operation without Autumn. Production OTP codes are never logged: configure a verified Resend sender for real user login.

Postgres and PlanetScale

Configure an independent PostgreSQL connection and run committed migrations:

bun run db:migrate

Never point Sandchest at Cap's MySQL production database. Cap access is permitted only through the reader-role benchmark collector. Configure database TLS, backups, recovery testing, and isolated credentials for Sandchest.

Run split production services with:

bun run worker:inference
bun run worker
bun run build
bun run start

Set SANDCHEST_INLINE_WORKER=false for external Postgres. Optional SQS signals wake workers, but authoritative claims, retries, leases, and usage remain in Postgres. Unsent Autumn usage events remain durable and are retried by the queue consumer.

Storage and retention

SANDCHEST_STORAGE_DRIVER=s3
SANDCHEST_S3_BUCKET=your-private-audio-bucket
AWS_REGION=us-east-1
SANDCHEST_AUDIO_RETENTION_HOURS=24

Use workload IAM roles, private buckets, and server-side encryption. Object keys are tenant-prefixed. Queue workers remove expired media, and AWS bucket lifecycle rules provide a secondary cleanup backstop. Transcript deletion is a separate database retention decision.

AWS infrastructure

The infra/aws directory is valid Terraform 1.7+ and OpenTofu. It provisions:

  • Two-AZ VPC, public/private subnets, internet gateway, and NAT.
  • Public HTTPS load balancer with a mandatory ACM certificate.
  • ECS/Fargate application and queue-consumer tasks.
  • ECS/EC2 NVIDIA GPU capacity with managed autoscaling.
  • Supported Amazon Linux 2023 GPU-optimized ECS hosts with GPU registration enabled.
  • Private DNS between the control plane, queue, and inference service.
  • Encrypted private S3 storage with retention lifecycle rules.
  • Encrypted SQS wake-up queue and dead-letter queue.
  • ECR repositories with image scanning.
  • Secrets Manager, scoped IAM permissions, and IMDSv2-only GPU instances.
  • CloudWatch logs, ECS Container Insights, and queue-age monitoring.
  • Existing externally managed PlanetScale Postgres.

Copy terraform.tfvars.example to a private, ignored terraform.tfvars. Terraform state contains sensitive input: configure an encrypted remote backend with restricted access and locking before production deployment.

tofu -chdir=infra/aws init
tofu -chdir=infra/aws fmt -check
tofu -chdir=infra/aws validate
tofu -chdir=infra/aws plan
tofu -chdir=infra/aws apply

Build and push immutable web and GPU-worker images to the ECR repositories. Point public DNS to the ALB hostname, verify the certificate, then call:

curl -fsS https://speech.your-domain.example/api/health

Do not apply, publish, or create AWS resources until the target account, region, domain, certificate, secrets, GPU quota, and spending envelope are authorized.

Scaling and latency

Before routing Cap traffic to a deployment, complete the Cap replacement gates. The current Parakeet worker does not cover Cap's full language list. Keep unverified routes on AssemblyAI, and verify the actual full-file/live workflows, durable queue, load/recovery behavior and rollback before enabling a canary. A local benchmark is not deployed evidence.

  • Keep one or more GPU instances warm when low latency is required.
  • Budget VRAM and model-cache storage for both the English and multilingual models when automatic language detection is enabled.
  • Automatic English refinement can be disabled with SANDCHEST_ENGLISH_REFINEMENT=false when throughput is preferred over its measured transcription-quality improvements.
  • Scale Postgres queue consumers independently from inference capacity.
  • Set SANDCHEST_QUEUE_CONCURRENCY to the measured durable-job parallelism and SANDCHEST_MAX_UPLOAD_BYTES to a size that fits the web and worker memory budgets.
  • Each native worker permits one inference and at most eight pending uploads/jobs. Increase capacity with independent workers only after measuring VRAM and throughput.
  • Track queue age, CPU, GPU utilization, audio duration, and p50/p95/p99 latency.
  • The benchmark runner automatically samples real utilization and VRAM when nvidia-smi is available; Apple Silicon reports no fabricated GPU telemetry.
  • The included autoscaler uses ECS CPU as a conservative proxy; add actual GPU telemetry before introducing GPU-utilization scaling policies.
  • Long recordings use 120-second windows with a 15-second overlap. Decoding is capped at three hours of PCM; measure memory at that limit before sizing.
  • Validate fragmented/live MP4 inputs separately from ordinary complete files.
  • Tune SANDCHEST_COMPATIBILITY_POLL_WAIT_MS to avoid unnecessary official-SDK three-second polling sleeps without holding web connections indefinitely.
  • Do not claim faster or better results without reproducible measurements.

Operating costs

Estimate from observed workload and current regional infrastructure prices:

GPU cost per audio hour =
  GPU dollars per wall-clock hour / observed audio-hours per wall-clock hour

Monthly infrastructure =
  GPU + Fargate + NAT + ALB + S3 + SQS + Secrets Manager
  + CloudWatch + PlanetScale Postgres + data transfer + email

The supplied historical Cap AssemblyAI reference was approximately 2,426 audio hours for $506.87, or $0.209 per audio hour. It is a comparison baseline, not a current price quote, production guarantee, or claim of savings.

Pass measured hourly compute cost when running a real benchmark:

SANDCHEST_GPU_HOURLY_USD=your_actual_hourly_cost \
SANDCHEST_BENCHMARK_LOCAL=true bun run benchmark

AssemblyAI-generated transcripts provide agreement, not ground-truth WER. Report WER only when a sample's words have been explicitly verified by a human.