Add Makefile targets, cloud deployment workflow, and local file endpoints#236
Open
jpatrickfarrell wants to merge 22 commits into
Open
Add Makefile targets, cloud deployment workflow, and local file endpoints#236jpatrickfarrell wants to merge 22 commits into
jpatrickfarrell wants to merge 22 commits into
Conversation
…Makefile quick-start section, and created comprehensive docs/deploy/DEPLOY_GCP.md. 📁 FILES CREATED/MODIFIED: - Makefile — 12 targets: build, run, stop, clean, auth, repo, push, deploy, logs, describe, test, help - docs/deploy/DEPLOY_GCP.md — Full 9-step guide covering project setup, APIs, service accounts, storage buckets, Artifact Registry, deployment, Cloud Run Jobs, monitoring, cost optimization, and troubleshooting - README.md — Added "Quick Start with Makefile" section and updated GCP installation link to point to new guide
…it API. 📁 FILES CREATED (14 total): Python CLI: - tools/nca.py — Single-file CLI with 18 subcommands covering all API endpoints. Zero external dependencies (uses only urllib and argparse). Config via NCA_API_URL + NCA_API_KEY env vars. Claude Skill: - tools/claude-skill/SKILL.md — Skill definition with trigger routing - tools/claude-skill/Workflows/Transcribe.md — Transcribe/translate media - tools/claude-skill/Workflows/ConvertMedia.md — Format conversion - tools/claude-skill/Workflows/CaptionVideo.md — Auto-caption videos - tools/claude-skill/Workflows/VideoOps.md — Trim, cut, split, concat, thumbnail - tools/claude-skill/Workflows/Screenshot.md — Webpage screenshots - tools/claude-skill/Workflows/Metadata.md — Media file info - tools/claude-skill/Workflows/Download.md — Download via yt-dlp - tools/claude-skill/Workflows/Silence.md — Silence detection - tools/claude-skill/Workflows/FFmpeg.md — Custom FFmpeg pipelines - tools/claude-skill/Workflows/Upload.md — S3/GCP uploads - tools/claude-skill/Workflows/TestConnect.md — Connectivity testing Installer: - tools/install-skill.sh — Symlinks skill to ~/.claude/skills/NCAToolkit/, prompts for env vars, adds them to shell profile
…setup command that validates credentials before saving, a config command to inspect settings, and updated the install script + skill to use the new system. 📁 CHANGES: Modified: - tools/nca.py — Added read_config_file(), write_config_file(), updated get_config() with config file fallback, added setup and config commands, supports --profile for multiple environments - tools/install-skill.sh — Now runs python3 nca.py setup instead of manual env var prompting - tools/claude-skill/SKILL.md — Updated prerequisites and routing table Created: - tools/claude-skill/Workflows/Setup.md — Auth/setup workflow for the skill - tools/config.example — Example config file showing the format Config flow: 1. python3 nca.py setup → prompts for URL + key → validates against API → saves to ~/.nca-toolkit/config (600 perms) 2. All commands check: env vars first → config file second 3. Multiple profiles supported: python3 nca.py setup --profile staging
…e down, added make setup for API credentials, and updated help menu with a new SETUP section.
…ress Enter to accept it when running locally. When you deploy to Cloud Run, re-run make setup and enter the Cloud Run URL instead.
…Ls for local vs Cloud Run, and provides a python3 -c "import secrets; ..." command to generate an API key if they don't have one yet.
- make setup — Generates .env from .env.example with an auto-generated API key. This is for setting up the server to run. - make connect — Connects the CLI tools to a running API instance (local or Cloud Run). Saves credentials to ~/.nca-toolkit/config. The flow is now: make setup → make up → make connect → use the API.
1. CLI was sending Content-Type: application/json on GET requests with no body — Flask sees the JSON content type, tries to parse the empty body, and returns 400. Fixed api_request() to only add Content-Type when there's a payload. 2. Same issue in the connect validation request — removed the unnecessary header. 3. make test was using POST instead of GET — the /v1/toolkit/test route only accepts GET. Try python3 tools/nca.py test again now.
…d, 0 route files touched. 📁 CAPTURE: The key insight — file:// URIs pass jsonschema format: uri validation (because FormatChecker isn't used), so zero schema/route changes were needed. download_file() and upload_file() were the only two intervention points. ➡️ NEXT: 1. Run make setup to generate .env (if not already done) 2. Run make up-local to start with volume mounts 3. Place a test file in ./local/input/ 4. Test: python3 tools/nca.py metadata --file ./local/input/test.mp4 5. Check ./local/output/ for results Files changed: ┌─────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ File │ What Changed │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Dockerfile │ Added /data/input and /data/output dirs with appuser ownership │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ services/file_management.py │ download_file() handles file:// URIs — copies local file instead of HTTP download, with path traversal protection │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ services/cloud_storage.py │ Added LocalStorageProvider that saves to /data/output; get_storage_provider() falls back to local instead of raising │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ config.py │ Added LOCAL_OUTPUT_PATH env var │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ .env.example │ Documented LOCAL_OUTPUT_PATH │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Makefile │ Added up-local target with -v mounts for ./local/input and ./local/output │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ tools/nca.py │ Added --file/-f flag to 11 commands; translates to file:///data/input/<basename>; translates file:// output paths for display │ ├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ .gitignore │ Added local/ to prevent committing user files │
…s remote API.
What was built:
┌─────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ File │ Change │
├─────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ routes/v1/files/upload.py │ New — accepts multipart POST /v1/files/upload, saves to configured storage, returns URL │
├─────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ routes/v1/files/__init__.py │ New — package init │
├─────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tools/nca.py │ Modified — resolve_file() auto-detects: local API → file:// URI, remote API → uploads via multipart then returns cloud URL │
├─────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Makefile │ Modified — added --use-http2 to deploy command │
└─────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
How it works end-to-end on Cloud Run:
python3 tools/nca.py transcribe --file ./my-video.mp4
↓ CLI detects remote API (not localhost)
Step 1: POST /v1/files/upload (multipart) → "https://bucket.../abc123.mp4"
Step 2: POST /v1/media/transcribe {"media_url": "https://bucket.../abc123.mp4"}
↓ Returns transcription result
On the 32 MB limit:
- Cloud Run has a hard 32 MB request body limit — cannot be increased via configuration
- Adding --use-http2 to the deploy command may bypass this (some users report success, not officially documented)
- For files > 32 MB, you'll need Option C (signed URLs) where the CLI uploads directly to the bucket bypassing Cloud Run entirely. The endpoint architecture is designed to
make this easy to add later
Updated workflow files (7): - ConvertMedia.md — Added --file option for both convert and convert-mp3, local file examples - CaptionVideo.md — Added --file option with local file examples, mutual exclusivity note - VideoOps.md — Added --file for trim/cut/split/thumbnail and --files for concat, with local examples for all 5 commands - Metadata.md — Added --file option and local file example - Silence.md — Added --file option and local file example - Upload.md — Added new "File Upload (for processing)" section documenting the manual upload command and 32 MB Cloud Run limit - FFmpeg.md — Added local file guidance and file:///data/input/ volume mount example Previously updated (2): - SKILL.md — Local File Support section, Input Options table, updated examples - Transcribe.md — --file option with local file examples No changes needed (3): - Screenshot.md — Takes webpage URLs, not media files - Download.md — Downloads from URLs, no local file input needed - Setup.md / TestConnect.md — Configuration commands, not media processing
Server-side fix (1 file):
- services/file_management.py — Added /data/output to ALLOWED_LOCAL_DIRS so the upload-then-process flow works on localhost
CLI overhaul (1 file, major rewrite):
- tools/nca.py — Key changes:
- resolve_file() — Simplified to always upload via multipart POST (removed is_local_api() branch). Any file path on disk now works.
- download_output() — New function that downloads output files (HTTP URLs) or copies from ./local/output/ (file:// URLs)
- extract_urls() — Recursively finds URLs in API responses for auto-download
- handle_output() — Replaces print_result(). Text commands print text directly; file commands download and print local path
- poll_job() — New function for polling async job status (addresses your concern about long-running tasks)
- Global --json flag — Full JSON response for any command
- Global --output-dir / -o flag — Control where output files are saved
- Removed translate_output_paths() and print_result() (dead code)
- Updated all --file help text: "uploaded automatically" instead of "must be in ./local/input/"
Documentation (9 files):
- SKILL.md — Updated with new workflow, global flags, generic file paths
- Transcribe.md, ConvertMedia.md, CaptionVideo.md, VideoOps.md, Metadata.md, Silence.md, Upload.md, FFmpeg.md — All updated to use generic paths (~/videos/, ~/recordings/)
and document output download behavior
Regarding your concern about long-running tasks: The CLI currently blocks synchronously (10-minute timeout). I added a poll_job() function that's ready to use — it can poll
the job status endpoint until completion. For the Claude Code workflow specifically, the synchronous blocking approach actually works well: Claude Code will wait for the
CLI to finish and read the stdout. The 10-minute timeout should cover most transcription/conversion jobs. If you want, we can add a --poll flag that submits with a webhook
and polls, which would be better for very long tasks — but that can be a follow-up.
…mands for polling. Here's how the workflow works now, Patrick: Synchronous (default — fine for short tasks): python3 tools/nca.py transcribe --file ~/audio.mp3 # Blocks, prints transcription text when done Background (for long-running tasks): # Step 1: Kick it off — returns immediately with a job ID python3 tools/nca.py transcribe --file ~/audio.mp3 --bg # Output: 550e8400-e29b-41d4-a716-446655440000 # Step 2: Check if it's done python3 tools/nca.py status 550e8400-e29b-41d4-a716-446655440000 # Output: running (or: queued, done, failed) # Step 3: Wait for completion and get the result python3 tools/nca.py wait 550e8400-e29b-41d4-a716-446655440000 # Polls every 5s, then prints the transcription text when done For Claude Code, the typical pattern would be: 1. Run with --bg, capture the job ID from stdout 2. Do other work 3. Run wait <job_id> when ready for the result The --bg flag works on every command (transcribe, convert, caption, etc.) — zero server-side changes needed.
…erve Docker build cache for Whisper model and Playwright
…transcribe output file support. Changes made: 1. upload-s3 and upload-gcp — added parents=[shared] so --json, -o, --bg work on all commands 2. --output-file flag on transcribe — writes transcription text (and SRT if requested) to a file, prints the path Usage: python3 tools/nca.py transcribe --file ~/audio.mp3 --output-file transcript.txt python3 tools/nca.py transcribe --file ~/audio.mp3 --srt --output-file transcript.txt python3 tools/nca.py transcribe --file ~/audio.mp3 --json # now works after subcommand
…flow docs + SKILL.md with --json, --bg, --output-file flags. Changes: - tools/nca.py — cmd_transcribe: moved --output-file check before --json early return. When both flags are set, writes JSON to the file via json.dump(). - tools/claude-skill/SKILL.md — Added wait command, --bg to Global Flags, new Transcribe-Specific Flags section, new examples for --output-file, --json --output-file, and --bg. - tools/claude-skill/Workflows/Transcribe.md — Full rewrite: added --output-file, --json, --bg options and examples. - All other workflow docs (ConvertMedia, CaptionVideo, VideoOps, Metadata, Screenshot, Download, Silence, FFmpeg, Upload) — Added --json and --bg documentation.
…/ outputs via HTTP.
What changed:
1. New routes/v1/files/content.py — GET endpoint that serves files from /data/output/ with path traversal protection (realpath + ALLOWED_DIRS whitelist). Requires API key
auth.
2. tools/nca.py download_output() — When a file:// URL is returned and the local volume mount path doesn't exist, falls back to downloading via GET
/v1/files/content?path=/data/output/xxx.mp4.
Flow:
1. CLI gets file:///data/output/uuid_captioned.mp4 from API
2. Tries ./local/output/uuid_captioned.mp4 (volume mount) — if exists, copies
3. If not, calls GET {api_url}/v1/files/content?path=/data/output/uuid_captioned.mp4 — downloads the file over HTTP
To test: Rebuild the container (make down && make up) and re-run the caption command.
…mis-match. What was wrong: 1. No env vars passed to Cloud Run — config.py raises ValueError on import if API_KEY is missing. The container crashed immediately on every startup attempt. 2. --use-http2 flag — Gunicorn sync workers don't support HTTP/2 (h2c). Cloud Run's health check couldn't communicate with the container. What changed in Makefile: - Added .env file check before deploy - Converts .env → temp YAML file, passes via --env-vars-file (handles commas in values safely) - Removed --use-http2 - Cleans up temp file after deploy The real blocker was architecture mismatch — arm64 image on amd64 Cloud Run. Two fixes applied: 1. --platform linux/amd64 on docker build — forces cross-compilation for Cloud Run's x86_64 2. --no-use-http2 on gcloud run deploy — clears the stuck h2c port protocol from the first deployment Heads up: The rebuild will be slower than usual because Docker is cross-compiling (QEMU emulation for amd64 on your arm64 Mac). The FFmpeg compile step especially will take a while.
New targets: ┌───────────────────┬─────────────────────────────────────────────────────────────────────────────────────┐ │ Command │ What it does │ ├───────────────────┼─────────────────────────────────────────────────────────────────────────────────────┤ │ make cloud-build │ Builds image remotely on Cloud Build (native amd64, 32-core machine, 30min timeout) │ ├───────────────────┼─────────────────────────────────────────────────────────────────────────────────────┤ │ make cloud-deploy │ Cloud Build + deploy to Cloud Run (recommended for Apple Silicon) │ └───────────────────┴─────────────────────────────────────────────────────────────────────────────────────┘ Existing targets unchanged: ┌─────────────┬──────────────────────────────────────────────────────────────┐ │ Command │ What it does │ ├─────────────┼──────────────────────────────────────────────────────────────┤ │ make deploy │ Local Docker build + push + deploy (cross-compiles on arm64) │ └─────────────┴──────────────────────────────────────────────────────────────┘ I'd recommend using make cloud-deploy — it avoids the cross-compilation entirely, builds on a beefy 32-core machine, and the image is natively amd64. The FFmpeg compile alone should be significantly faster.
What it does: 1. Enables Cloud Build, Cloud Run, Artifact Registry, and Container Registry APIs 2. Creates the Artifact Registry Docker repo (skips if already exists) 3. Configures Docker auth for pushing images New workflow for a fresh project: make setup # Generate .env with API key make cloud-setup # Enable GCP APIs + create registry make cloud-deploy # Build on Cloud Build + deploy to Cloud Run Run make cloud-setup now, then make cloud-deploy should work.
…s-central1.run.app"
…additions: Makefile targets, cloud deployment workflow (cloud-setup → cloud-deploy), local file upload endpoints (/v1/files/upload, /v1/files/content), local I/O mode, and CLI tool. 2. Updated README.md — Replaced the outdated Quick Start section with current commands (make setup → make cloud-setup → make cloud-deploy), added the two deploy paths table, added local file upload section, and linked to the new What's New doc.
Author
|
@stephengpope I made these changes so we can more easily use Claude Code integrations with the NCA Toolkit. I'd love to know what you think. I'm using it pretty effectively myself. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test plan