fix ytdlp job, add tests, and manual integration scripts

Harden yt-dlp wrapper with stderr capture, single failure reporting,
output-path collision handling, and focused unit tests. Document ytdlp
in README, remove the planning doc, and add manual integration scripts
for ghr (terraform) and a 500 MB HTTP download to USB test storage.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: Tanq16

README.md

@@ -23,6 +23,7 @@ The primary downloaders and their supported aliases are as follows:
 | `github-release` | `ghrelease`, `ghr` | Download a platform-correct release asset for a GitHub repo |
 | `google-drive` | `gdrive`, `gd`, `drive` | Download file/folder from Google drive with API key or OAuth flow authentication |
 | `s3` | - | Multi-threaded download for object, directory, or full AWS S3 bucket |
+| `ytdlp` | `yt-dlp`, `youtube-dl`, `ytdl` | Wraps the `yt-dlp` binary for sites Danzo doesn't natively support (YouTube, etc.) |
 | `resume` | - | Resume downloads from saved interrupted job state |
 | `clean` | - | Clear local cache for interrupted/incomplete downloads |
 
@@ -63,6 +64,11 @@ Following are examples to get started with various flags:
   danzo git "github.com/tanq16/private" --token $(cat /secrets/ghtoken) # (use a PAT; auto-manages for different providers)
   danzo git "github.com/tanq16/private" --ssh "/secrets/gh-ssh.key" # (use an SSH key to authenticate)
   ```
+- Download via `yt-dlp` (for sites Danzo doesn't natively support, like YouTube)
+  ```bash
+  danzo ytdlp "https://www.youtube.com/watch?v=jNQXAC9IVRw" -o me-at-the-zoo.mp4
+  danzo ytdlp "https://vimeo.com/22439234" # (default yt-dlp output template)
+  ```
 
 ## Installation
 
@@ -130,6 +136,7 @@ Follow these links to quickly jump to the relevant provider:
 - [AWS S3 Downloads](#aws-s3-downloads)
 - [GitHub Release Downloads](#github-release-downloads)
 - [Git Repository Cloning](#git-repository-cloning)
+- [yt-dlp Downloads](#yt-dlp-downloads)
 
 ### HTTP(S) Downloads
 
@@ -347,6 +354,33 @@ danzo gitclone github.com/tanq16/private --ssh "/secrets/gh-ssh.key"
 
 </details>
 
+### yt-dlp Downloads
+
+<details>
+<summary>Unfold to read</summary>
+
+For sites Danzo doesn't natively support (YouTube, Vimeo with audio, etc.), the `ytdlp` command wraps the `yt-dlp` binary and streams its progress into the Danzo TUI so it looks and behaves like every other Danzo job.
+
+> ✎ Requires `yt-dlp` to be installed and available on `PATH`. Some downloads (e.g., separate video + audio streams) additionally require `ffmpeg` for the final merge step.
+
+```bash
+danzo ytdlp "https://www.youtube.com/watch?v=jNQXAC9IVRw" -o me-at-the-zoo.mp4
+
+# Without -o, yt-dlp picks its own filename via its default output template.
+danzo ytdlp "https://vimeo.com/22439234"
+```
+
+The wrapper:
+
+- Streams `yt-dlp`'s structured `JSON_PROGRESS:` lines into the highway display so percent/byte counters move in real time.
+- Switches the bar into a "Merging" sub-status when `yt-dlp` reaches the `[Merger]`/`Merging formats` phase (the bar would otherwise be misleading during the ffmpeg merge).
+- Surfaces the failing `ERROR:` line from `yt-dlp`'s stderr in the failure message, so things like a bad URL produce the actual reason rather than a bare `exit status 1`.
+- If the chosen output path already exists, falls back to `name-(1).ext` (same behavior as `http` / `git-clone`).
+
+> ✎ This is intentionally a thin wrapper - any flags beyond `--output/-o` should be configured on the `yt-dlp` side (e.g., via its `--config-location`).
+
+</details>
+
 ## Tips and Notes
 
 - Use `--for-ai` when invoking Danzo from scripts or AI agents that need stable plain-text output.
@@ -365,6 +399,8 @@ Danzo uses issues for everything. Open an issue and I will add an appropriate ta
 
 Danzo uses `ffmpeg` for merging M3U8 stream segments.
 
+Danzo wraps `yt-dlp` for sites it doesn't natively support.
+
 Danzo draws inspiration from [aria2](https://github.com/aria2/aria2).
 
 Lastly, Danzo uses several Go packages referenced within `go.mod` that allow Danzo to be amazing.

integration-tests/01-ghr-terraform.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# Manual integration test: GitHub release download (unauthenticated, public repo).
+# Downloads the latest Terraform release asset for the current OS/arch from hashicorp/terraform.
+set -euo pipefail
+
+readonly DEST="${DANZO_TEST_DEST:-/mnt/usbdrive/danzo-tests}"
+readonly DANZO_BIN="${DANZO:-danzo}"
+
+mkdir -p "$DEST"
+cd "$DEST"
+
+echo "==> ghr integration: hashicorp/terraform -> $DEST"
+echo "    (asset filename is chosen by danzo from the release, e.g. terraform_*_linux_amd64.zip)"
+"$DANZO_BIN" ghr "hashicorp/terraform"
+echo "==> done"

integration-tests/02-http-500mb.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Manual integration test: large HTTP download (unauthenticated).
+# Fetches a 500 MB test file; uses multi-chunk behavior depending on server and danzo flags.
+set -euo pipefail
+
+readonly DEST="${DANZO_TEST_DEST:-/mnt/usbdrive/danzo-tests}"
+readonly DANZO_BIN="${DANZO:-danzo}"
+readonly URL="https://link.testfile.org/500MB"
+readonly OUT_NAME="500mb-testfile.bin"
+
+mkdir -p "$DEST"
+
+echo "==> http integration: 500 MB test file"
+echo "    URL:  $URL"
+echo "    dest: $DEST/$OUT_NAME"
+echo "    (this will take a while depending on bandwidth)"
+"$DANZO_BIN" http "$URL" -o "$DEST/$OUT_NAME"
+echo "==> done"

integration-tests/README.md

@@ -0,0 +1,42 @@
+# Integration tests (manual)
+
+These are **not** Go unit tests. They are **optional, manually run** checks against live networks and real endpoints. Run them when you want to validate downloaders end-to-end on your machine.
+
+## Requirements
+
+- A built `danzo` binary on `PATH`, or set `DANZO` to the full path of the binary.
+- The destination directory must exist or be mountable. By default scripts use:
+
+  **`/mnt/usbdrive/danzo-tests`**
+
+  Override with `DANZO_TEST_DEST` if that path is wrong on your system.
+
+- Enough free space on the destination (the 500 MB HTTP test alone needs ~500 MB).
+
+## Unauthenticated downloads
+
+| Script | What it exercises |
+|--------|-------------------|
+| [`01-ghr-terraform.sh`](./01-ghr-terraform.sh) | `danzo ghr` — latest **Hashicorp Terraform** release asset for this OS/arch (public GitHub API, no token). |
+| [`02-http-500mb.sh`](./02-http-500mb.sh) | `danzo http` — **500 MB** file from `link.testfile.org`. |
+
+### How to run
+
+From the repo root:
+
+```bash
+chmod +x integration-tests/*.sh   # once
+./integration-tests/01-ghr-terraform.sh
+./integration-tests/02-http-500mb.sh
+```
+
+Or with a custom binary or destination:
+
+```bash
+DANZO=/path/to/danzo DANZO_TEST_DEST=/mnt/other/volume/tests ./integration-tests/01-ghr-terraform.sh
+```
+
+## Notes
+
+- Scripts use `set -euo pipefail` and abort on failure.
+- Outputs land under `DANZO_TEST_DEST` (default `/mnt/usbdrive/danzo-tests`). Clean old artifacts there yourself when done.

internal/jobs/ytdlp/job.go

@@ -2,8 +2,12 @@ package ytdlpjob
 
 import (
 	"bufio"
+	"bytes"
 	"context"
 	"encoding/json"
+	"fmt"
+	"io"
+	"os"
 	"os/exec"
 	"strconv"
 	"strings"
@@ -12,6 +16,13 @@ import (
 	"github.com/tanq16/danzo/utils"
 )
 
+const (
+	progressPrefix   = "JSON_PROGRESS: "
+	progressTemplate = progressPrefix + `{"downloaded_bytes": "%(progress.downloaded_bytes)s", "total_bytes": "%(progress.total_bytes)s", "total_bytes_estimate": "%(progress.total_bytes_estimate)s", "status": "%(progress.status)s"}`
+)
+
+var ytdlpBinary = "yt-dlp"
+
 type YTDLPProgress struct {
 	DownloadedBytes string `json:"downloaded_bytes"`
 	TotalBytes      string `json:"total_bytes"`
@@ -41,80 +52,130 @@ func (j *YTDLPJob) ID() string {
 func (j *YTDLPJob) Type() string { return "ytdlp" }
 
 func (j *YTDLPJob) Run(ctx context.Context, prog chan<- highway.Progress) error {
-	args := []string{j.URL, "--newline", "--progress-template", `JSON_PROGRESS: {"downloaded_bytes": "%(progress.downloaded_bytes)s", "total_bytes": "%(progress.total_bytes)s", "total_bytes_estimate": "%(progress.total_bytes_estimate)s", "status": "%(progress.status)s"}`}
+	if j.OutputPath != "" && !strings.Contains(j.OutputPath, "%(") {
+		if _, err := os.Stat(j.OutputPath); err == nil {
+			j.OutputPath = utils.RenewOutputPath(j.OutputPath)
+		}
+	}
 
+	args := []string{j.URL, "--newline", "--progress-template", progressTemplate}
 	if j.OutputPath != "" {
 		args = append(args, "-o", j.OutputPath)
 	}
 
-	cmd := exec.CommandContext(ctx, "yt-dlp", args...)
+	cmd := exec.CommandContext(ctx, ytdlpBinary, args...)
 
 	stdout, err := cmd.StdoutPipe()
 	if err != nil {
-		return err
+		return fmt.Errorf("error creating stdout pipe: %v", err)
 	}
+	var stderrBuf bytes.Buffer
+	cmd.Stderr = &stderrBuf
 
 	if err := cmd.Start(); err != nil {
-		return err
+		return fmt.Errorf("error starting yt-dlp: %v", err)
+	}
+
+	streamErr := streamOutput(j.ID(), stdout, prog)
+	waitErr := cmd.Wait()
+
+	if waitErr != nil {
+		return ytdlpError(waitErr, stderrBuf.String())
+	}
+	if streamErr != nil {
+		return fmt.Errorf("error reading yt-dlp output: %v", streamErr)
+	}
+
+	prog <- highway.Progress{JobID: j.ID(), Done: true}
+	return nil
+}
+
+func ytdlpError(waitErr error, stderr string) error {
+	stderr = strings.TrimSpace(stderr)
+	if stderr == "" {
+		return fmt.Errorf("yt-dlp failed: %v", waitErr)
 	}
+	last := lastErrorLine(stderr)
+	return fmt.Errorf("yt-dlp failed: %v: %s", waitErr, last)
+}
+
+func lastErrorLine(stderr string) string {
+	var last string
+	for _, ln := range strings.Split(stderr, "\n") {
+		ln = strings.TrimSpace(ln)
+		if ln == "" {
+			continue
+		}
+		last = ln
+		if strings.HasPrefix(ln, "ERROR:") {
+			return strings.TrimSpace(strings.TrimPrefix(ln, "ERROR:"))
+		}
+	}
+	return last
+}
 
-	scanner := bufio.NewScanner(stdout)
+func streamOutput(jobID string, r io.Reader, prog chan<- highway.Progress) error {
+	scanner := bufio.NewScanner(r)
+	scanner.Buffer(make([]byte, 64*1024), 1024*1024)
 	currentPhase := "Downloading"
 
 	for scanner.Scan() {
 		line := scanner.Text()
 
-		if strings.Contains(line, "[Merger]") || strings.Contains(line, "Merging formats") {
+		if currentPhase != "Merging" && isMergingLine(line) {
 			currentPhase = "Merging"
 			prog <- highway.Progress{
-				JobID:     j.ID(),
+				JobID:     jobID,
 				Type:      highway.ProgressTypeSubStatus,
 				Message:   currentPhase,
-				SubStatus: "ffmpeg consolidating streams...",
+				SubStatus: "ffmpeg consolidating streams",
 			}
 			continue
 		}
 
-		if strings.HasPrefix(line, "JSON_PROGRESS: ") {
-			jsonStr := strings.TrimPrefix(line, "JSON_PROGRESS: ")
-			var p YTDLPProgress
-			if err := json.Unmarshal([]byte(jsonStr), &p); err != nil {
-				continue
-			}
-
-			if currentPhase == "Merging" {
-				continue
-			}
-
-			downBytes, _ := strconv.ParseInt(p.DownloadedBytes, 10, 64)
-			var totalBytes int64
-			if p.TotalBytes != "NA" {
-				totalBytes, _ = strconv.ParseInt(p.TotalBytes, 10, 64)
-			} else if p.TotalBytesEst != "NA" {
-				totalBytes, _ = strconv.ParseInt(p.TotalBytesEst, 10, 64)
-			}
+		if !strings.HasPrefix(line, progressPrefix) {
+			continue
+		}
+		if currentPhase == "Merging" {
+			continue
+		}
 
-			if totalBytes > 0 {
-				prog <- highway.Progress{
-					JobID:   j.ID(),
-					Type:    highway.ProgressTypeProgress,
-					Message: currentPhase,
-					Current: downBytes,
-					Total:   totalBytes,
-					Extra:   utils.FormatBytes(uint64(downBytes)) + "/" + utils.FormatBytes(uint64(totalBytes)),
-				}
-			}
+		current, total, ok := parseProgressJSON(strings.TrimPrefix(line, progressPrefix))
+		if !ok {
+			continue
+		}
+		prog <- highway.Progress{
+			JobID:   jobID,
+			Type:    highway.ProgressTypeProgress,
+			Message: currentPhase,
+			Current: current,
+			Total:   total,
+			Extra:   utils.FormatBytes(uint64(current)) + "/" + utils.FormatBytes(uint64(total)),
 		}
 	}
+	return scanner.Err()
+}
 
-	err = cmd.Wait()
-	if err != nil {
-		prog <- highway.Progress{JobID: j.ID(), Done: true, Error: err, ErrMsg: err.Error()}
-		return err
-	}
+func isMergingLine(line string) bool {
+	return strings.Contains(line, "[Merger]") || strings.Contains(line, "Merging formats")
+}
 
-	prog <- highway.Progress{JobID: j.ID(), Done: true}
-	return nil
+func parseProgressJSON(s string) (current, total int64, ok bool) {
+	var p YTDLPProgress
+	if err := json.Unmarshal([]byte(s), &p); err != nil {
+		return 0, 0, false
+	}
+	current, _ = strconv.ParseInt(p.DownloadedBytes, 10, 64)
+	switch {
+	case p.TotalBytes != "" && p.TotalBytes != "NA":
+		total, _ = strconv.ParseInt(p.TotalBytes, 10, 64)
+	case p.TotalBytesEst != "" && p.TotalBytesEst != "NA":
+		total, _ = strconv.ParseInt(p.TotalBytesEst, 10, 64)
+	}
+	if total <= 0 {
+		return 0, 0, false
+	}
+	return current, total, true
 }
 
 type ytdlpJobState struct {