Troubleshooting Guide

This guide helps resolve common issues when using go-broadcast. It combines systematic diagnostic procedures with solutions for specific problems.

Quick Reference

Issue Type	Quick Check	Common Fix
Authentication	`gh auth status`	`gh auth login`
Configuration	`go-broadcast validate`	Fix YAML syntax
API Rate Limits	`gh api rate_limit`	Wait or reduce concurrency
Git Operations	`ssh -T git@github.com`	Check SSH keys
Performance	`go-broadcast sync --json`	Enable debug logging
File Sync	`go-broadcast sync --dry-run`	Check file mappings

Quick Reference
Systematic Troubleshooting
Authentication Issues
Configuration Problems
GitHub API Issues
Git Operations
File Synchronization
Directory Synchronization
Performance Issues
Getting Help
Debugging
Related Documentation

Systematic Troubleshooting

Follow this systematic approach for efficient problem resolution:

1. Information Gathering

# Collect system diagnostics
go-broadcast diagnose > diagnostics-$(date +%Y%m%d-%H%M%S).json

# Check basic connectivity
gh auth status
git --version
echo $GITHUB_TOKEN | cut -c1-10

2. Initial Assessment

What operation was being performed?
What error message was displayed?
When did the issue start occurring?
Has this worked before?

3. Log Collection

# Enable comprehensive logging
go-broadcast sync -vvv --debug-git --debug-api --debug-transform --debug-config --debug-state 2> full-debug.log

# Save with timestamp
DEBUG_LOG="debug-$(date +%Y%m%d-%H%M%S).log"
go-broadcast sync --json -vv --debug-git --debug-api 2> "$DEBUG_LOG"

Authentication Issues

"gh: To get started with GitHub CLI, please run: gh auth login"

Problem: GitHub CLI is not authenticated.

Solution:

# Authenticate with GitHub CLI
gh auth login

# Verify authentication
gh auth status

Alternative: Set a GitHub token directly:

# Use GH_PAT_TOKEN as the preferred token (checked first)
export GH_PAT_TOKEN="your_personal_access_token"
# Or use GITHUB_TOKEN as a fallback
export GITHUB_TOKEN="your_personal_access_token"

"gh: Not Found (HTTP 404)"

Problem: Repository doesn't exist, or you don't have access.

Solutions:

Check repository name format: Must be owner/repository, not just repository
Verify access: Ensure you have read access to source repo and write access to target repos
Check repository existence: Use gh repo view owner/repo to verify the repository exists
Private repositories: Ensure your GitHub token has appropriate scopes (repo for private repos)

"Permission denied (publickey)"

Problem: Git authentication failure when pushing changes.

Solutions:

Check SSH keys: Ensure your SSH key is added to GitHub
```
ssh -T git@github.com
```
Use HTTPS with token: Configure git to use HTTPS with your personal access token
```
git config --global credential.helper store
```

Configuration Problems

"Configuration file not found"

Problem: go-broadcast can't find the sync.yaml file.

Solutions:

Specify config path explicitly:

go-broadcast sync --config /path/to/sync.yaml

Check current directory: Default looks for sync.yaml in current directory

Create example config:

go-broadcast validate --help  # Shows example configuration

"unsupported config version"

Problem: Configuration file uses wrong version format.

Solution: Update your configuration to use version 1:

version: 1  # Must be version 1
source:
  repo: "org/template-repo"
  branch: "master"
targets:
  - repo: "org/target-repo"
    files:
      - src: "file.txt"
        dest: "file.txt"

"Configuration validation failed"

Common validation errors and fixes:

Empty source repository:

source:
  repo: "org/template-repo"  # Must not be empty
  branch: "master"           # Must not be empty

No targets specified:

targets:  # Must have at least one target
  - repo: "org/target-repo"
    files:
      - src: "file.txt"
        dest: "file.txt"

Invalid repository format:

source:
  repo: "org/repo"  # Correct: owner/repository
  # repo: "repo"    # Wrong: missing owner

GitHub API Issues

"API rate limit exceeded"

Problem: Too many GitHub API requests.

Solutions:

Wait for rate limit reset: GitHub resets limits hourly
Use authenticated requests: Higher rate limits with authentication

Reduce concurrent operations:

go-broadcast sync --config sync.yaml  # Uses default concurrency

Check rate limit status:
```
gh api rate_limit
```

"Commit not found"

Problem: Specified commit SHA doesn't exist in repository.

Solutions:

Check commit exists: gh api repos/owner/repo/commits/SHA
Use branch name instead: Ensure branch exists and is accessible
Check repository sync: Source repository might be ahead of your local state

"Branch already exists"

Problem: Sync branch name conflicts with existing branch.

Solutions:

Delete existing branch (if safe):

git push origin --delete chore/sync-files-YYYYMMDD-HHMMSS-commit

Wait for branch name to change: Branch names include timestamps
Check for stuck sync processes: Multiple sync processes might conflict

Git Operations

"fatal: could not read Username for 'https://github.com'"

Problem: Git credentials not configured for HTTPS.

Solutions:

Configure git credentials:

git config --global credential.helper store
echo "https://username:token@github.com" > ~/.git-credentials

Use SSH instead: Configure repositories to use SSH URLs

Set GitHub token:

# Use GH_PAT_TOKEN as the preferred token (checked first)
export GH_PAT_TOKEN="your_token"
# Or use GITHUB_TOKEN as a fallback
export GITHUB_TOKEN="your_token"

"fatal: destination path already exists"

Problem: Target directory already exists during clone.

Solutions:

Clean up temporary directories: go-broadcast should clean up automatically
Check disk space: Insufficient space might prevent cleanup
Kill stuck processes: Check for other go-broadcast instances
```
ps aux | grep go-broadcast
```

"Your branch is ahead of 'origin/master'"

Problem: Local repository state is inconsistent.

Solutions:

This is normal: go-broadcast creates commits as part of sync process
Check sync completed: Ensure pull request was created successfully
Manual cleanup (if needed):
```
git reset --hard origin/master
```

File Synchronization

"No changes detected"

Problem: Files appear unchanged but should be different.

Solutions:

Check file transformations: Ensure transform rules are configured correctly
Verify source files: Check files exist in source repository

Review file mappings: Ensure src/dest paths are correct

files:
  - src: ".github/workflows/ci.yml"    # Must exist in source
    dest: ".github/workflows/ci.yml"   # Destination path

Use debug logging:
```
go-broadcast sync --log-level debug
```

"Binary file detected, skipping transformation"

Problem: Binary files not being transformed (this is correct behavior).

Solutions:

This is expected: Binary files are copied without transformation
Check file type: Use file command to verify file type
For text files misidentified as binary: Check for null bytes or unusual characters

"Transform failed"

Problem: File transformation encountered an error.

Solutions:

Check regex patterns: Ensure transformation patterns are valid

Review template variables: Ensure all required variables are defined

transform:
  variables:
    SERVICE_NAME: "my-service"  # Must be defined if used in templates

Test transformations locally: Create small test files to debug transforms

Directory Synchronization

"Directory sync completed but no files were synced"

Problem: Directory mapping completes successfully but results in 0 files synchronized.

Common Causes & Solutions:

All files excluded by patterns:

# Use debug logging to see exclusion details
go-broadcast sync --log-level debug --config sync.yaml 2>&1 | grep -i "excluded"

Check smart defaults (automatically applied):

# These are ALWAYS excluded for safety:
# *.out, *.test, *.exe, **/.DS_Store, **/tmp/*, **/.git

# Review your additional exclusions
directories:
  - src: "src"
    dest: "src"
    exclude:
      - "*.go"  # This would exclude all Go files!

Source directory doesn't exist:

# Verify source directory exists in template repo
gh api repos/company/template-repo/contents/source-directory

Empty source directory:

# Check if source directory is empty
gh api repos/company/template-repo/git/trees/main --recursive | \
jq '.tree[] | select(.path | startswith("source-directory/"))'

Fix: Review exclusion patterns and verify source directory content.

"Directory sync is extremely slow"

Problem: Directory synchronization takes much longer than expected performance targets.

Expected Performance:

<50 files: <3ms
50-150 files: 1-7ms
500+ files: 16-32ms
1000+ files: ~32ms

Diagnosis:

# Check actual processing time
go-broadcast sync --log-level debug --config sync.yaml 2>&1 | \
grep -E "(processing_time_ms|duration)"

Performance Issues & Solutions:

Large individual files:

# Exclude large files that don't need sync
directories:
  - src: "assets"
    dest: "assets"
    exclude:
      - "**/*.zip"        # Large archives
      - "**/*.tar.gz"     # Compressed files
      - "**/*.iso"        # Disk images
      - "**/*.dmg"        # macOS images

Too many excluded files (paradoxically slow):

# Instead of excluding many files, sync specific subdirectories
directories:
  # Instead of this (slow):
  # - src: "large-dir"
  #   exclude: ["sub1/**", "sub2/**", "sub3/**", ...]

  # Do this (fast):
  - src: "large-dir/wanted-subdir"
    dest: "large-dir/wanted-subdir"

Deep directory nesting:

# Avoid very deep recursion in exclusion patterns
exclude:
  - "**/deeply/nested/path/**"  # Can be slow
# Prefer specific patterns:
  - "deeply/nested/path/**"     # Faster

"Exclusion patterns not working as expected"

Problem: Files that should be excluded are still being synchronized.

Common Pattern Issues:

Case sensitivity:

# Patterns are case-sensitive
exclude:
  - "*.TMP"    # Won't match *.tmp
  - "*.tmp"    # Won't match *.TMP
  - "*.[Tt][Mm][Pp]"  # Matches both

Directory vs file patterns:

exclude:
  - "temp"      # Matches files named 'temp'
  - "temp/"     # Matches directories named 'temp'
  - "temp/**"   # Matches everything under temp directories
  - "**/temp/**" # Matches temp directories at any depth

Relative path confusion:

# Patterns match against paths relative to src directory
directories:
  - src: ".github"
    dest: ".github"
    exclude:
      - "workflows/local-*"      # Matches .github/workflows/local-*
      - "**/workflows/local-*"   # Also matches nested workflows

Debugging Exclusions:

# See detailed pattern matching
go-broadcast sync --log-level debug --config sync.yaml 2>&1 | \
grep -E "(excluded|pattern|matching)"

# Test specific patterns with dry-run
go-broadcast sync --dry-run --config sync.yaml

"Transform errors on directory files"

Problem: Some files in directory fail transformation but sync continues.

Expected Behavior: Transform errors on individual files don't fail entire directory sync (by design for robustness).

Common Issues:

Binary files processed as text:
```
# Check logs for binary detection
go-broadcast sync --log-level debug --config sync.yaml 2>&1 | \
grep -i "binary"
```
Solution: Binary files are automatically detected and skipped for transforms. This is correct behavior.

Invalid variable substitution:

directories:
  - src: "configs"
    dest: "configs"
    transform:
      variables:
        SERVICE_NAME: "my-service"
        # Missing variable will cause transform errors
        # Check logs for "undefined variable" messages

Malformed Go module paths (for repo_name transform):

# repo_name transform only works on text files with valid Go syntax
transform:
  repo_name: true  # May fail on config files, docs, etc.

Solutions:

# Exclude problematic files from transforms
directories:
  - src: "mixed-content"
    dest: "mixed-content"
    exclude:
      - "**/*.bin"      # Skip binary files
      - "**/*.jpg"      # Skip images
      - "**/*.pdf"      # Skip PDFs
    transform:
      repo_name: true   # Only applies to remaining text files

"API rate limiting during large directory sync"

Problem: GitHub API rate limits exceeded during directory operations.

Understanding the Issue:

go-broadcast uses GitHub tree API for 90%+ call reduction
Rate limits should rarely be hit with proper optimization
If you're hitting limits, there may be a configuration issue

Diagnosis:

# Check API usage in logs
go-broadcast sync --log-level debug --config sync.yaml 2>&1 | \
grep -E "(api|rate|tree|calls)"

# Check current rate limit status
gh api rate_limit

Solutions:

Verify tree API is being used:

# Should see "using tree API" in debug logs
# Should see very few individual file API calls

Split very large directories:

# Instead of syncing huge directory
directories:
  # Split into logical chunks
  - src: "large-dir/core"
    dest: "large-dir/core"
  - src: "large-dir/modules"
    dest: "large-dir/modules"

Optimize concurrent repositories:

# Sync specific repositories to reduce load
go-broadcast sync company/repo1 company/repo2 --config sync.yaml

"Memory usage too high during directory sync"

Problem: go-broadcast consuming excessive memory during large directory operations.

Expected Memory Usage:

~1.2MB per 1000 files processed
Linear scaling with file count
Memory released immediately after processing

Diagnosis:

# Monitor memory during sync
go-broadcast sync --config sync.yaml &
PID=$!
while kill -0 $PID 2>/dev/null; do
  ps -o pid,vsz,rss,comm -p $PID
  sleep 2
done

Solutions:

Segment large directories:

# Process in smaller chunks
directories:
  - src: "docs/section1"
    dest: "docs/section1"
  - src: "docs/section2"
    dest: "docs/section2"

Exclude large files:

exclude:
  - "**/*.zip"     # Large archives
  - "**/*.tar.*"   # Compressed files
  - "**/*.iso"     # Disk images
  - "data/**"      # Large data files

Reduce concurrent operations:

# Process fewer repositories simultaneously
go-broadcast sync specific/repo --config sync.yaml

"Directory structure not preserved correctly"

Problem: Nested directory structure is flattened or modified unexpectedly.

Understanding preserve_structure:

directories:
  - src: "docs/api/v1"
    dest: "api-docs"
    preserve_structure: true   # Results in: api-docs/nested/file.md
    preserve_structure: false  # Results in: api-docs/file.md (flattened)

Common Issues:

Unexpected flattening:

# Default is preserve_structure: true
# If you're seeing flattening, check for:
directories:
  - src: "nested/content"
    dest: "content"
    preserve_structure: false  # This causes flattening

Path conflicts (when flattening):

# Flattening can cause conflicts with same filenames
# src/dir1/config.yml and src/dir2/config.yml both → dest/config.yml
# Solution: Use preserve_structure: true or rename during sync

"Hidden files not syncing"

Problem: Files starting with . are not being synchronized.

Understanding include_hidden:

directories:
  - src: "configs"
    dest: "configs"
    include_hidden: true   # Syncs .env, .gitignore, etc. (default)
    include_hidden: false  # Skips hidden files

Solutions:

# Ensure include_hidden is not set to false
directories:
  - src: "project-root"
    dest: "project-root"
    include_hidden: true   # Explicit enable
    exclude:
      - "**/.DS_Store"     # Still excluded by smart defaults
      - "**/.git/**"       # Still excluded by smart defaults

Note: Smart defaults always exclude certain hidden files (.DS_Store, .git) for safety, even with include_hidden: true.

Performance Issues

"Sync is very slow"

Problem: Synchronization takes longer than expected.

Solutions:

Reduce concurrency (if system is overloaded):

# System handles concurrency automatically, but you can limit targets
go-broadcast sync org/specific-repo  # Sync only specific repositories

Check network connectivity: Slow GitHub API responses
Reduce file count: Large numbers of files take longer to process
Use dry-run first: Test configuration before full sync
```
go-broadcast sync --dry-run
```

"Out of memory errors"

Problem: go-broadcast consuming too much memory.

Solutions:

Reduce concurrent operations: Sync fewer repositories at once
Check file sizes: Large files consume more memory during transformation
Split large configurations: Sync subsets of repositories separately
Monitor system resources: Ensure adequate RAM available

Debugging

Enable Debug Logging

go-broadcast sync --log-level debug

This shows detailed information about:

Configuration loading and validation
GitHub API requests and responses
Git operations and their results
File transformations and their effects
State discovery process

Advanced Debug Options

# Component-specific debugging
go-broadcast sync --debug-git --debug-api --debug-transform --debug-config --debug-state

# JSON output for structured analysis
go-broadcast sync --json -vv 2>&1 | jq 'select(.level=="error")'

# Performance monitoring
go-broadcast sync --json 2>&1 | jq 'select(.duration_ms > 5000)'

Check Sync State

go-broadcast status --config sync.yaml

This shows:

Current state of all target repositories
Last sync times and commit SHAs
Open pull requests
Sync status for each repository

Validate Configuration

go-broadcast validate --config sync.yaml

This checks:

Configuration file syntax
Required fields
Repository name formats
File path validity
Transform configuration

Dry Run Mode

go-broadcast sync --dry-run --config sync.yaml

This shows what would happen without making changes:

Which files would be synchronized
What transformations would be applied
Which pull requests would be created

Manual State Check

Check GitHub state manually:

# List branches in target repository
gh api repos/owner/target-repo/branches

# Check for existing sync PRs
gh pr list --repo owner/target-repo --label automated-sync

# View specific branch
gh api repos/owner/target-repo/branches/chore/sync-files-YYYYMMDD-HHMMSS-commit

Getting Help

If you're still experiencing issues:

Check the logs: Always run with --log-level debug first
Create minimal reproduction: Simplify your configuration to isolate the problem
Check GitHub status: Visit GitHub Status for API issues
Review configuration: Double-check repository names, file paths, and permissions
Test components individually:
- Test GitHub authentication: gh auth status
- Test repository access: gh repo view owner/repo
- Test configuration: go-broadcast validate

Common Patterns

Testing New Configuration

# 1. Validate configuration
go-broadcast validate --config new-sync.yaml

# 2. Test with dry-run
go-broadcast sync --dry-run --config new-sync.yaml

# 3. Test with single repository
go-broadcast sync org/test-repo --config new-sync.yaml

# 4. Full sync
go-broadcast sync --config new-sync.yaml

Recovering from Failed Sync

# 1. Check current status
go-broadcast status --config sync.yaml

# 2. Check for stuck branches/PRs in GitHub web interface

# 3. Clean up if needed (be careful!)
gh pr close 123 --repo org/target-repo  # Close stuck PR
git push origin --delete branch-name     # Delete stuck branch

# 4. Retry sync
go-broadcast sync --config sync.yaml

Monitoring Ongoing Sync

# Check status periodically
watch -n 30 "go-broadcast status --config sync.yaml"

# Monitor logs in another terminal
go-broadcast sync --log-level info --config sync.yaml 2>&1 | tee sync.log

## Developer Workflow Integration

For comprehensive go-broadcast development workflows, see [CLAUDE.md](../.github/CLAUDE.md#️-troubleshooting-quick-reference) which includes:
- **Troubleshooting development issues** with build failures, test failures, and linting errors
- **go-broadcast debugging procedures** with verbose logging and component-specific debugging
- **Environment troubleshooting steps** for GitHub authentication and Go environment issues
- **Development workflow integration** for effective problem-solving

## Operational Procedures

### Health Check Procedures

#### Basic Health Check
```bash
#!/bin/bash
# go-broadcast-health-check.sh

echo "=== go-broadcast Health Check ==="

# Check binary exists and is executable
if ! command -v go-broadcast &> /dev/null; then
    echo "❌ go-broadcast not found in PATH"
    exit 1
fi

# Check version
echo "✅ Version: $(go-broadcast --version 2>/dev/null || echo 'Unknown')"

# Check dependencies
echo "✅ Git: $(git --version)"
echo "✅ GitHub CLI: $(gh --version | head -1)"

# Check authentication
if gh auth status &> /dev/null; then
    echo "✅ GitHub authentication: OK"
else
    echo "❌ GitHub authentication: Failed"
fi

# Test basic functionality
if go-broadcast diagnose > /dev/null 2>&1; then
    echo "✅ Basic functionality: OK"
else
    echo "❌ Basic functionality: Failed"
fi

echo "=== Health Check Complete ==="

Performance Baseline

#!/bin/bash
# performance-baseline.sh

echo "=== Performance Baseline Test ==="

# Small repository test
time go-broadcast sync --config test-config.yaml --dry-run

# Memory usage test
/usr/bin/time -l go-broadcast sync --config test-config.yaml --dry-run 2>&1 | \
  grep "maximum resident set size"

# API rate limit check
go-broadcast sync --debug-api --json --config test-config.yaml --dry-run 2>&1 | \
  jq 'select(.rate_limit_remaining) | .rate_limit_remaining' | tail -1

Log Analysis Workflows

Error Pattern Analysis

# Extract all errors with context
go-broadcast sync --json 2>&1 | \
  jq 'select(.level=="error") | {time: ."@timestamp", component, operation, message, error}' | \
  jq -s 'sort_by(.time)'

# Find error patterns
go-broadcast sync --json 2>&1 | \
  jq -r 'select(.level=="error") | .error' | \
  sort | uniq -c | sort -rn

Performance Analysis

# Operation timing analysis
go-broadcast sync --json 2>&1 | \
  jq 'select(.duration_ms) | {operation, duration_ms, repo}' | \
  jq -s 'sort_by(.duration_ms) | reverse'

# Component performance breakdown
go-broadcast sync --json 2>&1 | \
  jq 'select(.duration_ms) | {component, operation, duration_ms}' | \
  jq -s 'group_by(.component) | map({component: .[0].component, total_ms: map(.duration_ms) | add, operations: length})'

Security Event Monitoring

# Authentication events
go-broadcast sync --json 2>&1 | \
  jq 'select(.component=="audit" and .event=="authentication") | {time: ."@timestamp", success, method}'

# Repository access tracking
go-broadcast sync --json 2>&1 | \
  jq 'select(.event=="repo_access") | {time: ."@timestamp", repo, action, user}'

Escalation Procedures

Level 1: Self-Service

Review this troubleshooting guide
Check logs with basic verbose flags (-v)
Verify configuration and authentication
Test with --dry-run flag

Level 2: Advanced Debugging

Collect comprehensive logs (-vvv with all debug flags)
Run go-broadcast diagnose
Performance analysis with JSON output
Search existing GitHub issues

Level 3: Support Engagement

Collect the following information:

# Support information package
SUPPORT_DIR="go-broadcast-support-$(date +%Y%m%d-%H%M%S)"
mkdir "$SUPPORT_DIR"

# System diagnostics
go-broadcast diagnose > "$SUPPORT_DIR/diagnostics.json"

# Configuration (with sensitive data redacted)
cp sync.yaml "$SUPPORT_DIR/config.yaml"
sed -i.bak 's/ghp_[a-zA-Z0-9_]*/ghp_***REDACTED***/g' "$SUPPORT_DIR/config.yaml"

# Full debug logs
go-broadcast sync --json -vvv --debug-git --debug-api --debug-transform --debug-config --debug-state 2> "$SUPPORT_DIR/debug.log"

# Error extraction
jq 'select(.level=="error")' < "$SUPPORT_DIR/debug.log" > "$SUPPORT_DIR/errors.json"

# Create support archive
tar -czf "${SUPPORT_DIR}.tar.gz" "$SUPPORT_DIR"
echo "Support package created: ${SUPPORT_DIR}.tar.gz"

Uh oh!

FilesExpand file tree

troubleshooting.md

Latest commit

History

troubleshooting.md

File metadata and controls

Troubleshooting Guide

Quick Reference

Table of Contents

Systematic Troubleshooting

1. Information Gathering

2. Initial Assessment

3. Log Collection

Authentication Issues

"gh: To get started with GitHub CLI, please run: gh auth login"

"gh: Not Found (HTTP 404)"

"Permission denied (publickey)"

Configuration Problems

"Configuration file not found"

"unsupported config version"

"Configuration validation failed"

GitHub API Issues

"API rate limit exceeded"

"Commit not found"

"Branch already exists"

Git Operations

"fatal: could not read Username for 'https://github.com'"

"fatal: destination path already exists"

"Your branch is ahead of 'origin/master'"

File Synchronization

"No changes detected"

"Binary file detected, skipping transformation"

"Transform failed"

Directory Synchronization

"Directory sync completed but no files were synced"

"Directory sync is extremely slow"

"Exclusion patterns not working as expected"

"Transform errors on directory files"

"API rate limiting during large directory sync"

"Memory usage too high during directory sync"

"Directory structure not preserved correctly"

"Hidden files not syncing"

Performance Issues

"Sync is very slow"

"Out of memory errors"

Debugging

Enable Debug Logging

Advanced Debug Options

Check Sync State

Validate Configuration

Dry Run Mode

Manual State Check

Getting Help

Common Patterns

Testing New Configuration

Recovering from Failed Sync

Monitoring Ongoing Sync

Performance Baseline

Log Analysis Workflows

Error Pattern Analysis

Performance Analysis

Security Event Monitoring

Escalation Procedures

Level 1: Self-Service

Level 2: Advanced Debugging

Level 3: Support Engagement

Related Documentation