IMPROVEMENTS.md

Watcher Service Improvements

Issues Fixed

1. Token Refresh Not Occurring ✓

Problem: The Dropbox access token expires after ~4 hours, but the watcher was created with a single token at startup and never refreshed.

Solution:

• Added background token_refresh_loop task that runs every hour
• Automatically refreshes the access token using the saved refresh token
• Updates the shared DropboxClient state with the new token
• All in-flight and new uploads automatically use the refreshed token

Implementation: See watcher.rs:273-328

2. Upload Queue with Background Processing ✓

Problem: Uploads were blocking the main event loop, preventing new file events from being processed during uploads.

Solution:

• Implemented proper upload queue using mpsc::channel
• Background upload_processor task handles uploads asynchronously
• Main event loop continues processing file events while uploads happen
• Supports configurable parallelism (respects --parallel setting)

Implementation: See watcher.rs:330-387

3. File Stability Checking ✓

Problem: Files being written (large copies in progress) could start uploading before complete, resulting in partial uploads.

Solution:

• Track file stability by monitoring size and modification time
• Only queue files for upload after they've been stable for the debounce period
• Prevents uploading incomplete files

Implementation: See watcher.rs:389-440 and watcher.rs:442-524

4. Deep Folder Support ✓

Problem: Concern that nested folders weren't being monitored.

Solution:

• Already using RecursiveMode::Recursive for watching (line 151)
• Path mapping correctly handles nested paths via strip_prefix
• Verified to work with deep directory structures

Implementation: Works out of the box, see watcher.rs:151 and watcher.rs:605-622

5. Smart File Deduplication with Size Comparison ✓

Problem: Files already in Dropbox were being re-uploaded unnecessarily, but modified files needed to be updated.

Solution:

• Added DropboxClient::get_file_size() method to check if file exists AND get its size
• Upload if file doesn't exist in Dropbox (new file)
• Upload if file exists BUT size differs (modified file - replaces existing)
• Skip if file exists AND size matches (no changes)
• Logs decision for each file (new/modified/skipped)

Logic:

File exists in Dropbox?
├─ No → Upload (new file)
└─ Yes → Compare sizes
    ├─ Different → Upload with replace (modified file)
    └─ Same → Skip (already synced)

Implementation: See dropbox/client.rs:62-81 and size comparison in watcher.rs

6. Startup File Scanning ✓

Problem: When watcher starts, existing files in monitored folders weren't being checked/uploaded.

Solution:

• On startup, recursively scan all watched folders
• For each file, check if it exists in Dropbox at the target path
• Queue files that don't exist for upload
• Logs summary of files found and queued

Implementation: See watcher.rs:526-618

7. Upload Progress Tracking ✓

Problem: No visibility into what's uploading or in the queue.

Solution:

• Track in-progress uploads in shared state (state.uploading set)
• Prevent duplicate uploads of the same file
• Log when files are queued, uploading, and completed
• Periodic queue status logging (every 60s if queue is active)

Implementation: See watcher.rs:79-85 and logging throughout

8. Comprehensive Logging ✓

Problem: Insufficient logging made it hard to understand what the watcher was doing.

Solution - Added log entries for:

• Token refresh events
• File detection and stability tracking
• Queue status (pending + uploading counts)
• Individual upload start/complete/failure
• Startup scan results
• Files skipped because they already exist

Examples:

log::info!("🔄 Refreshing Dropbox access token...");
log::info!("✓ Token refreshed successfully (expires in {}s)", expires_in);
log::info!("📤 Queued upload: {} -> {} ({} bytes)", ...);
log::info!("✓ Upload complete: {}", path);
log::info!("Queue status: {} pending, {} uploading", ...);
log::info!("File already exists in Dropbox, skipping: {}", ...);

Architecture Changes

Before

File Event → Process Immediately → Block until Upload Complete

• Single-threaded processing
• Uploads blocked event loop
• No token refresh
• No deduplication

After

File Event → Stability Tracker → Queue → Background Uploader
                                           ↓
                                    Parallel Uploads
                                           ↓
                                    Auto Token Refresh

• Multi-threaded with async tasks
• Non-blocking event loop
• Automatic token refresh every hour
• Deduplication via Dropbox checks

New Background Tasks

1. Upload Processor (watcher.rs:330)

   • Processes upload queue
   • Respects parallelism limit
   • Tracks in-progress uploads
   • Logs upload progress

2. Token Refresh Loop (watcher.rs:273)

   • Runs every hour
   • Refreshes access token
   • Updates shared client
   • Logs refresh status

Configuration Requirements

The watcher constructors now require app_key and app_secret:

FolderWatcher::new(config, client, app_key, app_secret)
FolderWatcher::new_with_config_file(config, config_file, client, app_key, app_secret)

This enables automatic token refresh without user intervention.

Testing Recommendations

1. Token Refresh: Run watcher for >4 hours and verify uploads continue working
2. File Stability: Copy large file into watched folder and verify it's not uploaded until complete
3. Size-Based Deduplication:
   • Add file that already exists with same size in Dropbox → verify it's skipped
   • Modify file (change size) → verify it's uploaded and replaces existing
   • Add new file → verify it's uploaded
4. Startup Scan: Start watcher with files already in watched folders, verify missing ones are queued
5. Deep Folders: Create nested folders and verify all files are detected
6. Queue Status: Monitor logs for queue status updates
7. Concurrent Uploads: Add multiple files and verify they upload in parallel

9. Persistent Upload Queue ✓

Problem: If service restarts or crashes, upload queue is lost and files must be re-detected.

Solution:

• Queue saved to disk (~/.dbxup_queue.json)
• Auto-saves every 30 seconds
• Loads on startup and validates entries
• Merges with startup file scan (deduplication)
• Tracks upload session IDs for resumable chunked uploads
• Retry logic with failure tracking
• Invalid/stale entries automatically pruned

Benefits:

• No lost uploads on restart/crash
• Large files can resume from where they left off
• Failed uploads automatically retried
• Queue can be inspected anytime

Implementation: See src/persistent_queue.rs and PERSISTENT_QUEUE.md

Files Modified

• src/watcher.rs - Complete refactor with persistent queue integration
• src/dropbox/client.rs - Added get_file_size() method
• src/main.rs - Updated watcher constructors to pass app_key/app_secret
• src/persistent_queue.rs - New module for persistent queue (300+ lines)