Problem
When using --remote for Spark test execution, each run creates a directory under dbt-remote-runs/artifacts/{run_id}/ on the lakehouse. These are never cleaned up automatically, so storage usage grows unbounded over time.
Proposed solution
Add a cleanup mechanism — either:
- A
--remote-cleanup flag that prunes artifacts older than N days
- Automatic cleanup at the start of each run (delete artifacts older than 7 days)
- A separate maintenance script
Context
This was identified during code review of PR #208. The current design correctly isolates artifacts per run for concurrency safety, but has no garbage collection.
Problem
When using
--remotefor Spark test execution, each run creates a directory underdbt-remote-runs/artifacts/{run_id}/on the lakehouse. These are never cleaned up automatically, so storage usage grows unbounded over time.Proposed solution
Add a cleanup mechanism — either:
--remote-cleanupflag that prunes artifacts older than N daysContext
This was identified during code review of PR #208. The current design correctly isolates artifacts per run for concurrency safety, but has no garbage collection.