-
Notifications
You must be signed in to change notification settings - Fork 59
Description
Problem to be solved
When a user develops models in their Quary project, builds the project (quary build) to materialize those models in the target schema/database, but later deletes some of those model definitions from the project, the corresponding objects remain in the target schema even though they are no longer part of the current project. Over time, this can lead to the accumulation of stale, unused objects in the target schema, increasing cloud storage costs and cluttering the data warehouse.
Quary currently lacks an easy way for users to clean up these orphaned objects and keep their target schema in sync with the current state of their project.
Suggested solution
Implement a new Quary operation, tentatively called quary clean, that will:
- Introspect the current Quary project to determine what models/objects it defines
- Compare this to the objects actually present in the target schema
- Identify any objects in the target that no longer map to any models in the current project
- Provide the user clear output listing out these stale objects
- Optionally, allow the user to specify a flag that will generate the DDL statements to drop those stale objects
- Optionally, allow the user to specify an additional flag to actually execute those drop statements
This will allow users to manually trigger clean-up of their target schemas and remove any models that are no longer part of their current project.
As a potential incremental step towards this goal, we could first implement a simpler quary diff operation that just performs steps 1-4 above and outputs the diff between project and target, without handling the dropping of stale objects. This would still provide visibility to users about stale objects.