chore(data-warehouse): Removed the compaction workflow and trigger it from the import workfl…

[Gilbert09]

Changes

• Move the compaction logic back into the main import pipeline workflow instead of triggering a second workflow
• Updated tests
• TODO post release: decommission infra bits for the compaction pods

[greptile-apps]

_{8 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: compaction failure now causes entire import to fail

Previously, trigger_compaction_job wrapped compaction in try/except with capture_exception, allowing the import to succeed even if compaction failed. Now if compact_table() raises an exception, the entire import workflow fails despite data being successfully imported.

Wrap in try/except:

Suggested change

	self._logger.debug("Triggering compaction and vacuuming on delta table")
	self._delta_table_helper.compact_table()
	self._logger.debug("Triggering compaction and vacuuming on delta table")
	try:
	self._delta_table_helper.compact_table()
	except Exception as e:
	from posthog.exceptions_capture import capture_exception
	capture_exception(e)
	self._logger.exception(f"Compaction failed: {e}", exc_info=e)

Prompt To Fix With AI

This is a comment left during a code review.
Path: posthog/temporal/data_imports/pipelines/pipeline/pipeline.py
Line: 341:342

Comment:
**logic:** compaction failure now causes entire import to fail

Previously, `trigger_compaction_job` wrapped compaction in try/except with `capture_exception`, allowing the import to succeed even if compaction failed. Now if `compact_table()` raises an exception, the entire import workflow fails despite data being successfully imported.

Wrap in try/except:

```suggestion
        self._logger.debug("Triggering compaction and vacuuming on delta table")
        try:
            self._delta_table_helper.compact_table()
        except Exception as e:
            from posthog.exceptions_capture import capture_exception
            capture_exception(e)
            self._logger.exception(f"Compaction failed: {e}", exc_info=e)
```

How can I resolve this? If you propose a fix, please make it concise.