feat: clear destination for iceberg/s3 #238

vishalm0509 · 2025-10-23T05:21:32Z

Description

This PR introduces the Clear Destination feature that allows users to manually or automatically clear data from a destination linked to a job when configuration or stream changes are detected.

Added two new API endpoints:

POST /api/v1/project/:projectid/jobs/:id/clear-destination – Triggers a clear destination workflow for the specified job.
GET /api/v1/project/:projectid/jobs/:id/clear-destination – Returns the current status of the clear destination workflow (running or not running).

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Run clear destination manually
Update job and check if configured streams are getting dropped from the destination

…e-ui into feat/clear_destination

…tion

vikaxsh · 2025-10-24T12:05:54Z

server/internal/dto/response.go

+type ClearDestinationStatusResposne struct {
+	Running bool `json:"running"`
+}


shouldn't this status as key and value running, failed, completed etc

Removed status endpoint now. Using the response from JobTask API

vikaxsh · 2025-10-24T12:17:30Z

server/internal/dto/response.go

+}
+
+type StreamDifferenceResponse struct {
+	DifferenceStreams map[string]interface{} `json:"difference_streams"`


it should be string , similar to streams

vikaxsh · 2025-10-24T12:31:34Z

server/internal/dto/response.go

+	LastRunType   string               `json:"last_run_type,omitempty"` // "sync" | "clear-destination"
 	CreatedAt     string               `json:"created_at"`
 	UpdatedAt     string               `json:"updated_at"`
 	Activate      bool                 `json:"activate"`
 	CreatedBy     string               `json:"created_by,omitempty"`
 	UpdatedBy     string               `json:"updated_by,omitempty"`
 }

 type JobTask struct {
 	Runtime   string `json:"runtime"`
 	StartTime string `json:"start_time"`
 	Status    string `json:"status"`
 	FilePath  string `json:"file_path"`
+	JobType   string `json:"job_type"` // "sync" | "clear-destination"


why different fields for job sync or clear destination

We need to show in the JobTasks API if the last run is sync or clear

vikaxsh · 2025-10-24T12:33:16Z

server/internal/handlers/job.go

 	projectID := c.Ctx.Input.Param(":projectid")
 	id := GetIDFromPath(&c.Controller)
-	resp, err := c.jobService.CancelJobRun(c.Ctx.Request.Context(), projectID, id)
+	resp, err := c.jobService.CancelJobRun(projectID, id)


Context is not being used by that service, that's why I have removed

vikaxsh · 2025-10-24T12:34:56Z

server/internal/handlers/job.go

+// @router /project/:projectid/jobs/:id/clear-destination [get]
+func (c *JobHandler) GetClearDestinationStatus() {


why different apis, job task api wouldn't be enough to tell status of last running workflow for that particular job?

Yes, removed

vikaxsh · 2025-10-24T12:45:11Z

server/internal/services/job.go

+	}
+
+	// start clear destination
+	var diffCatalog map[string]interface{}


user can update the running frequency of the job as well do we need to clear in that case as well, and can add new streams as well

No, in case of frequency, we will get empty diff_streams so clear will not run

vikaxsh · 2025-10-24T12:48:30Z

server/internal/services/job.go

+	running, err := isClearRunning(ctx, s.tempClient, projectID, jobID)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to get clear destination status: %s", err)
+	}
+	if running {
+		return nil, constants.ErrInProgress
+	}
+


instead of checking here can we create clear activity and sync activity and execuete based on isClear flag in sync api

Discussed already. We need a new endpoint

…rogress

* feat: show error logs in test connection in source and destination * fix: fix test connection fail modal * fix: resolve comments * chore: add comments (#224) * chore: add comments * chore: 🚨 lint fix * feat: add error logs in check connection for source and destination (#195) * feat: add error logs i check conenction for source and destination * fix: parse json message as string * chore: update response key * chore: refactor * chore: remove logEntry struct * fix: log parsing * fix: log parsing * fix: logs parsing logic * docs: updated api contract * test: ✅ base setup for automation testing using play… (#203) * test: ✅ base setup for automation testing using playwright , add test cases fro major flows * fix: integration workflow changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * fix: ci changes * test: add postgres flow * fix: lint changes * test: add iceberg specific test * fix: tests * fix: stream name * fix: test fix * fix: test fix * fix: test fix * fix: test fix * fix: changes * fix: ci changes * fix: ci changes * fix: ci changes * test: fail test on test connection failure * fix: ci changes * fix: changes * fix: changes * fix: changes * fix: changes * fix: changes * fix: changes * fix: test fix * fix: changes * fix: changes * fix: test-data fix * fix: test-data fix * fix: changes * fix: changes * fix: changes * fix: host and port changes * fix: changes * fix: change * test: login test updated * test: refactor create source and destination page * test: refactor tests * fix: test fix * fix: changes * fix: fix changes * fix: changes * fix: changes * fix: review changes * refactor: refactor tests to use authenticated state of playwright * fix: changes * docs: update tests readme * test: intercept and log destination spec response * docs: update readme * fix: update timeouts for playwright tests * test: update readme * fix: add retries in playwright config * test: fix timeout duration * fix: update readme * test: add connector data test-id * fix: resolve comments * refactor(test): make and use enums instead of strings * fix: fix test data builder * fix: test data job name * fix: add todo --------- Co-authored-by: Duke Dhal <[email protected]> Co-authored-by: deepanshupal09-datazip <[email protected]> * feat: show refetch success message in job history page (#227) * Added success message on Job History refetch * Update ui/src/modules/jobs/pages/JobHistory.tsx Co-authored-by: deepanshupal09-datazip <[email protected]> --------- Co-authored-by: vikash choudhary <[email protected]> Co-authored-by: deepanshupal09-datazip <[email protected]> * fix: replace deprecated Phosphor icons with latest *Icon alternatives (#216) * fix: replace deprecated Phosphor icons with latest *Icon alternatives * fix: update Phosphor icons to latest alternatives in DestinationEdit and SourceEdit components and fix Lint issue * refactor: remove EditSourceModal * fix: update icon import --------- Co-authored-by: vikash choudhary <[email protected]> * feat: add clear destination * fix: merge conflicts * fix: build error fix * feat: append only mode (#232) * feat: add append only mode in ui * fix: data filter bug * fix: fix color of append only mode switch * fix: add custom option in all streams ingestion mode change * fix: use clsx for conditional styling * fix: fix icons * chore: add utils --------- Co-authored-by: Taraka Swathi <[email protected]> Co-authored-by: Ankit Sharma <[email protected]> Co-authored-by: vikash choudhary <[email protected]> Co-authored-by: Duke Dhal <[email protected]> Co-authored-by: SARTHAK RANA <[email protected]> Co-authored-by: vishalm0509 <[email protected]>

vishalm0509 · 2025-11-05T06:51:38Z

Created a New PR for this feature: #243

vishalm0509 added 12 commits October 22, 2025 18:27

feat: add clear-dest bff

36f4691

Merge branch 'refactor/worker' of https://github.com/datazip-inc/olak…

22c6626

…e-ui into feat/clear_destination

fix: add hash to clear-destination workflowID

42961a7

chore: add api-contract and minor refactoring

062fb45

chore: remove hardcoded version used for testing

c2676bd

fix: add proper error handling

33962d6

chore: update api contract for job task and logs

b30d579

fix: add response type for clear-destination status

7ecd1bb

fix: add cancel sync in job update & remove cancel from clear destina…

6c620fa

…tion

fix: add stream difference endpoint

242ef5d

fix: change streamsDifference to differenceStreams

e20f2e6

fix: rename workflow names and remove req ctx from cancelJob

85775bd

vikaxsh reviewed Oct 24, 2025

View reviewed changes

vishalm0509 and others added 5 commits October 27, 2025 16:28

chore: remove clear-destination status api endpoint

b380856

fix: resolve comments and add proper error for clear-destination in p…

de00a75

…rogress

chore: add proper error for SyncJob

40ddd64

fix: block sync/clear if job is paused

892282e

		// @router /project/:projectid/jobs/:id/clear-destination [get]
		func (c *JobHandler) GetClearDestinationStatus() {

feat: clear destination for iceberg/s3 #238

Are you sure you want to change the base?

feat: clear destination for iceberg/s3 #238

Uh oh!

Conversation

vishalm0509 commented Oct 23, 2025

Description

Type of change

How Has This Been Tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vishalm0509 commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants