Can't stop running sync job in Kubernetes using UI

[lkzcgfvf]

Environment

• Deploy method: Kubernetes using latest Helm chart
• Olake version: for instance 0.7.2

Description

I'm running OLake in Kubernetes with AWS 8.0.mysql_aurora.3.10.3. I created job, successfully started it in initial data load mode and it was running for 15 mins. After that I needed to stop job because it was running for too long.
In UI I stopped job ( pressed "Cancel run" ) but after 5 min it was still running. I tried to delete job, delete source but it didnt help. Job was still running and constantly querying MySQL database.

Steps to reproduce

1. Login to UI
2. Run Job in initial data load mode
3. Press "Cancel run"
4. Wait 2-3 mins
5. Check MySQL status ( queries are still running)

Expected behavior

If I press "Cancel Run" - it should

1. Kill queries that are already running in database ( MySQL/etc..)
2. Stop loading process and exit

Actual behavior

Sync job is still running and queries in database are also running

[vishalm0509]

@lkzcgfvf we have tried reproducing this issue internally, and not able to do it. Could you please share the worker logs for the affected job.

[hash-data]

@lkzcgfvf are you still facing this?

[lkzcgfvf]

yes, its still a problem.

• What exactly UI does to stop job that is running?
• If job is running with 10+ threads - how does it stop running query in every thread?

[vishalm0509]

@lkzcgfvf

What exactly UI does to stop job that is running?

In OLake, A Job is a Temporal workflow (a process). When you click Cancel Run, the UI sends a cancellation request to the workflow. Once the workflow context is cancelled, the workflow begins shutting down gracefully and the job is cancelled.

If job is running with 10+ threads - how does it stop running query in every thread?

Since all the threads belong to the same workflow, when the workflow context is cancelled all the connections to the database instance from that workflow are closed.

---

Also, could you please let us know which one of this are you facing:

1. The database query remains in the EXECUTING state even after the job is cancelled.
2. The job state in the OLake UI does not change to CANCELLED after clicking Cancel Run.
3. Both of the above.

and share the logs of the Job of the affected Jobs. You can find an option to download the logs in the Jobs Logs and History page -> Affected Job's log

[lkzcgfvf]

option #3 ( Both of the above )
trying to rerpoduce to get logs

[vishalm0509]

@lkzcgfvf Regarding the issues:

Issue #1: This is a limitation of the MySQL driver. When a query is already executing, cancellation may not immediately terminate the query. The query will stop once the current scan operation completes. Since the table might not have a primary key, the scan can take longer to finish.

Issue #2: We'd like to investigate this further. Could you please share the logs for the affected job?