Missing activeDeadlineSeconds in pr-exp.yaml

[Ekam219]

Description:

@DonggeLiu
In the pr-exp.yaml file, the first Kubernetes job includes the activeDeadlineSeconds: 604800 (7 days) field to prevent indefinite execution. However, the second job is missing this field, which could result in:

• The job running indefinitely.
• Excessive resource consumption.
• Unpredictable behavior.

Suggested Fix:

Add activeDeadlineSeconds: 604800 to the second job in pr-exp.yaml to match the first job's configuration.

[Kartikayy007]

Hello @Ekam219 I think there are some flaws adding it like if there is a job which already have more than 7 days (604800) of job it'll force fully terminate the job on 7th day

[Ekam219]

Hello @Kartikayy007,

The 7-day activeDeadlineSeconds in large-pr-exp.yaml was likely chosen as a safe upper bound for large experiments. For smaller jobs in pr-exp.yaml, we could use a shorter time limit based on their expected duration. Aligning with this approach or creating a tiered system would ensure efficient resource management without prematurely terminating jobs with longer durations.

[Ekam219]

@DonggeLiu, could you confirm if the 7-day value was chosen based on historical job durations or another reason? Would you agree with tiering timeouts for smaller jobs? we can also make it configurable through environment variable
The default timeout is set to 7 days (604800 seconds), but it can be overridden by setting the GKE_EXP_TIMEOUT environment variable.
activeDeadlineSeconds: ${GKE_EXP_TIMEOUT:-604800}