[SPARK-60018] Add ApplicationStateSummary.Suspended and ApplicationStatus#resume

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add ApplicationStateSummary.Suspended and ApplicationStatus#resume.

• Add Suspended to ApplicationStateSummary, and make isStopping() exclude it. Otherwise, AppCleanUpStep would end a suspended application. The result for the existing states is unchanged.
• Add ApplicationStatus#resume, which starts the next attempt of a Suspended application from Submitted. Unlike a restart, it does not increase the restart counters, and it resets only schedulingFailureRestartCounter.
• Add Suspended to the SparkApplication CRD of spark.apache.org/v1, and update docs/migration_guide.md.

Why are the changes needed?

To support spec.suspend for a running SparkApplication in a follow-up PR. Currently, it takes effect only before the driver is requested.

Does this PR introduce any user-facing change?

Yes, in the API only. Unlike 1.0.0 (2026-07-23), the SparkApplication CRD accepts the Suspended state, an exhaustive switch over ApplicationStateSummary needs a case for Suspended, and ApplicationStatus has the new public method resume. There is no behavior change, since nothing uses them until the follow-up PR.

How was this patch tested?

Pass the CIs with the newly added test cases.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

[dongjoon-hyun]

Could you review this PR when you have some time, @viirya ?

[viirya]

The API changes look reasonable to me. Appending Suspended preserves the existing enum ordinals, and the revised isStopping() preserves the classification of all existing states. Extracting startNextAttempt() also keeps the existing restart history handling intact.

The distinction between resuming and restarting is clear: resuming advances the attempt ID without consuming the restart budget, while resetting the consecutive scheduling-failure counter. The tests cover both history modes and useful interactions with subsequent failures.

No blocking concerns found in this API-only change. For the separate PR that wires up running-application suspension, please cover reconciliation of persisted Suspended states (the current reconciler routes them to the unknown-state handler) and decide whether explicit resumption should bypass restart backoff (AppInitStep currently applies it whenever previousAttemptSummary is non-null, including after resume()). These are follow-up integration notes, not requests to expand this PR.

[viirya]

Non-blocking documentation suggestion: could we explicitly state that this method only constructs the next status and that callers must complete the required cleanup of the suspended attempt before starting the next one? This would make the caller's responsibility clear, similar to the resource-release precondition documented on terminateOrRestart().

[dongjoon-hyun]

Thank you, @viirya. I added a paragraph saying that it only creates the status of the new attempt, and that the resources of the suspended attempt are expected to be released already, like terminateOrRestart.

[dongjoon-hyun]

Thank you for the review and approval, @viirya. Both notes are covered by the follow-up PR.

• SparkAppReconciler will route Suspended to a new AppSuspendStep instead of AppUnknownStateStep.
• AppInitStep will apply the restart backoff only in ScheduledToRestart, so an attempt which resume starts from Submitted will not wait for it.

[viirya]

Thank you @dongjoon-hyun !

[dongjoon-hyun]

Thank you always, @viirya !

[dongjoon-hyun]

Merged to main