[wip] pipeline optimization

[ayushRanjanCnflt]

Description

What behavior do you want to change, why, how does your patch achieve the changes?

Testing done

Describe the testing strategy. Unit and integration tests are expected for any behavior changes.

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")
• Do these changes have compatibility implications for rollback? If so, ensure that the ksql command version is bumped.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

Pull Request Overview

This PR introduces CI pipeline optimizations by modifying Semaphore configuration to improve build queue management and resource utilization.

• Excludes the pipeline-opt branch from parallel processing restrictions
• Adds auto-cancellation of queued builds specifically for the master branch

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Hardcoding the branch name 'pipeline-opt' in the CI configuration creates a maintenance burden. Consider using a more generic pattern or environment variable to avoid needing to update the CI config for each optimization branch.

Suggested change

	- when: "branch != 'master' and branch !~ '[0-9]+\\.[0-9]+\\.x' and branch != 'pipeline-opt'"
	- when: "branch != 'master' and branch !~ '[0-9]+\\.[0-9]+\\.x' and branch !~ 'pipeline-opt.*'"

[Copilot]

Auto-cancelling queued builds only on the master branch may not be the intended behavior. Typically, auto-cancellation is more useful for feature branches to avoid running outdated builds when new commits are pushed. Consider if this should apply to all branches or if the logic should be inverted.

Suggested change

	when: "branch == 'master'"
	when: "branch != 'master' and branch !~ '[0-9]+\\.[0-9]+\\.x' and branch != 'pipeline-opt'"