PS-111-mirrors use withTimeout that will cancel after expiry

[SorchaAbel]

Description

There is an error (intermittently) mirror fails when a node trying to acquire the lock and checkout

agent/internal/job/checkout.go

Lines 491 to 493 in dd959ad

	if err != nil {
	return fmt.Errorf("getting/updating git mirror: %w", err)
	}

Ideally the customer wants a new exit code for the git mirror failure so they can retry on that
or
any time element so the lock would get auto-released after a certain period

This PR attempts the latter but I could be way off

Context

Plain: https://app.plain.com/workspace/w_01J0T5SBKJHGBT6FZZHAMH3GRB/thread/th_01J80NPYZRCDEGXAVPNYRD5NT1/
Coda:
https://coda.io/d/Escalations-Feedback_dHnUHNps1YO/Pipelines-Escalations_suQ4B7FT#Pipelines-Escalations-Board_tu66S__K/r711&view=modal

Linear
https://linear.app/buildkite/issue/PS-111/new-exit-code-when-lock-for-git-mirrors

Changes

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go fmt ./...)

[DrJosh9000]

GitMirrorsLockTimeout should already apply within updateGitMirrors, e.g.:

agent/internal/job/checkout.go

Lines 296 to 309 in 054e0bd

	lockTimeout := time.Second * time.Duration(e.GitMirrorsLockTimeout)
	if e.Debug {
	e.shell.Commentf("Acquiring mirror repository clone lock")
	}
	// Lock the mirror dir to prevent concurrent clones
	cloneCtx, canc := context.WithTimeout(ctx, lockTimeout)
	defer canc()
	mirrorCloneLock, err := e.shell.LockFile(cloneCtx, mirrorDir+".clonelock")
	if err != nil {
	return "", err
	}
	defer mirrorCloneLock.Unlock()

agent/internal/job/checkout.go

Lines 342 to 348 in 054e0bd

	updateCtx, canc := context.WithTimeout(ctx, lockTimeout)
	defer canc()
	mirrorUpdateLock, err := e.shell.LockFile(updateCtx, mirrorDir+".updatelock")
	if err != nil {
	return "", err
	}
	defer mirrorUpdateLock.Unlock()