fix(preempt): keep nominated gang tasks eligible and retry stranded gang placements

[pfernandes21]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Prio-0 multi-worker training_task gangs (reserved-only, no spillover) stalled for 20+ minutes on Delphi while the reserved B200 pool was 100% allocated by lower-priority gangs, even though preempting them was allowed. Two compounding causes in preempt, both only reachable by gangs above their victims' priority in a saturated pool:

1. Nominated node bail-out (upstream, e9040d33). taskEligibleToPreempt returned an error when Pod.Status.NominatedNodeName was set and the node passed predicates:

not eligible due to the pod's nominated node is already schedulable, which should not happen as preemption means no node is schedulable

Predicates ignore resource fit, so this fires whenever a task was pipelined + un-pipelined in a previous cycle (allocate/preempt found one node, the gang did not reach minAvailable, the statement was discarded, but the nominatedNodeName stuck — Statement.Discard records LastTransaction and the cache only ever sets NominatedNodeName, never clears it). From then on worker-0 is skipped every cycle, worker-1 alone can never satisfy the gang, and the transaction is discarded forever. Observed sequence on akbmmrhwvjgn4tgfwlrb-fjkyifby-0:

22:44:29  worker-0 pipelined to ip-10-82-77-191 ; worker-1: no candidates ; stmt discarded
22:44:29  worker-0 nominatedNodeName = ip-10-82-77-191
22:44:32+ worker-0: "nominated node is already schedulable"   (every cycle until capacity freed naturally)

Fix: drop that branch (marked as an intentional divergence from upstream). The nominated-node fast path is allocate's job (allocate.go already checks FutureIdle on the nominated node); preempt keeps the task eligible and lets pipelineOnFittingNode / candidate search decide. The UnschedulableAndUnresolvable and terminating-victim checks are unchanged. PrePredicateFn now runs before the eligibility check, since predicating the nominated node reads the task's cycle state and allocate does not always leave it behind (e.g. task skipped because the queue was overused).

2. Stranded gang placement. Tasks are placed one at a time, so the node chosen for worker-0 can leave worker-1 with zero candidates — here ip-10-82-77-191 sat in a 3-node capacity reservation whose other two nodes ran prio-0 jobs, so the EFA pod affinity (karpenter.k8s.aws/capacity-reservation-id) gave worker-1 nowhere to go. A single pass discards and repeats the same choice next session.

Fix: preemptForJob retries the job's transaction, excluding the nodes the abandoned attempt pipelined onto, up to gangPlacementRetries times (new action arg, default 2, 0 disables):

for attempt := 0; ; attempt++ {
    stmt := framework.NewStatement(ssn)
    assigned = pmpt.preemptJobTasks(ssn, stmt, job, tasks[job.UID], jobPredicateHelper(helpers, job.UID), excludedNodes)
    if ssn.JobPipelined(job) { stmt.Commit(); return assigned, true }
    chosen := pipelinedNodes(job, alreadyCommitted)
    stmt.Discard()
    if attempt >= pmpt.gangPlacementRetries || chosen.Len() == 0 {
        if attempt > 0 { clearLastTxContexts(job) }   // don't publish the worst attempt as the nomination hint
        return false, false
    }
    excludedNodes = excludedNodes.Union(chosen)
    helpers[job.UID] = util.NewPredicateHelper()       // error cache is keyed by task role
    tasks[job.UID] = pendingPreemptorTasks(ssn, job)
}

• Excluded nodes are removed from the candidate list before PredicateNodes samples/truncates it.
• Predicate helpers are held per job and the job's helper is replaced on retry, so attempt-specific cache entries do not leak into the job's later rounds or the same-job loop.
• A discarded attempt never issues a real eviction (Statement.Evict only mutates session state; the API call happens in Commit).

Execute's cross-job loop body moved into preemptForJob/preemptJobTasks; the same-job loop is unchanged apart from sharing pendingPreemptorTasks and the per-job helper.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Heron review addressed (see comments): allNodes = ssn.NodeList bypass dropped — allocate checks resource fit before predicates, so on a full node it records Insufficient cpu (Unschedulable), never a sibling-affinity U&U verdict, so there was nothing to bypass.

New cases:

• TestPreempt and TestTopologyAwarePreempt: task nominated to a node that passes predicates still preempts for its gang — runs allocate then preempt (so the predicate cycle state exists, as in production). With the old bail-out restored it fails with 0 evictions and logs the exact production error.
• TestTopologyAwarePreempt: retry gang placement when the first task lands in a domain its siblings cannot join — zone-a node holds the cheapest victim but is alone in its zone; the retry moves the gang to zone b. Deterministically fails with gangPlacementRetries: 0, without node exclusion, or without the fresh predicate helper.
• TestTopologyAwarePreempt: retry gang placement when the first task fits a draining node its siblings cannot join — the production shape: allocate pipelines worker-0 onto a node with releasing resources, strands worker-1, discards; preempt's first attempt repeats it for free; the retry moves the gang.

Retry cases predicate every node (as Delphi does with minCandidateNodesPercentage: 100) and every gang case asserts ExpectTaskStatusNums: {Pipelined: 2}.

Verified: gofmt, go vet, go test ./pkg/scheduler/actions/... ./pkg/scheduler/framework/... ./pkg/scheduler/util/...; preempt package green at -count=10.

Not a training_task regression: the generated pod spec (priority class, reserved-only affinity, EFA pod affinity, resources) is identical to gangs that scheduled fine earlier the same day.

Does this PR introduce a user-facing change?

preempt: a task whose nominatedNodeName passes predicates is no longer rejected as "already schedulable"; gang preemption transactions are retried with previously chosen nodes excluded (new `gangPlacementRetries` action argument, default 2).

Link to Devin session: https://app.devin.ai/sessions/9618e4fbaeb345ba9c161ec18bfdec6c
Open in Devin Desktop: https://app.devin.ai/desktop/session/9618e4fbaeb345ba9c161ec18bfdec6c?variant=devin
Requested by: @pfernandes21

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[exa-heron-staging]

Review

I traced the whole latch end-to-end and ran ablations against the two new cases. The diagnosis is right and the Statement bookkeeping is sound. Two things I'd fix before merge: the retry test is only a ~40%-green guard, and the allNodes = ssn.NodeList escape hatch is both unjustified by its comment and untested.

---

1. Root cause confirmed — and it's a permanent latch, not a transient one

The chain is exactly as described, and it's worth writing down because it explains why the pod never recovers:

1. topologyAwarePreempt pipelines with stmt.Pipeline(preemptor, bestCandidate.Name(), true) → EvictionOccurred = true.
2. Gang misses minAvailable → Statement.Discard() calls op.task.GenerateLastTxContext() before undoing each op (framework/statement.go:392-399), so the preemptor keeps LastTransaction = {Pipelined, node, EvictionOccurred: true}.
3. JobInfo.TaskSchedulingReason returns nominatedNodeName = ctx.NodeName for exactly that combination (api/job_info.go:829-834).
4. RecordJobStatusEvent → taskUnschedulable(..., nominatedNodeName) (cache/cache.go:1640,1644) patches the pod.
5. updateNomiNode := len(nominatedNodeName) > 0 && ... (cache/cache.go:1044) — volcano only ever sets NominatedNodeName, never clears it. Upstream kube-scheduler clears it (ClearNominatedNodeName / NewPostFilterResultWithNominatedNode("") in preemption.go).
6. Next session, the old taskEligibleToPreempt bails with "already schedulable" — forever.

So the deeper bug is (5): a discarded preemption publishes a nomination that nothing can retract. This PR neutralises the harm, which is the right call for a hotfix, but consider a follow-up that clears NominatedNodeName when a preempt statement is discarded, mirroring upstream. Otherwise the stale nomination keeps steering allocate's fast path (allocate/allocate.go:601-604) at a node that was deliberately abandoned.

2. Change (1) — eligibility: correct, and it converges on upstream kube-scheduler

The removed branch came from upstream volcano e9040d33 ("Preempt action support topology"), not from this fork. Upstream kube-scheduler PodEligibleToPreemptOthers has no such rule — it is only preemptionPolicy=Never → ineligible, nominated node UnschedulableAndUnresolvable → eligible, terminating-by-preemption pod on the nominated node → ineligible (default_preemption.go). The rewritten function is that, exactly.

One consequence that's easy to miss reading the diff and is worth a line in the description: the only semantic delta is the "predicates pass" case, and it now falls through to the terminating-victim loop, which master skipped via the early return. That's the upstream behaviour and it's what keeps this from becoming a re-eviction thrash source — plus pipelineOnFittingNode (topology path) and the ssn.Allocatable(...) && InitResreq.LessEqual(node.FutureIdle()) break in normalPreempt both refuse to evict when the task already fits. I convinced myself there's no gratuitous-eviction regression here.

Please file this upstream (volcano-sh/volcano) or at least leave a // EXA: marker — it's a silent divergence that the next rebase will happily reintroduce.

3. Change (2) — retry loop: bookkeeping is safe, blast radius is well scoped

Things I verified rather than assumed:

• Retries cannot cause real evictions. Statement.Evict only mutates session state (statement.go:72-108); the API call happens in Commit → s.evict → ssn.cache.Evict (statement.go:418-425, 111-121). A discarded attempt is free at the API level. This is the load-bearing property of the whole design and it holds.
• committedTasks snapshot is correct. sets.KeySet copies keys, pipelinedNodes is computed before Discard() (which nulls task.NodeName in UnPipeline), and previously-committed pipelined tasks are correctly exempt from exclusion.
• Re-queueing after discard is correct. UnPipeline → UpdateTaskStatus(task, api.Pending) puts tasks back in TaskStatusIndex[Pending], so the post-Discard() pendingPreemptorTasks refill picks them up.
• excludedNodes = nil from the same-job call site is safe — Len()/Has() on a nil sets.Set[string] are fine.
• No new infinite-loop risk in the preemptors.Push / assigned interaction; the (false,false) and (false,true) paths both terminate as on master.
• Normal (non-topology) preemption is barely affected. chosen.Len() == 0 means a job that pipelined nothing never retries, and in practice a single-task job that pipelines is immediately JobPipelined. Without the gang plugin, ssn.JobPipelined returns true by default (framework/session_plugins.go), so retries never fire at all.

The residual cost case is a large gang: 63 of 64 workers pipeline, the 64th finds no capacity, and we now throw away all 63 and re-run the full pass twice with those 63 nodes excluded — near-certain to fail, at the price of 2 extra DryRunPreemption sweeps per starving job per session. On Delphi that's up to 1000 candidate nodes per task (minCandidateNodesPercentage: 100, maxCandidateNodesAbsolute: 1000, infra/core/delphi/training-stack.ts:508-510). Same end state as master, just more CPU. Not blocking, but the retry would be better gated on "the failing task had zero candidates" (the stranded-domain signature) than on "the job isn't pipelined" (which is also the signature of plain insufficient capacity).

Minor, but real: after a fully-failed retry sequence, the nomination published to the API (§1) comes from the last attempt — i.e. the node chosen after the good ones were excluded. So for gangs that never succeed, the retry actively degrades the hint that allocate uses next session. ClearLastTxContext() on the preemptor tasks before the final return false, false would avoid that.

4. allNodes = ssn.NodeList on retry — please drop or justify

This is the change I'd push back on hardest.

The comment says allocate "recorded those statuses against the sibling placement now being abandoned". FilterOutUnschedulableAndUnresolvableNodesForTask reads job.NodesFitErrors[task.UID] (framework/session.go:673-696), and that map is only written by allocate (allocate.go:582,594,624) and backfill (backfill.go:78,84,107) — never by preempt. Those statuses were recorded before preempt ran and cannot have been influenced by the placement the retry is abandoning. The real justification (if there is one) is narrower: allocate's own U&U verdicts for gang members can depend on allocate's own sibling placements. The comment as written doesn't match the code.

Effects of the bypass:

• Every task in every retry now predicates the entire cluster.
• If everything fails predicates, the fork's empty-predicateNodes fallback re-adds every non-U&U node from ssn.NodeList, so findCandidates can dry-run the whole cluster rather than the allocate-filtered subset.
• It is pinned by nothing. Removing it entirely: 0 failures in 15 runs of TestTopologyAwarePreempt (table below).

I'd delete it. If it's genuinely needed, it needs a case that fails without it.

5. Predicate error cache — the reset is one level too low

predicateHelper = util.NewPredicateHelper() rebinds the parameter, so the outer ph built in Execute still holds the attempt-0 entries, keyed job/taskRole (util/predicate_helper.go:137). That same ph is then handed to (a) the re-queued job's next preemptForJob round and (b) the same-job preemption loop at the end of Execute. The exact leak the comment describes therefore still happens, just one frame up. Either thread a per-job helper through both call sites or drop the reset — the current state is half a fix with a comment claiming a whole one. (For what it's worth, it is load-bearing today: removing it fails the retry case 9/15.)

6. Node exclusion is applied after truncation

PredicateNodes truncates its result to CalculateNumOfFeasibleNodesToFind (predicate_helper.go:57,131) before the new slices.DeleteFunc runs on it. With the default --percentage-nodes-to-find=0 / --minimum-feasible-nodes=100, any cluster above 100 nodes gets an adaptive cap (e.g. 230 of 500), so a retry can spend its entire node budget on nodes it is about to delete and end up with a short — or empty — candidate list. Filtering allNodes before PredicateNodes is cheaper, immune to this, and removes the need for the slices.Clone.

7. Do the new cases pin the behaviour? Case 1 yes, case 2 no

I ablated each mechanism and ran TestTopologyAwarePreempt 15× per variant (-count=15):

ablation                                        retry case   nominated case
restore "already schedulable" bail-out            0/15          15/15  FAIL
gangPlacementRetries: 0                           9/15           0/15
drop fresh NewPredicateHelper() on retry          9/15           0/15
drop slices.DeleteFunc node exclusion             3/15           0/15
drop allNodes = ssn.NodeList on retry             0/15           0/15
baseline (this branch)                            0/15           0/15

Case 1 is a good deterministic regression test — 15/15 red when the bail-out is restored. Ship it.

Case 2 is a flaky guard. With retries completely disabled it still goes green 6 times out of 15, so a future regression has a ~40% chance of sailing through CI, and the node-exclusion mechanism it's ostensibly testing is caught only 3 times in 15. The nondeterminism is GetOffsetAndNumCandidates → rand.Intn(numNodes) combined with the test's MinCandidateNodesAbsoluteKey/MaxCandidateNodesAbsoluteKey: 2 over a 3-node cluster: only 2 of 3 nodes are dry-run, from a random offset. (lastProcessedNodeIndex in predicate_helper.go:78,134 adds a second rotation.) The case passes mostly because a re-roll finds a good placement, not because exclusion steered it.

Concrete fix, which I verified: make the candidate sampling exhaustive for this case — MinCandidateNodesPercentageKey: 100 plus candidate absolutes ≥ node count. That also matches what Delphi actually runs. Result over 15 runs each:

• baseline: 15/15 pass
• gangPlacementRetries: 0: 15/15 fail
• no node exclusion: 15/15 fail

Two smaller test points:

• ExpectPipeLined only checks that each pipelined task's node is in the expected list — it never checks the count (uthelper/helper.go:374-388). Neither case actually asserts the gang reached minAvailable, which is the entire point of both. Add ExpectTaskStatusNums: {"c1/pg3": {api.Pipelined: 2}}.
• Both changes live on the shared preempt/preemptForJob path, but both cases only run with enableTopologyAwarePreemption: true. The nominated-node case in particular should also exist in TestPreempt so the normal path is covered.

8. Nits

• if test.Name == nominatedNodeCase { test.Run(...) } is now the second name-string branch in that loop (after enableNodeOrderScore). An Actions []framework.Action field on the case struct would read better and scale.
• GangPlacementRetriesKey isn't documented — consistent with the other preempt args, but since it's the knob an oncall would reach for when this misbehaves, a line in the action's doc comment would help.
• gofmt/go vet clean; full pkg/scheduler/actions/preempt suite passes on the branch.

---

Summary: change (1) is a clean, upstream-aligned fix for a real permanent latch — I'd merge that on its own today. Change (2) is sound in its Statement handling but ships one unjustified/untested behaviour (ssn.NodeList), one half-applied cache reset, one ordering bug (exclude-after-truncate), and a test that only catches its own regression 60% of the time. All four are small fixes.

[exa-heron-staging]

(continued — my previous comment was cut off mid-section 5)

4. allNodes = ssn.NodeList on retry — please drop or justify

This is the change I'd push back on hardest.

The comment says allocate "recorded those statuses against the sibling placement now being abandoned". But FilterOutUnschedulableAndUnresolvableNodesForTask reads job.NodesFitErrors[task.UID] (framework/session.go:673-696), and that map is written only by allocate (allocate.go:582,594,624) and backfill (backfill.go:78,84,107) — never by preempt. Those statuses were recorded before preempt ran and cannot have been influenced by the placement the retry is abandoning. The defensible version of the argument is narrower: allocate's own U&U verdicts for gang members can depend on allocate's own sibling placements. As written the comment doesn't match the code.

Cost: every task in every retry now predicates the whole cluster, and if everything fails predicates the fork's empty-predicateNodes fallback re-adds every non-U&U node from ssn.NodeList, so findCandidates can dry-run the entire cluster rather than the allocate-filtered subset.

And it is pinned by nothing — removing it entirely gives 0 failures in 15 runs (table in the next comment). I'd delete it; if it's genuinely needed, it needs a case that fails without it.

5. Predicate error cache — the reset is one level too low

predicateHelper = util.NewPredicateHelper() rebinds the parameter. The outer ph built in Execute still holds the attempt-0 entries, keyed job/taskRole (util/predicate_helper.go:137), and that same ph is then handed to (a) the re-pushed job's next preemptForJob round and (b) the same-job preemption loop later in Execute. So the leak the comment describes still happens, one level up. Either thread a per-job helper through both call sites, or drop the reset — as written it's half a fix.

6. Node exclusion is applied after truncation

PredicateNodes truncates its result to CalculateNumOfFeasibleNodesToFind (predicate_helper.go:57,131) before slices.DeleteFunc runs on it. Defaults are MinNodesToFind=100, PercentageOfNodesToFind=0 (adaptive), so above ~100 nodes the retry can spend its whole node budget on nodes it is about to delete and come back with a short or empty candidate list — exactly on the large clusters where the retry matters. Filtering allNodes before PredicateNodes is cheaper, immune to this, and removes the need for slices.Clone.

[exa-heron-staging]

7. The two new cases: one is a solid guard, one is a coin flip

I ablated each piece of the fix and ran TestTopologyAwarePreempt -count=15 at c0fb7b02:

ablation	nominated case	retry case
restore "already schedulable" bail-out	15/15 fail	0/15
gangPlacementRetries: 0	0/15	9/15 fail
no fresh NewPredicateHelper() on retry	0/15	9/15 fail
no slices.DeleteFunc node exclusion	0/15	3/15 fail
no allNodes = ssn.NodeList	0/15	0/15

Case 1 is a proper regression test. Case 2 is not: with retries fully disabled it still goes green 40% of the time, and the node-exclusion mechanism it is named after is caught only 20% of the time. The nondeterminism is GetOffsetAndNumCandidates → rand.Intn(numNodes), combined with the test's MinCandidateNodesAbsoluteKey/MaxCandidateNodesAbsoluteKey: 2 over 3 nodes — only 2 of 3 nodes are dry-run, from a random offset. (lastProcessedNodeIndex, the package global at predicate_helper.go:78,134, adds a second source.) The PR description's claim that the case "fails with gangPlacementRetries: 0" is true only most of the time.

Verified fix — make the sampling exhaustive for this case, which also matches what Delphi actually runs (minCandidateNodesPercentage: 100, maxCandidateNodesAbsolute: 1000, infra/core/delphi/training-stack.ts:508-510):

MinCandidateNodesPercentageKey: 100,
MinCandidateNodesAbsoluteKey:   100,  // >= len(Nodes)
MaxCandidateNodesAbsoluteKey:   100,

With that, over 15 runs each: baseline 15/15 pass, gangPlacementRetries: 0 15/15 fail, no-node-exclusion 15/15 fail. That turns case 2 into a real guard.

Two more test gaps:

• ExpectPipeLined only checks membership of each pipelined task's node in the expected list (uthelper/helper.go:374-388); it never asserts how many tasks pipelined. So neither case actually asserts the gang reached minAvailable, which is the entire point of both. Add ExpectTaskStatusNums: {"c1/pg3": {api.Pipelined: 2}}.
• Both cases live in TestTopologyAwarePreempt, so enableTopologyAwarePreemption: false is untested for either change even though taskEligibleToPreempt and preemptForJob are on the shared path. The nominated-node case at least should also go in TestPreempt.

8. Nits

• if test.Name == nominatedNodeCase { test.Run(...) } is the second name-based branch in that loop (after enableNodeOrderScore := test.Name != "disable node-order score..."). An Actions []framework.Action field on the case struct would kill both.
• After a fully-failed retry sequence the nomination published to the API comes from the last attempt — the node picked after the good ones were excluded — so for gangs that never succeed the retry actively degrades the hint allocate uses next session (allocate.go:601-604). ClearLastTxContext() on the preemptor tasks before the final return false, false would avoid that.
• gangPlacementRetries isn't documented, though neither are the other preempt args, so that's consistent.

---

Bottom line: ship (1) — minimal, matches upstream kube-scheduler, properly tested. For (2) I'd want the retry case made deterministic and allNodes = ssn.NodeList dropped (or covered by a test) before merge; §5 and §6 are worth fixing in the same pass. Nothing I found breaks Statement pipeline/unpipeline/evict bookkeeping, gang accounting, or same-job preemption — in particular Statement.Evict only mutates session state, so a discarded retry never issues a real eviction.

[devin-ai-integration]

Thanks — all of §2–§7 addressed in 13e374e and 038e250 (replying top-level since the review comments are not inline threads).

§1 latch chain — agreed on all six steps. Clearing NominatedNodeName on discard (mirroring kube-scheduler) is a follow-up, not this PR.

§2 — added an explicit "diverges from upstream" note on the removed bail-out. Also moved PrePredicateFn ahead of taskEligibleToPreempt: predicating the nominated node reads the task's cycle state, which allocate does not always leave behind (task skipped because the queue was overused), so the old order could reject the task with cycleState: not found.

§3 stale nomination — clearLastTxContexts(job) now runs on the fully-failed exit when at least one retry happened, so the last attempt's placement (made with the better nodes excluded) is not published as the hint. A single failed attempt keeps upstream behaviour. Large-gang cost: kept the unconditional retry (gangPlacementRetries: 0 is the off switch); gating on "the failing task had zero candidates" is a reasonable refinement if the extra dry-runs show up in latency.

§4 allNodes = ssn.NodeList — dropped. I tried to build the case that would pin it (allocate-first, draining node in zone a, sibling affinity) and it cannot exist: alloc.predicate checks resource fit before running predicates, so on a full node allocate records Insufficient cpu (Unschedulable), never an affinity U&U verdict. Nothing needs to bypass FilterOutUnschedulableAndUnresolvableNodesForTask. The scenario survives as the drainingNodeCase test since it is the production shape.

§5 — helpers are now held per job (map[api.JobID]util.PredicateHelper) and the job's entry is replaced on retry; both preemptForJob and the same-job loop go through jobPredicateHelper.

§6 — excluded nodes are removed from allNodes before PredicateNodes; the post-truncation DeleteFunc is gone.

§7 — retry cases run with min/max candidates = len(test.Nodes); both gang cases assert ExpectTaskStatusNums: {Pipelined: 2}; the nominated-node case is duplicated into TestPreempt (normal preemption, 2 victims, 2/2 pipelined). Whole preempt package green at -count=10.

§8 — TestCommonStruct is the shared uthelper type, so I used two named case sets (allocateFirstCases, exhaustiveCandidateCases) instead of adding an Actions field there. GangPlacementRetriesKey has a doc comment on the const.

[devin-ai-integration]

CI: E2E about Sequence failed on 038e250 in Job Error Handling … PodFailed→RestartPod / PodEvicted→TerminateJob (job stuck in Restarting, expected Running — no preemption involved). The same suite passed on 13e374e two minutes earlier (https://github.com/exa-labs/volcano/actions/runs/33580611490) and on c0fb7b0; the only diff since is clearLastTxContexts on the preempt retry-exhaustion path, which that test never reaches. Needs a manual re-run of the failed job — the agent token can't rerun in this repo and the workflow-trigger service is monorepo-only.