handle multiple bad referenced selections in combinator build

[ziaddevv]

when a combinator references multiple builders and more than one has a bad selection status, the backend currently stops at the first error it encounters. This means a retryable reference can hide a later failed reference depending on builder order.

so I guess we have to collect all referenced selection errors or making fatal failures take priority over retryable ones + showing an actionable message for each referenced builder

[ziaddevv]

@audiodude WDYT ?

[audiodude]

It depends on the entire CUJ for users when they schedule a build of a Combinator that references builders with errors.

What would a user expect? Clearly "whack-a-mole" is undesirable, with messages like "Builder X is broken, please retry" and then immediately returning "Builder Y is broken, please retry" after they fix Builder X. So I agree that all failures should be batched.

As far as fatal/retryable, the whole idea of a retryable failure is that we can.....retry it, right? So if the Combinator is trying to build a dependency, it should be able to enter a retry loop. Only after the retries expire does that dependent builder now become a Fatal failure ("Could not build after N attempts....").

This is very important for what I've been saying about "Getting it right" essentially, but you should be able to complete the first pass of the frontend and end-to-end feature without tackling this.

[ziaddevv]

That makes sense , for now I'll leave the current behavior as-is and track the better behavior here at this issue . So as far as I can tell Longer term the combinator should check all referenced builders, report all dependency problems together, keep retryable dependency problems retryable, and only surface fatal status when there is a real fatal dependency failure or retries have been expired.

also to mention that current user experience the user already knows if the builder is ready , retry , failed before building or saving the combinator

as currently we don't block saving even if user chooses a failed builder