In #1179 we are trying to add evals for a project without any toolset yet. Currently this flow will pass argocd as a toolset when starting the server, which will fail since that toolset does not exist.
The matrix approach should somehow allow for us to indicate whether or not a toolset needs to be enabled, or we just need to match a set of eval tasks to run
In general, we want to be able to run evals without any toolset being enabled, as for many projects the core toolset will support their needs and we just want to show compatibility
In #1179 we are trying to add evals for a project without any toolset yet. Currently this flow will pass
argocdas a toolset when starting the server, which will fail since that toolset does not exist.The matrix approach should somehow allow for us to indicate whether or not a toolset needs to be enabled, or we just need to match a set of eval tasks to run
In general, we want to be able to run evals without any toolset being enabled, as for many projects the core toolset will support their needs and we just want to show compatibility