k8s_resource() doesn't recognize CRD types

[bobjackman]

If yaml is applied which includes instances of a CRD, then k8s_resource() doesn't pick it up -- and as a result, a subsequent call to config. set_enabled_resources() which would effectively exclude that resource from being applied, leaks objects through to k8s.

Example:

yaml = """
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        name: foo
    blah...
    ---
    apiVersion: v1
    kind: Service
    metadata:
        name: foo
    blah...
    ---
    apiVersion: some/other/thing
    kind: SomeCrd
    metadata:
        name: foo
"""
k8s_yaml(yaml)
k8s_resource('foo')

config.set_enabled_resources(['anythingExceptFoo']);

tilt up
kubectl get deployment foo (no results 👍 )
kubectl get service foo (no results 👍 )
kubectl get somecrd foo (1 result found! 👎 )

This can be sidestepped by passing the objects argument when calling k8s_resource():

k8s_resource('foo', objects=['foo:SomeCrd'])

HOWEVER, this shouldn't be necessary. Tilt should recognize ALL types applied via k8s_yaml() whether they're k8s native types or otherwise.

In particular this is causing issues when using git_resource() since it abstracts the call to k8s_resource() and we have no way to tell tilt about the additional objects. I can add an argument to git_resource() and forward it to the subsequent k8s_resource() call (which I did in order to verify this issue), but that complicates things (especially if you're using custom yaml/deploy callbacks) and really shouldn't be necessary anyway.

[maiamcc]

Some clarifying questions here:

1. WHY you want that CRD to be attached to the same resource as this deployment and service? Does it create pods that you want to see logs for? Some other reason? What makes it a logical grouping in your mind? This will inform:
2. how would you expect Tilt to figure out that these k8s entities are all grouped -- just by matching on the name? That seems brittle to me.

For context, by default Tilt assembles resources as: one workload (i.e. pod-creating) entity, and any k8s entities that we can say with certainty should be grouped with that workload. Right now, the only grouping we do automatically is for services that select on precisely the right set of pod selectors to match the pods produced by that workload.

It's definitely possible that we can do more automatic grouping here, but our original intent was to be conservative to avoid miscategorizing anything, and give users an escape hatch for doing grouping that we missed (k8s_resource.objects) -- so I'd love to hear about the specifics of WHY you want this grouping, and how the engine might infer it.

[maiamcc]

(FYI Tilt is on vacation tomorrow, so if you don't hear back from us til Monday, that's why!)

[bobjackman]

@maiamcc for question 1 above, it might be better to have a chat in the slack channel to explain that. But for question 2, I had just assumed that you were already simple-matching on name, but it makes sense that you would be parsing the service-selector to find the relationship, and in my scenario, a simple name-matching wouldn't really be a tenable solution. In that case, I have a couple thoughts:

1. Whenever there are items that tilt is aware of (via k8s_yaml for example) that have NOT been grouped into a resource by the time we're finished executing the tiltfile, maybe a warning would be helpful to alert the user that there may be some items slipping through the cracks.

2. Allow multiple calls to k8s_resource() for the same resource name.
   One of the ways I attempted to work around this issue was by making a second call to k8s_resource() i.e.

   git_resource(blah) # internally calls `k8s_resource(blah)`
   
   k8s_resource(blah, objects=['additionalItems']) # error: resource blah has already been declared
   k8s_resource(blah, objects=['additionalItems'], new_name='newBlah') # also errored

   This would allow ammending the resource "group" a little more dynamically. Maybe something like:

   k8s_resource(blah, objects=['additionalItems'], append=True)
   ## OR ##
   k8s_resource(blah, objects=['additionalItems'], reopen=True)

[nicks]

Ya, i think this is just an oversight when we added k8s_resource(objects)...seems like you should be able to specify it additively, like other api calls that do settings

(i also feel like the resource-grouping primitives still require way too much effort to use well, but that's a slightly tangential tension)

[landism]

I've fleshed out options amending a bit in #3738.