Develop Component resource for: kubernetes:apiextensions.k8s.io:CustomResource

[mjeffryes]

CustomResource is a generic resource for creating a Kubernetes custom resource; see example. Today it is implemented as an 'overlay' resource and has two major limitations:

• not supported in Pulumi YAML; see Implement a blocklist and improve errors when unsupported resources are created pulumi-yaml#158
• not supported in Pulumi Java SDK

Conceptually,CustomResource represents a single Kubernetes object akin to ConfigMap or Deployment. It's not a component resource and should continue to benefit from the usual resource lifecycle (e.g. diff).

For most use-cases, a reasonable alternative to CustomResource is ConfigGroup with its objs input property, which allows for literal Kubernetes objects. That said, a component resource is an awkward substitute for a custom resource.

Note that the Pulumi Converter for Kubernetes (kube2pulumi and pulumi-converter-kubernetes) doesn't support custom resources (see Limitations).

### Resolves
- [ ] https://github.com/pulumi/pulumi-kubernetes/issues/1082
- [ ] https://github.com/pulumi/pulumi-kubernetes/issues/1115

[EronWright]

I was curious as to why Pulumi YAML is unable to simply use the CustomResource as provided by the kubernetes provider without an overlay. Imagine we removed the special-case logic (here) and allowed the YAML engine to register the resource. We see the essential problem is a limitation of Pulumi schema: lack of support for open-ended types w/ extra properties.

For example, given a YAML program:

resources:
  myCustomResource:
    type: kubernetes:apiextensions.k8s.io:CustomResource
    properties:
      apiVersion: stable.example.com/v1
      kind: CronTab
      metadata:
        name: my-new-cron-object
      spec:
        cronSpec: "* * * * */5"

The YAML engine obtains schema info from the provider, and is unable to handle the spec property.

    Error: Property spec does not exist on 'kubernetes:apiextensions.k8s.io:CustomResource'
      on Pulumi.yaml line 16:
      16:       spec:
    Cannot assign '{apiVersion: string, kind: string, metadata: {name: string}, spec: {cronSpec: string, image: string}}' to 'kubernetes:apiextensions.k8s.io:CustomResource':
      Existing properties are: kind, others, apiVersion, metadata

Here's the schema for CustomResource (overlays.go).

At the SDK level, the overlay types are defined in a way that Pulumi schema cannot represent, e.g. the special otherFields section in customResource.ts:

export interface CustomResourceArgs {
    apiVersion: pulumi.Input<string>;

    kind: pulumi.Input<string>;

    metadata?: pulumi.Input<inputs.meta.v1.ObjectMeta>;

    [othersFields: string]: pulumi.Input<any>;
}

The SDK simply disregards the underlying schema. In other words, the SDKs emulate support for open-ended schemas, and take advantage of the fact that inputs aren't validated against the schema by the engine.

[EronWright]

We chose to address the use-case behind this issue in a limited way, by implementing CustomResource for the Java SDK.

For the Pulumi YAML SDK, it isn't practical to provide an implementation at this time. Please consider using the ConfigGroup resource, i.e. the objs property that lets you define Kubernetes objects similarly to CustomResource. See this blog post:
New: ConfigGroup, ConfigFile resources for Java, YAML SDKs