feat: auto-detect Kubernetes crd schema

[qvalentin]

What does this PR do?

This PR adds an option to automatically detect the Kubernetes schema by parsing the contents of the document. In case a Kubernetes GroupVersionKind (GVK) is detected, the matching schema is retrieved from the CRD catalog.

What issues does this PR fix or reference?

#605

Is it tested? How?

Tests for parsing the Kubernetes Group Version Kind have been added

Updated version of #962 including the following changes:

• only run the auto-detect for files that are registered with the kubernetes schema
• check if the Kubernetes GroupVersionKind is already defined in the main kubernetes schema, if so, don't look it up in the CRD catalog
• add config option for crd catalog uri
• more tests

TODO:

• final testing
• add new options to readme

[tricktron]

@qvalentin Thanks for the nice work. I actually started working on this as well: https://github.com/tricktron/yaml-language-server/tree/f-kubernetes-improvements

However, I just hardcoded the core kubernetes groups but yours looks better in my opinion. I then actually got sidetracked by adding some more features to my https://github.com/tricktron/crd2jsonschema tool so that I could easily test end-to-end with the newest openshift and tekton crds.

Maybe you can also take over some tests from my my branch, e.g. main...tricktron:yaml-language-server:f-kubernetes-improvements#diff-5d1a1b48130b3cb749d3ae36d72e52f8882d6fa5dc9eaedc5d6e606e411b176cR1194-R1219

And lastly, we should then probably add some documentation as well. This might be a breaking change for the helm language server where you hardcode the kubernetes schemas to templates/**. E.g. this is not needed anymore with this pr and I think it even results in some errors if I remember correctly from my e2e tests. Anyway, I won´t have much time to look at this until next Wednesday but still wanted to give you a fast feedback.

[qvalentin]

@tricktron I took a look at the schemaValidation.test.ts but since in my logic the customSchemaProvider has higher priority then the CRD logic the tests don't work with my branch.

But I might copy the config options format.

[qvalentin]

Hi @msivasubramaniaan what are your thoughts on this?

[datho7561]

I'll try to take a look at this sometime this afternoon, it seems really helpful.

[datho7561]

The tests are failing it seems, with a type error

[coveralls]

coverage: 83.637% (-0.04%) from 83.677%
when pulling ba55b1a on qvalentin:crd
into 769a8f7 on redhat-developer:main.

[datho7561]

I'm trying out:

apiVersion: route.openshift.io/v1
kind: Route
spec:

It seems it was trying to use a custom schema provider instead of the crd schema autodetection, so I made this change:

diff --git a/src/languageservice/services/yamlSchemaService.ts b/src/languageservice/services/yamlSchemaService.ts
index 432851f..9102233 100644
--- a/src/languageservice/services/yamlSchemaService.ts
+++ b/src/languageservice/services/yamlSchemaService.ts
@@ -430,6 +430,22 @@ export class YAMLSchemaService extends JSONSchemaService {
       return resolveSchemaForResource([modelineSchema]);
     }

+    if (this.yamlSettings?.kubernetesCRDStoreEnabled) {
+      for (const entry of this.filePatternAssociations) {
+        if (entry.uris && entry.uris[0] == KUBERNETES_SCHEMA_URL && entry.matchesPattern(resource)) {
+          return resolveSchemaForResource([KUBERNETES_SCHEMA_URL]).then((schema) => {
+            const kubeSchema = autoDetectKubernetesSchemaFromDocument(
+              doc,
+              this.yamlSettings.kubernetesCRDStoreUrl ?? CRD_CATALOG_URL,
+              schema
+            );
+            if (kubeSchema) {
+              return resolveSchemaForResource([kubeSchema]);
+            }
+          });
+        }
+      }
+    }
     if (this.customSchemaProvider) {
       return this.customSchemaProvider(resource)
         .then((schemaUri) => {
@@ -473,22 +489,6 @@ export class YAMLSchemaService extends JSONSchemaService {
           }
         );
     }
-    if (this.yamlSettings?.kubernetesCRDStoreEnabled) {
-      for (const entry of this.filePatternAssociations) {
-        if (entry.uris && entry.uris[0] == KUBERNETES_SCHEMA_URL && entry.matchesPattern(resource)) {
-          return resolveSchemaForResource([KUBERNETES_SCHEMA_URL]).then((schema) => {
-            const kubeSchema = autoDetectKubernetesSchemaFromDocument(
-              doc,
-              this.yamlSettings.kubernetesCRDStoreUrl ?? CRD_CATALOG_URL,
-              schema
-            );
-            if (kubeSchema) {
-              return resolveSchemaForResource([kubeSchema]);
-            }
-          });
-        }
-      }
-    }
     return resolveSchema();
   }

After I do that and try to validate the YAML, I get "Unable to load schema..." as an error in the YAML, since the calculated URL doesn't line up with the location of the schema in the repo (the OpenShift CRDs are split across multiple group ids, whereas in the repo with the CRDs all the schemas are in the folder "openshift/v4.11-strict").

After this I decided to try out:

apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
spec:

And that appears to work after applying the above patch, but the patch appears to break many other tests.

[qvalentin]

Hi @datho7561 I'm not sure what should have higher priority, a custom schema provider or the crd check. I would guess the custom schema provider?

I'm just using the lsp with neovim where no custom schema provider is active. I wonder which provider is active in your setup.

So the mysql example is working for me, but indeed the openshift one is not.
I wonder if there is anything we can do for those schemas that do not respect the common convention of apiVersion and Kind for the schema url.

Edit: I took a quick look at the catalog and it seems openshift is the only directory that is not using an API group name. I would say it's just a bad example and should be treated separately.

[datho7561]

@qvalentin

I wonder which provider is active in your setup.

I'm using vscode-yaml; when vscode-yaml is used, there's a custom schema provider active in yaml-language-server that sends a custom LSP request custom/schema/request to the client (vscode-yaml). If vscode-yaml provides a schema in the response, then that one is used, and if vscode-yaml provides an empty response, resolveSchema is called. This means that even if no custom schema is provided, the code to resolve the CRD schema is never called when a custom schema provider is present (which is always for vscode-yaml).

If you move the CRD resolving code into the resolveSchema function instead, that seems to fix it. Then, you can clean up the code by adding the code into the existing loop over the file associations. I'll add a commit that does this.

I wonder if there is anything we can do for those schemas that do not respect the common convention of apiVersion and Kind for the schema url.

I have written a hack to get the correct URL for OpenShift CRDs, I'll also push this as a commit.

[datho7561]

@qvalentin I think this feature is ready for merge after the changes I made. Let me know if you have objections to anything I changed or if it doesn't work for you. I'm planning on merging this Monday otherwise.