Proposal: Transition Resource Generation to Jsonnet

[prashanthjos]

I am planning to start moving parts of the codebase to use Jsonnet for resource generation, beginning with serverlessservice.go.
Before opening a PR, I’d like to check if anyone has strong opinions against this approach.

Here’s a snippet of the current Go-based implementation:

vs := resources.MakeVirtualService(sks)
if _, err := istioaccessor.ReconcileVirtualService(ctx, sks, vs, r); err != nil {
	return fmt.Errorf("failed to reconcile VirtualService: %w", err)
}

dr := resources.MakeDestinationRule(sks)
if _, err := istioaccessor.ReconcileDestinationRule(ctx, sks, dr, r); err != nil {
	return fmt.Errorf("failed to reconcile DestinationRule: %w", err)
}

Advantages of a Jsonnet Approach

Declarative vs. Imperative

• Go: Imperative code builds resources step-by-step, with conditional logic scattered throughout.
• Jsonnet: Declarative templates describe the desired end state, making the resource structure immediately visible.

Separation of Concerns

• Go: Business logic (when to create resources) is mixed with resource structure (how resources look).
• Jsonnet: Go handles the “when,” Jsonnet defines the “what.”

Template Reusability & Consistency

• Go: Resource generation logic is often duplicated across controllers.
• Jsonnet: Shared templates/libraries (e.g., istio.libsonnet) ensure consistency across resource generation.

Easier Maintenance & Customization

• Go: Modifying structures requires diving into complex nested conditionals.
• Jsonnet: Resource modifications are straightforward template changes, easier to review and test.

Better Resource Completeness

• Previous Jsonnet implementations generated more complete and spec-compliant Istio resources compared to Go.

Configuration Flexibility

• Go: Hard-coded resource structures with limited parameterization.
• Jsonnet: Highly parameterizable templates adapt to different configurations without code changes.

Next Steps

• Start by migrating serverlessservice.go resource generation to Jsonnet as a pilot PR.
• Gather feedback from maintainers and contributors on readability, reviewability, and maintainability.
• If successful, progressively migrate other controllers/resources to Jsonnet.
• Document the Jsonnet workflow to help contributors adopt the new approach smoothly.

[dprotaso]

I am planning to start moving parts of the codebase to use Jsonnet for resource generation, beginning with serverlessservice.go.

Can you elaborate on your use case/needs?

Previous Jsonnet implementations generated more complete and spec-compliant Istio resources compared to Go.

This doesn't really make any sense. This makes me think this issue is generated by AI

[prashanthjos]

Not AI generated, what I meant previous was other existing CNCF projects.

I have a working PR for moving to jsonnet for serverlessservice, the ingress one seems more complicated and I want to the see what the community thinks about moving to jsonnet starting with serverlessservice.

Personally I want to move to the yaml/json structures for cleaner abstractions.

Please let me know your thoughts.

[dprotaso]

Not AI generated,

Really? Because the section Declarative vs. Imperative doesn't really apply if you look at the code you're looking to refactor.

net-istio/pkg/reconciler/serverlessservice/resources/virtualservice.go

Line 37 in 0300afe

return &v1beta1.VirtualService{

I still don't understand your use case and saying 'cleaner abstractions.' doesn't really say anything specific.

In fact introducing jsonnet into a Go controller introduces more complexity than simplifying things.

[prashanthjos]

Usecase

We are currently using Knative with Istio, along with our own custom networking layer implementation. Due to the complexity of our Istio mesh, we had to build this layer (we implemented the network spec partially), but we think we can reuse much of the existing code provided by net-istio. Our idea is to move the VS/DR generation logics into Jsonnet templates, and then replace those with our own templates during runtime while leveraging the existing net-istio controller code.

Something like this:

// Library consumer registers their templates
templateRegistry.Register("virtualservice", "my-custom-template", customVSTemplate)
templateRegistry.Register("destinationrule", "my-custom-template", customDRTemplate)

// Library uses registered templates
generator := resources.NewJsonnetGeneratorWithTemplate("my-custom-template")

We see this as a feasible approach and consider it the first step in our migration journey.

[dprotaso]

The use of a global registry seems like a smell. A reconciler/controller reconciles N child resources that are highly coupled together. But the registry seems like it could be easy to introduce conflicts eg. two reconcilers that are unrelated use the same template name by accident.

Secondly, the Knative controller framework allows you to bundle different controllers into the same process. If there was a registry conflict you wouldn't know until someone tried to do this bundling. You wouldn't be able to change the name of the original template because you would break a whole different number of implementations. So this seems pretty fragile.

I think a common use case people ask is to tweak existing resources and I've seen examples of this in envoy gateway where people provide patches. Given that I don't see how you would accomplish that using jsonnet. It seems like you would have to recreate the existing template and make patches to your own - again this leads to drift and fragility.

I think overall customization of our resources requires more thought and probably a proposal. Back in the day I was looking at server side apply as a solution but now I'm wondering if MutatingAdmissionPolicy is a newer and potentially better option (https://kubernetes.io/docs/reference/access-authn-authz/mutating-admission-policy/)

[prashanthjos]

I see your point, but what if we don’t want to generate certain resources like IngressGateways, Secrets, or TLSServers that are already created by the Ingress controller, and instead want to use our own?
How would you suggest we approach this, do you have any ideas or thoughts around it?

Honestly, I was also thinking about how to make this code more generic so that it can serve a wider set of users who want to use net-istio but selectively customize parts of the resource generation.

[dprotaso]

I see your point, but what if we don’t want to generate certain resources like IngressGateways, Secrets, or TLSServers that are already created by the Ingress controller, and instead want to use our own?

Then it sounds like it's unique enough that you would want to write your own reconciler/controller.

You could define your own ingress-class so that other net-* plugins don't reconcile the KIngress resources your plugin is interested in. The default can be configured here: https://github.com/knative/networking/blob/8cc248b445a0575054d8a9b551de39a93d4fe267/config/config-network.yaml#L43-L52

[dprotaso]

Honestly, I was also thinking about how to make this code more generic so that it can serve a wider set of users who want to use net-istio but selectively customize parts of the resource generation.

I'm wary of adding complex features to net-istio as I've been working towards making sure the GatewayAPI will have the features we need to replace all the net-* plugins.

https://github.com/knative-extensions/net-gateway-api