dns gateway: create manifest for every filter

[wbpcode]

• make the dns gateway a rust extension bundle by creating sub manifests for every sub extensions.

we even could do this:

boe run --local ../extensions/dns-gateway/manifests/dns-gateway-resolver --config '{
            "domains": [
              {
                "domain": "*.aws.com",
                "base_ip": "10.0.0.0",
                "prefix_len": 24,
                "metadata": {
                  "cluster": "aws_cluster"
                }
              }
            ]
          }' --local ../extensions/dns-gateway/manifests/dns-gateway-lookup

Manifest is use unique identifier of an unique filter.

[nacx]

Nice! I see this works fine for local extensions, but I think we want to have a consistent experience for remote extensions as well.

How can this be packaged in a way that allows usign it with --extension as well?
I guess the idea is to use the bundle concept and do something like:
boe run --extesion dns-gateway/dns-gateway-lookup.

If that's the idea:

• What about renaming to remove repetition (e.g. dns-gateway/lookup)?
• What needs to tb changed in the packaging so that --extension works fine?

[wbpcode]

How can this be packaged in a way that allows usign it with --extension as well?

It's has been supported natively. Both the legacy:

./cli/out/boe-linux-amd64 run   --extension dns-gateway:0.1.0 --filter-type udp_listener \
          --extension dns-gateway:0.1.0 --filter-type network  \
          --registry registry.local:5000 --insecure   --log-level dynamic_modules:debug --config '
          {
            "domains": [
              { "domain": "*.aws.com", "base_ip": "10.0.0.0", "prefix_len": 24 }
            ]
          }' --config '{"filter_state_prefix": "egress.dns"}'

and the new:

./cli/out/boe-linux-amd64 run   --extension dns-gateway/dns-gateway-resolver \
          --extension dns-gateway/dns-gateway-lookup \
          --registry registry.local:5000 --insecure   --log-level dynamic_modules:debug --config '
          {
            "domains": [
              { "domain": "*.aws.com", "base_ip": "10.0.0.0", "prefix_len": 24 }
            ]
          }' --config '{"filter_state_prefix": "egress.dns"}'

works. I personally thought the local is more interesting so I used that in the description. We needn't any additional work at packing. Same to future other extensions, our bundle design works perfect for both local and remote extensions if you create the manifests correctly.

[wbpcode]

What about renaming to remove repetition (e.g. dns-gateway/lookup)?

SGTM. Let me update the naming.

[wbpcode]

The name updated and fixed an example configuration spelling error.

[nacx]

Thanks!
I didn't remember this part

built-on-envoy/extensions/Dockerfile.rust

Lines 29 to 31 in 91a860c

	# Collect all sub-extension manifest.yaml and config.schema.json files preserving their directory structure.
	RUN mkdir -p /metadatas; find . -mindepth 2 \( -name 'manifest.yaml' -o -name 'config.schema.json' \) \
	-exec sh -c 'install -D "$1" /metadatas/"$1"' _ {} \;

[nacx]

@wbpcode I've pushed a commit to this PR, PTAL.

It basically fixes the extension catalog generation and UI, as they currently skip bundle extensions, assuming they're not runnable. This caused the parent dns-gateway to not be rendered properly.
I've deprecated the extensionSet: bool in favor of an extensionSetType: <metadata-only|runnable>. We cannot just replace the existing type from bool to enum as it would break the boe run for older CLI versions as they'll fail unmarshalling the manifest.

[wbpcode]

It basically fixes the extension catalog generation and UI, as they currently skip bundle extensions, assuming they're not runnable. This caused the parent dns-gateway to not be rendered properly.

But the new lookup and cache will be there, right? So, at the UI, users still could run the dns-gateway as they expected except they cannot see the single dns-gateway?

I am prepare a PR to update the UI to show the bundle name beside the extension name. So, the users known it come from the same bundle.

[wbpcode]

I am a little doubt should we introduce this new concept. 🤔 In my opinion, all extension sets should be metadata only, include the dns-gateway.
We kept the previous dns-gateway name as alias of extension to keep backwards compatibility, to ensure it won't break users in production or CLI.

But in the UI/doc, we may needn't take the backwards compatibility into account? My thoughts is:

• For the the UI, just show new lookup and cache extensions (compare to single extension with different filter types, I guess multiple extensions in same bundle is easier to understand and also match the reality). We can render the bundle name beside the extension name to distinguish them, so users could understand.
• For the document, we should also generate a description page for extension set, and also link the extension set's description to the extensions?

Then users finally will see lookup|dns-gateway or dns-gateway|cache in the UI and document. And if you click, it will see a description to the bundle dns-gateway?

Let me know your thought 🙇

[nacx]

That is fine for the UI.

Now, for the extension catalog on the website, what's your take? I kinda like having the "dns-gateway" bundle as an extension there. If we want to remove it, we also need the "lookup" and "register" child extensions to be shown in a way that they appear related. I mean, one extension alone won't help, so we should render teh catalog in a way that they show "close together", like showing them as "dns-gateway/lookup", for example (bundle_name/extension_name).
The caveat with this is that I don't want the composer extensions to show as composer/opa in the catalog, because the nature of this bundle is different than the nature of the dns-gateway bundle, and I wouldn't want to hardcode "composer" in there... This is why ended up adding a property, but I'm open to suggestions.
Another thing that I considered is just changing the existing extensionSet to a string, but that would break noe run for all existing clients once we push the updated manifests.

[wbpcode]

like showing them as "dns-gateway/lookup", for example (bundle_name/extension_name).

Yeah. That's what I was thinking. And the composer extensions will also be annotated that they are part of composer bundle (what's the side effect for composer extensions if we show them belongs to same composer bundle?)

Basically I personally inclined to design the document/catalog based on the abstracted bundle/extensions and ignore the specific implementation of different bundle/extensions. And don't want to treat the composer as a special case.

Although the nature of composer and dns-gateway bundle are different, it's only something should be documented clearly at the bundle's description page.

From BoE market's perspective, they should be equivalent: a bundle of extensions, and in the future, we may have other bundles with other nature. We could use same solution to handle it. By this way, the code tree, the packaging, and configuration, the UI and catalog, all match the bundle/extensions concept unanimously.

[nacx]

But the new lookup and cache will be there, right?

Yes, the old one, and the two new ones, will appear. The 3 of them.

[nacx]

Ok, I've replaced my last commit with this one to revert my changes and just show the bundle in the catalog and UI: 29cbd9f.

With this, the caveat is that "parent" extensions won't appear in the catalog; only the child ones.

WDYT?

[wbpcode]

Ok, I've replaced my last commit with this one to revert my changes and just show the bundle in the catalog and UI: 29cbd9f.

With this, the caveat is that "parent" extensions won't appear in the catalog; only the child ones.

WDYT?

Thanks so much. I personally prefer this direction. And we can optimize the UI based on feedback to improve experience if necessary.