dynamic metadata matching on routes doesn't work

[uhthomas]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: dynamic metadata matching on routes doesn't work

Description:

I just cannot get dynamic_metadata matching working on https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-routematch. I am using the envoy.filters.http.jwt_authn and have confirmed that the dynamic metadata exists, but the behavior of the match is the same regardless of whether it does exist or not.

Repro steps:

Setup envoy with the envoy.filters.http.jwt_authn and use payload_in_metadata. The dynamic metadata will be visible in lua, but is impossible to match against in routes.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

admin:
  access_log:
    - name: envoy.access_loggers.stdout
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
  address:
    socket_address:
      address: "::"
      port_value: 9901
      ipv4_compat: true

overload_manager:
  resource_monitors:
    - name: envoy.resource_monitors.global_downstream_max_connections
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
        max_active_downstream_connections: 4096

static_resources:
  listeners:
    - name: listener_http
      address:
        socket_address:
          address: "::"
          port_value: 8080
          ipv4_compat: true
      filter_chains:
        - filters:
            # HTTP connection manager with JWT validation filter configured first
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                            dynamic_metadata:
                              - filter: envoy.filters.http.jwt_authn
                                path:
                                  - key: payload
                                value:
                                  null_match: {}
                          route:
                            cluster: http_backend
                            timeout: 135s
                            retry_policy:
                              retry_on: 5xx
                              num_retries: 2
                              per_try_timeout: 45s
                        - match:
                            prefix: "/"
                          route:
                            cluster: abc_backend
                            timeout: 135s
                http_filters:
                  - name: envoy.filters.http.jwt_authn
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
                      providers:
                        abc:
                          remote_jwks:
                            http_uri:
                              uri: "http://abc/.well-known/jwks.json"
                              cluster: abc_backend
                              timeout: 5s
                            async_fetch:
                              fast_listener: true
                          forward: true
                          from_cookies:
                            - abc_token
                          payload_in_metadata: payload
                      rules:
                        - match:
                            prefix: "/"
                          requires:
                            # provider_name: abc
                            requires_any:
                              requirements:
                                - provider_name: abc
                                - allow_missing_or_failed: {}
                  - name: envoy.lua
                    typed_config:
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
                      inlineCode: |
                        function dump(o)
                          if type(o) == 'table' then
                              local s = '{ '
                              for k,v in pairs(o) do
                                if type(k) ~= 'number' then k = '"'..k..'"' end
                                s = s .. '['..k..'] = ' .. dump(v) .. ','
                              end
                              return s .. '} '
                          else
                              return tostring(o)
                          end
                        end


                        function envoy_on_request(request_handle)
                          local meta = request_handle:streamInfo():dynamicMetadata():get("envoy.filters.http.jwt_authn")
                          request_handle:logInfo(dump(meta))
                        end
                  # Router filter should be last.
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
  clusters:
    - name: http_backend
      type: STRICT_DNS
      lb_policy: LEAST_REQUEST
      connect_timeout: 0.25s
      load_assignment:
        cluster_name: http_backend
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: whoami
                      port_value: 8081
      health_checks:
        - timeout: 1s
          interval: 5s
          unhealthy_threshold: 3
          healthy_threshold: 1
          http_health_check:
            path: /-/readiness
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 4096
            max_pending_requests: 4096
            max_requests: 4096
            max_retries: 4096
            retry_budget: {}
      outlier_detection:
        consecutive_5xx: 3
        interval: 1s
        base_ejection_time: 5s
        max_ejection_percent: 50

    - name: abc_backend
      type: STRICT_DNS
      lb_policy: LEAST_REQUEST
      connect_timeout: 0.25s
      load_assignment:
        cluster_name: abc_backend
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: abc
                      port_value: 3000
      health_checks:
        - timeout: 1s
          interval: 5s
          unhealthy_threshold: 3
          healthy_threshold: 1
          http_health_check:
            path: /-/readiness
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 4096
            max_pending_requests: 4096
            max_requests: 4096
            max_retries: 4096
            retry_budget: {}
      outlier_detection:
        consecutive_5xx: 3
        interval: 1s
        base_ejection_time: 5s
        max_ejection_percent: 50

Logs:

Include the access logs and the Envoy logs.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required.
Please refer to the Bazel Stack trace documentation.

[phlax]

cc @TAOXUY @yanavlasov

[yanavlasov]

Route match occurs at the very beginning of request processing before any HTTP filters had a chance to run. Metadata under envoy.filters.http.jwt_authn will not be there when the route is matched.
However you can call clearRouteCache from LUA filter to make Envoy re-evaluate the route table after JWT filter has completed.

[uhthomas]

Okay, I see, thank you for clarifying. Is there anyway this can be communicated more clearly in the documentation? I didn't really see any mentioned about order of operations, and it isn't really intuitive. What's the point of dynamic metadata from filters if the metadata can't be used by route matching under normal circumstances?

In addition, is there a better way to achieve the desired behavior? I'm looking to essentially send requests with a valid JWT to one backend, and requests without a JWT or an invalid JWT to another. The only thing I could come up with was the dynamic metadata, but it appears that JWT validation occurs after route matching?

[phlax]

@uhthomas we would definitely appreciate clarification in the docs - please do open a PR to address

[yanavlasov]

The call to clearRouteCache from LUA filter will make Envoy re-evaluate routing table and possibly use a different route. You can make it work with the the LUA filter you have.

[uhthomas]

Am I doing something wrong? The function clearRouteCache is nil?

- name: envoy.lua
  typed_config:
    "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
    inlineCode: |
      function envoy_on_request(request_handle)
        request_handle:clearRouteCache()
      end

envoy-1   | [2025-04-19 18:37:09.694][14][error][lua] [source/extensions/filters/common/lua/lua.cc:32] script log: [string "function envoy_on_request(request_handle)..."]:2: attempt to call method 'clearRouteCache' (a nil value)

[yanavlasov]

@mattklein123 @wbpcode to help with LUA script

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.