How do I check connectivity when using VCL Socket Interface?

[d-uzlov]

Title: Documentation for VCL Socket Interface doesn't have instructions to check connectivity

Description:

Currently example for VCL integration in the documentation goes like this:

1. Modify Envoy config
2. Run VPP with specified config
3. Use specified VCL config

There is no info about what I can use to check that VCL works, check connectivity, check performance, etc.
Simple curl localhost:port doesn't work when VCL is enabled. I seems like I need additional configuration for VPP but so far I couldn't figure it out.

Relevant Links:

• https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/vcl

What I have tried so far, in case this is helpful:

A lot of text, click to open

At first I followed the documentation for VCL Socket Interface. I used startup.conf and vcl.conf suggested by documentation and Envoy envoy-demo.yaml Envoy config from quickstart, with additional sections for VCL.

sudo vpp -c ./vpp-startup.conf
sudo docker run --name envoy --rm -it -v /run/vpp/app_ns_sockets/:/run/vpp/app_ns_sockets/ -v /etc/vpp/:/etc/vpp/ -e ENVOY_UID=0 envoyproxy/envoy:contrib-v1.28-latest --config-yaml "$(cat ./envoy-vcl.yaml)"

In envoy logs there are no errors:

Envoy logs, click to open

[2024-01-22 06:20:51.808][1][info][main] [source/server/server.cc:413] initializing epoch 0 (base id=0, hot restart version=11.104)
[2024-01-22 06:20:51.808][1][info][main] [source/server/server.cc:415] statically linked extensions:
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.xds_delegates: envoy.xds_delegates.kv_store
[2024-01-22 06:20:51.809][1][info][main] [source/server/server.cc:417]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-01-22 06:20:51.810][1][info][main] [source/server/server.cc:417]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-01-22 06:20:51.810][1][info][main] [source/server/server.cc:417]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-01-22 06:20:51.810][1][info][main] [source/server/server.cc:417]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2024-01-22 06:20:51.810][1][info][main] [source/server/server.cc:417]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-01-22 06:20:51.810][1][info][main] [source/server/server.cc:417]   envoy.connection_handler: envoy.connection_handler.default
[2024-01-22 06:20:51.811][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-01-22 06:20:51.811][1][info][main] [source/server/server.cc:417]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-01-22 06:20:51.811][1][info][main] [source/server/server.cc:417]   quic.http_server_connection: quic.http_server_connection.default
[2024-01-22 06:20:51.811][1][info][main] [source/server/server.cc:417]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-01-22 06:20:51.811][1][info][main] [source/server/server.cc:417]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-01-22 06:20:51.812][1][info][main] [source/server/server.cc:417]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.checksum, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.golang, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.language, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.squash, envoy.filters.http.stateful_session, envoy.filters.http.sxg, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2024-01-22 06:20:51.812][1][info][main] [source/server/server.cc:417]   envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2024-01-22 06:20:51.812][1][info][main] [source/server/server.cc:417]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-01-22 06:20:51.812][1][info][main] [source/server/server.cc:417]   envoy.tls.key_providers: cryptomb, qat
[2024-01-22 06:20:51.813][1][info][main] [source/server/server.cc:417]   envoy.resolvers: envoy.ip
[2024-01-22 06:20:51.813][1][info][main] [source/server/server.cc:417]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-01-22 06:20:51.813][1][info][main] [source/server/server.cc:417]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-01-22 06:20:51.813][1][info][main] [source/server/server.cc:417]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-01-22 06:20:51.813][1][info][main] [source/server/server.cc:417]   envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.golang
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   network.connection.client: default, envoy_internal
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.matching.generic_proxy.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2024-01-22 06:20:51.814][1][info][main] [source/server/server.cc:417]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-01-22 06:20:51.815][1][info][main] [source/server/server.cc:417]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-01-22 06:20:51.815][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-01-22 06:20:51.815][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.protocols: dubbo
[2024-01-22 06:20:51.815][1][info][main] [source/server/server.cc:417]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-01-22 06:20:51.815][1][info][main] [source/server/server.cc:417]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2024-01-22 06:20:51.816][1][info][main] [source/server/server.cc:417]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-01-22 06:20:51.816][1][info][main] [source/server/server.cc:417]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-01-22 06:20:51.816][1][info][main] [source/server/server.cc:417]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-01-22 06:20:51.816][1][info][main] [source/server/server.cc:417]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2024-01-22 06:20:51.819][1][info][main] [source/server/server.cc:417]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-01-22 06:20:51.819][1][info][main] [source/server/server.cc:417]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-01-22 06:20:51.819][1][info][main] [source/server/server.cc:417]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-01-22 06:20:51.819][1][info][main] [source/server/server.cc:417]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-01-22 06:20:51.819][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.geoip_providers: envoy.geoip_providers.maxmind
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-01-22 06:20:51.820][1][info][main] [source/server/server.cc:417]   envoy.generic_proxy.access_loggers: envoy.generic_proxy.access_loggers.file
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.vcl.vcl_socket_interface
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-01-22 06:20:51.821][1][info][main] [source/server/server.cc:417]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2024-01-22 06:20:51.822][1][info][main] [source/server/server.cc:417]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2024-01-22 06:20:51.822][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-01-22 06:20:51.822][1][info][main] [source/server/server.cc:417]   filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2024-01-22 06:20:51.822][1][info][main] [source/server/server.cc:417]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-01-22 06:20:51.822][1][info][main] [source/server/server.cc:417]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-01-22 06:20:51.823][1][info][main] [source/server/server.cc:417]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-01-22 06:20:51.823][1][info][main] [source/server/server.cc:417]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-01-22 06:20:51.823][1][info][main] [source/server/server.cc:417]   envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.header_mutation, envoy.filters.http.upstream_codec
[2024-01-22 06:20:51.824][1][info][main] [source/server/server.cc:417]   envoy.network.connection_balance: envoy.network.connection_balance.dlb
[2024-01-22 06:20:51.824][1][info][main] [source/server/server.cc:417]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2024-01-22 06:20:51.824][1][info][main] [source/server/server.cc:417]   envoy.common.key_value: envoy.key_value.file_based
[2024-01-22 06:20:51.825][1][info][main] [source/server/server.cc:417]   envoy.generic_proxy.filters: envoy.filters.generic.router
[2024-01-22 06:20:51.825][1][info][main] [source/server/server.cc:417]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-01-22 06:20:51.826][1][info][main] [source/server/server.cc:417]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-01-22 06:20:51.834][1][info][main] [source/server/server.cc:417]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-01-22 06:20:51.834][1][info][main] [source/server/server.cc:417]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-01-22 06:20:51.835][1][info][main] [source/server/server.cc:417]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2024-01-22 06:20:51.835][1][info][main] [source/server/server.cc:417]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-01-22 06:20:51.835][1][info][main] [source/server/server.cc:417]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-01-22 06:20:51.836][1][info][main] [source/server/server.cc:417]   envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.hyperscan, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2024-01-22 06:20:51.836][1][info][main] [source/server/server.cc:417]   envoy.request_id: envoy.request_id.uuid
[2024-01-22 06:20:51.836][1][info][main] [source/server/server.cc:417]   envoy.regex_engines: envoy.regex_engines.google_re2, envoy.regex_engines.hyperscan
[2024-01-22 06:20:51.836][1][info][main] [source/server/server.cc:417]   envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo
[2024-01-22 06:20:51.837][1][info][main] [source/server/server.cc:417]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.generic_proxy, envoy.filters.network.golang, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.kafka_mesh, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sip_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-01-22 06:20:51.837][1][info][main] [source/server/server.cc:417]   envoy.sip_proxy.filters: envoy.filters.sip.router
[2024-01-22 06:20:51.837][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-01-22 06:20:51.851][1][info][main] [source/server/server.cc:471] HTTP header map info:
[2024-01-22 06:20:51.852][1][info][main] [source/server/server.cc:474]   request header map: 680 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-01-22 06:20:51.853][1][info][main] [source/server/server.cc:474]   request trailer map: 128 bytes: 
[2024-01-22 06:20:51.853][1][info][main] [source/server/server.cc:474]   response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-01-22 06:20:51.853][1][info][main] [source/server/server.cc:474]   response trailer map: 152 bytes: grpc-message,grpc-status
[2024-01-22 06:20:51.877][1][info][main] [source/server/server.cc:837] runtime: {}
[2024-01-22 06:20:51.878][1][warning][main] [source/server/server.cc:701] No admin address given, so no admin HTTP server started.
[2024-01-22 06:20:51.878][1][info][config] [source/server/configuration_impl.cc:159] loading tracing configuration
[2024-01-22 06:20:51.879][1][info][config] [source/server/configuration_impl.cc:118] loading 0 static secret(s)
[2024-01-22 06:20:51.879][1][info][config] [source/server/configuration_impl.cc:124] loading 1 cluster(s)
[2024-01-22 06:20:51.884][1][info][config] [source/server/configuration_impl.cc:128] loading 1 listener(s)
[2024-01-22 06:20:51.887][1][error][connection] [contrib/vcl/source/vcl_io_handle.cc:505] setOption() SOL_SOCKET: sh 1 optname 15 unsupported!
[2024-01-22 06:20:51.888][1][error][connection] [contrib/vcl/source/vcl_io_handle.cc:505] setOption() SOL_SOCKET: sh 2 optname 15 unsupported!
[2024-01-22 06:20:51.891][1][error][connection] [contrib/vcl/source/vcl_io_handle.cc:505] setOption() SOL_SOCKET: sh 3 optname 15 unsupported!
[2024-01-22 06:20:51.892][1][info][config] [source/server/configuration_impl.cc:145] loading stats configuration
[2024-01-22 06:20:51.896][1][warning][main] [source/server/server.cc:901] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-01-22 06:20:51.896][1][info][main] [source/server/server.cc:942] starting main dispatch loop
[2024-01-22 06:20:51.896][1][info][runtime] [source/common/runtime/runtime_impl.cc:579] RTDS has finished initialization
[2024-01-22 06:20:51.897][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2024-01-22 06:20:51.897][1][info][main] [source/server/server.cc:923] all clusters initialized. initializing init manager
[2024-01-22 06:20:51.897][1][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:923] all dependencies initialized. starting workers
^C[2024-01-22 06:21:06.584][1][warning][main] [source/server/server.cc:878] caught SIGINT
[2024-01-22 06:21:06.584][1][info][main] [source/server/server.cc:1005] shutting down server instance
[2024-01-22 06:21:06.584][1][info][main] [source/server/server.cc:947] main dispatch loop exited
[2024-01-22 06:21:06.586][1][info][main] [source/server/server.cc:997] exiting
[2024-01-22 06:21:06.587][1][info][misc] [contrib/vcl/source/vcl_io_handle.cc:146] [0] destroying listener sh 3
[2024-01-22 06:21:06.587][1][info][misc] [contrib/vcl/source/vcl_io_handle.cc:146] [0] destroying listener sh 2
[2024-01-22 06:21:06.588][1][info][misc] [contrib/vcl/source/vcl_io_handle.cc:146] [0] destroying listener sh 1

I then tried to check networking:

$ sudo docker exec envoy netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address
$ sudo vppctl show app
Index     Name                Namespace                               
2         envoy               default
$ sudo vppctl show session verbose
Connection                                                  State          Rx-f      Tx-f      
[0:0][CT:T] 0.0.0.0:10000->0.0.0.0:0                        LISTEN         0         0         
[0:1][T] 0.0.0.0:10000->0.0.0.0:0                           LISTEN         0         0         
Thread 0: active sessions 2
Thread 1: no sessions
$ sudo vppctl show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
local0
$ curl localhost:10000
curl: (7) Failed to connect to localhost port 10000: Connection refused
$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init              
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      632/sshd: /usr/sbin 
tcp        0      0 127.0.0.1:36485         0.0.0.0:*               LISTEN      1764/docker-proxy   
tcp        0      0 127.0.0.1:42125         0.0.0.0:*               LISTEN      162348/node         
tcp6       0      0 :::111                  :::*                    LISTEN      1/init              
tcp6       0      0 :::22                   :::*                    LISTEN      632/sshd: /usr/sbin 
udp        0      0 0.0.0.0:68              0.0.0.0:*                           456/dhclient        
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init              
udp        0      0 127.0.0.1:323           0.0.0.0:*                           647/chronyd         
udp6       0      0 :::111                  :::*                                1/init              
udp6       0      0 ::1:323                 :::*                                647/chronyd

I see that envoy doesn't listen on kernel interface inside docker container, but it is registered in VPP, and there are 2 active sessions, listening on 0.0.0.0:10000.
On the host system there are also no listeners on port 10000.
curl localhost:10000 fails.

I though that the issue is that vppctl show int doesn't have any interfaces, except local0.
I figured out I could setup VPP to connect to host interface. For easier network setup I deployed it in a Kubernetes pod, Envoy and VPP in the same pod:

Kubernetes config, click to open

---
apiVersion: v1
kind: Pod
metadata:
  name: envoy-vpp
  namespace: default
spec:
  containers:
  - name: envoy
    image: envoyproxy/envoy:contrib-v1.28-latest
    command:
    # - sleep
    # - infinity
    - envoy
    - -c
    - /envoy-config.yaml
    volumeMounts:
    - mountPath: /envoy-config.yaml
      name: config
      readOnly: true
      subPath: envoy-config.yaml
    - mountPath: /etc/vpp/vcl.conf
      name: config
      readOnly: true
      subPath: vcl.conf
    - mountPath: /run/vpp
      name: run
  - name: vpp
    image: ligato/vpp-base:23.10
    securityContext:
      privileged: true
    command:
    - vpp
    - -c
    - /vpp.conf
    volumeMounts:
    - mountPath: /vpp.conf
      name: config
      readOnly: true
      subPath: vpp.conf
    - mountPath: /run/vpp
      name: run
  volumes:
  - name: config
    configMap:
      name: config
  - name: run
    emptyDir: {}
---
apiVersion: v1
kind: Pod
metadata:
  name: envoy-novcl
  namespace: default
spec:
  containers:
  - name: envoy
    image: envoyproxy/envoy:contrib-v1.28-latest
    command:
    # - sleep
    # - infinity
    - envoy
    - -c
    - /envoy-config.yaml
    volumeMounts:
    - mountPath: /envoy-config.yaml
      name: config
      readOnly: true
      subPath: envoy-config-novcl.yaml
  - name: vpp
    image: ligato/vpp-base:23.10
    securityContext:
      privileged: true
    command:
    - vpp
    - -c
    - /vpp.conf
    volumeMounts:
    - mountPath: /vpp.conf
      name: config
      readOnly: true
      subPath: vpp.conf
  volumes:
  - name: config
    configMap:
      name: config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config
  namespace: default
data:
  envoy-config.yaml: |
    static_resources:
      listeners:
      - name: listener_0
        address:
          socket_address:
            address: 0.0.0.0
            port_value: 10000
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      host_rewrite_literal: www.envoyproxy.io
                      cluster: service_envoyproxy_io

      clusters:
      - name: service_envoyproxy_io
        type: LOGICAL_DNS
        # Comment out the following line to test on v6 networks
        dns_lookup_family: V4_ONLY
        load_assignment:
          cluster_name: service_envoyproxy_io
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: www.envoyproxy.io
                    port_value: 443
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            sni: www.envoyproxy.io

    bootstrap_extensions:
      - name: envoy.extensions.vcl.vcl_socket_interface
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.vcl.v3alpha.VclSocketInterface
    default_socket_interface: "envoy.extensions.vcl.vcl_socket_interface"

  envoy-config-novcl.yaml: |
    static_resources:
      listeners:
      - name: listener_0
        address:
          socket_address:
            address: 0.0.0.0
            port_value: 10000
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      host_rewrite_literal: www.envoyproxy.io
                      cluster: service_envoyproxy_io

      clusters:
      - name: service_envoyproxy_io
        type: LOGICAL_DNS
        # Comment out the following line to test on v6 networks
        dns_lookup_family: V4_ONLY
        load_assignment:
          cluster_name: service_envoyproxy_io
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: www.envoyproxy.io
                    port_value: 443
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            sni: www.envoyproxy.io

  vpp.conf: |
    unix {
      nodaemon
      # log /var/log/vpp/vpp.log
      full-coredump
      cli-listen /var/run/vpp/cli.sock
      gid vpp
    }

    buffers {
      buffers-per-numa 16384
    }

    session {
      use-app-socket-api
      enable
      event-queue-length 100000
    }
    
    plugins {
      plugin dpdk_plugin.so { disable }
    }
  vcl.conf: |
    vcl {
      # Max rx/tx session buffers sizes in bytes. Increase for high throughput traffic.
      rx-fifo-size 400000
      tx-fifo-size 400000

      # Size of shared memory segments between VPP and VCL in bytes
      segment-size 1000000000

      # App has access to global routing table
      app-scope-global

      # Allow inter-app shared-memory cut-through sessions
      app-scope-local

      # Pointer to session layer's socket api socket
      app-socket-api /var/run/vpp/app_ns_sockets/default

      # Message queues use eventfds for notifications
      use-mq-eventfd

      # VCL worker incoming message queue size
      event-queue-size 40000
    }
---
apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - name: alpine
    image: docker.io/alpine:3.17.3
    command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]
    resources:
      limits:
        memory: 50Mi
---

I manually connected VPP to pod interface:

envoy_ip=$(k1 exec pods/envoy-vpp -it -c vpp -- ip -brief address show eth0 | awk '{print $3}' | awk -F/ '{print $1}')
envoy_cidr=$(k1 exec pods/envoy-vpp -it -c vpp -- ip -brief address show eth0 | awk '{print $3}')
envoy_mac=$(k1 exec pods/envoy-vpp -it -c vpp -- cat /sys/class/net/eth0/address)

k1 exec pods/envoy-vpp -it -c vpp -- vppctl create host-interface v2 name eth0 hw-addr "$envoy_mac"
k1 exec pods/envoy-vpp -it -c vpp -- vppctl set int state host-eth0 up
k1 exec pods/envoy-vpp -it -c vpp -- vppctl set int ip address host-eth0 "$envoy_cidr"
k1 exec pods/envoy-vpp -it -c vpp -- vppctl set interface mtu 1500 host-eth0

k1 exec pods/envoy-vpp -it -c vpp -- vppctl show int
k1 exec pods/envoy-vpp -it -c vpp -- vppctl show session verbose

k1 exec pods/alpine -it -- apk add curl
k1 exec pods/alpine -it -- curl "${envoy_ip}:10000"
# curl: (7) Failed to connect to 10.244.0.71 port 10000 after 0 ms: Couldn't connect to server
# command terminated with exit code 7

It didn't work, curl with VPP IP address fails.

[zuercher]

cc @florincoras @KfreeZ

[florincoras]

Hi @d-uzlov!

When VCL socket interface + vpp are configured, all of envoy's sockets are moved into vpp. Consequently, envoy is only reachable through vpp's interfaces. Those interfaces can be hw interfaces detached from kernel and attached via dpdk or native drivers to vpp or sw interfaces like virtio/af_packet.

In the output above, we can see that envoy is attached to vpp (sh app) and that a listener was created (show session verbose). However, as you've noted, we can also see that vpp has no interfaces (show int). So you've done right by trying to create a new host (af_packet) interface. From the output I can't tell if configuration went through and if eth0 was removed from linux, i.e., it can't be shared by container's linux stack and vpp. Once that's done, and ping from outside the container to vpp's interface works, curl should also start working.

[d-uzlov]

Hi @florincoras
Thank you for your help. It's nice to hear I was going in the right direction.

From the output I can't tell if configuration went through and if eth0 was removed from linux, i.e., it can't be shared by container's linux stack and vpp

I think it wasn't removed. I didn't do anything special to remove it, and ip a doesn't show anything changed, so the problem is likely in this.

$ k1 exec pods/envoy-vpp -it -c vpp -- vppctl show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
host-eth0                         1      up          1500/0/0/0     rx packets                   140
                                                                    rx bytes                   18641
                                                                    tx packets                    13
                                                                    tx bytes                     546
                                                                    drops                        150
                                                                    ip4                           67
                                                                    ip6                            2
local0                            0     down          0/0/0/0       
$ k1 exec pods/envoy-vpp -it -c vpp -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if81: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1544 qdisc noqueue state UP group default 
    link/ether ba:4e:5f:55:31:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.0.80/24 brd 10.244.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b84e:5fff:fe55:310c/64 scope link 
       valid_lft forever preferred_lft forever

Could you hint me what I need to do to remove eth0 from linux?

[d-uzlov]

I mean, surely I can't just do ip link delete eth0. I don't think anything network-related would be able to work in this case. But just in case, I checked, and nothing worked.
ip addr del 10.244.0.80/24 dev eth0 didn't help.
ip link set dev eth0 down didn't change anything either.
The only change I observe when making these changes is curl no longer gets an answer "nobody is listening this port", so it hangs instead of immediately returning an error.

I can't figure out the right way to correctly reroute eth0 traffic to vpp.

[florincoras]

I would've expected the deletion of the the ip address to be enough. Could you make sure you don't copy the mac address of the linux veth to vpp's host interface? Once that's done, and you try pinging the ip that's now only within vpp, run vppctl show error