fix(local): restore metallb e2e path (#65)

* fix(local): restore metallb e2e path

Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

* fix(local): keep cli broker defaults local

Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

* fix(local): narrow localhost defaults

Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

---------

Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

Author: FrankSpitulski

docs/integrator-quickstart.md

@@ -47,22 +47,21 @@ Set the broker endpoint, authentication material, and topic configuration you
 received from the operator in your application configuration.
 
 If you are using the local evaluation environment and it is already deployed,
-start the broker port forwards in one terminal and leave that terminal open
-while you test. The script starts `kubectl port-forward` processes, then opens a
-shell. The port forwards stop when you exit that shell. To create the local
-broker first, use the [Deployment](getting-started.md) evaluation install.
+connect to the CSC Envoy Gateway at its MetalLB address. To create the local
+broker first, use the [Deployment](getting-started.md) evaluation install. On
+macOS, install and start `docker-mac-net-connect` from the local quick start so
+the host can reach the MetalLB IPs.
 
 ```bash
 cd local
-./infra/scripts/with-gateway-port-forwards.sh sh
+make setup-infra deploy-nats
 ```
 
-In the shell opened by that script, or in another terminal while that shell stays
-open, use the local CSC broker endpoint:
+Use the local CSC broker endpoint:
 
 ```bash
-export DSX_MQTT_HOST=127.0.0.1
-export DSX_MQTT_PORT=11883
+export DSX_MQTT_HOST=172.18.200.1
+export DSX_MQTT_PORT=1883
 export DSX_MQTT_TOPIC=test/hello
 ```
 

docs/testing.md

@@ -19,7 +19,15 @@ The local environment includes MQTT benchmark tooling for smoke runs and
 operator-driven benchmark runs. These tests report the observed behavior of the
 current deployment; they do not define product targets.
 
-**Prerequisite:** Performance and benchmark targets require MetalLB or an equivalent LoadBalancer (installed by `make setup-infra`). Without it, `kubectl port-forward` cannot sustain benchmark throughput and tests fail silently.
+**Prerequisite:** Performance and benchmark targets require MetalLB or an
+equivalent LoadBalancer installed by `make setup-infra`. On macOS, start
+`docker-mac-net-connect` so the host can reach the MetalLB IPs. Linux hosts
+normally reach the Docker bridge IPs directly.
+
+Full benchmark targets are sensitive to local host capacity. When they report
+EOFs, readloop stalls, or success-rate misses, capture host CPU and Kubernetes
+pod metrics with the run; the smoke targets are the portable local validation
+path.
 
 The MQTT performance suite exercises combinations of QoS level, retention, and
 deployment topology:

llms-full.txt

@@ -550,7 +550,10 @@ kubectl delete ns dsx --ignore-not-found
 
 ## Local Benchmark Tooling
 
-**Prerequisite:** Performance and benchmark targets require MetalLB or an equivalent LoadBalancer (installed by `make setup-infra`). Without it, `kubectl port-forward` cannot sustain benchmark throughput and tests fail silently.
+**Prerequisite:** Performance and benchmark targets require MetalLB or an
+equivalent LoadBalancer installed by `make setup-infra`. On macOS, start
+`docker-mac-net-connect` so the host can reach the MetalLB IPs. Linux hosts
+normally reach the Docker bridge IPs directly.
 
 The MQTT performance suite exercises combinations of QoS level, retention, and deployment topology (local and federated, both directions).
 
@@ -581,9 +584,11 @@ Most software integrations should use OAuth2. BMS, OT, and device integrations c
 
 ## Connection Settings
 
+Use the local CSC broker endpoint:
+
 ```bash
-export DSX_MQTT_HOST=127.0.0.1
-export DSX_MQTT_PORT=11883
+export DSX_MQTT_HOST=172.18.200.1
+export DSX_MQTT_PORT=1883
 export DSX_MQTT_TOPIC=test/hello
 ```
 

local/Makefile

@@ -5,6 +5,7 @@
 
 PERFORMANCE_E2E_ENV ?= PERF_TEST_PAIRS=1 PERF_TEST_DURATION=2s PERF_TEST_WARMUP=1s PERF_PUBLISH_DELAY=5ms PERF_MIN_SUCCESS_RATE=99
 BENCHMARK_BASIC_FLAGS ?= --duration 5s --connection-clients 50 --connection-rate 25 --fanout-subscribers 20 --p2p-clients 20 --fanin-publishers 20 --fanin-subscribers 3 --fanin-topics 20
+CSC_BROKER_URL ?= tcp://172.18.200.1:1883
 GO_PATH ?= $(shell go env GOPATH 2>/dev/null)
 GO_BIN ?= $(shell go env GOBIN 2>/dev/null)
 E2E_PREREQS_BIN ?= $(if $(GO_BIN),$(GO_BIN),$(if $(GO_PATH),$(GO_PATH)/bin,$(HOME)/.local/bin))
@@ -110,13 +111,13 @@ check-e2e-env: ## Check local Kind/NATS prerequisites for e2e tests
 	done
 
 test-functional: check-e2e-env ## Run functional tests (MQTT + federation)
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqtt-client && go test -count=1 -v ./tests/functional/ -timeout 60s'
+	cd mqtt-client && go test -count=1 -v ./tests/functional/ -timeout 60s
 
 test-performance: check-e2e-env ## Run performance e2e smoke tests
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqtt-client && $(PERFORMANCE_E2E_ENV) go test -count=1 -v ./tests/performance/ -timeout 10m'
+	cd mqtt-client && $(PERFORMANCE_E2E_ENV) go test -count=1 -v ./tests/performance/ -timeout 10m
 
 dummy-bms: check-e2e-env ## Publish looping dummy BMS data to the CSC MQTT broker
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqtt-client && go run ./cmd/dummy-bms --broker "$$CSC_BROKER_URL" --csv examples/dsx_exemplar.csv --schema ../../schemas/asyncapi/bms/bms.yaml'
+	cd mqtt-client && go run ./cmd/dummy-bms --broker "$(CSC_BROKER_URL)" --csv examples/dsx_exemplar.csv --schema ../../schemas/asyncapi/bms/bms.yaml
 
 validate-nats: ## Validate NATS deployment on all clusters
 	@echo "Validating CSC..."
@@ -156,10 +157,10 @@ lint: ## Run linters
 # ============================================================================
 
 benchmark-basic: check-e2e-env ## Run MQTT benchmark basic suite
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqttbs && go run ./cmd/mqttbs run basic-suite --broker "$$CSC_BROKER_URL" $(BENCHMARK_BASIC_FLAGS)'
+	cd mqttbs && go run ./cmd/mqttbs run basic-suite --broker "$(CSC_BROKER_URL)" $(BENCHMARK_BASIC_FLAGS)
 
 benchmark-basic-full: check-e2e-env ## Run full MQTT benchmark basic suite
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqttbs && go run ./cmd/mqttbs run basic-suite --broker "$$CSC_BROKER_URL"'
+	cd mqttbs && go run ./cmd/mqttbs run basic-suite --broker "$(CSC_BROKER_URL)"
 
 benchmark-performance: check-e2e-env ## Run full performance benchmarks
-	./infra/scripts/with-gateway-port-forwards.sh sh -c 'cd mqtt-client && go test -count=1 -v ./tests/performance/ -timeout 10m'
+	cd mqtt-client && go test -count=1 -v ./tests/performance/ -timeout 10m

local/README.md

@@ -57,9 +57,23 @@ make deploy-nats
 
 ### Run Tests
 
-Performance and benchmark targets require MetalLB (installed by `make setup-infra`). Without it, `kubectl port-forward` is used as a fallback but cannot sustain benchmark throughput — tests fail silently with connectivity errors that do not indicate the root cause.
+Performance and benchmark targets require MetalLB, installed by
+`make setup-infra`, so local clients connect through the Envoy Gateway
+LoadBalancer IPs. On macOS, keep `docker-mac-net-connect` running so the host can
+reach those IPs. Linux hosts normally reach the Docker bridge IPs directly.
+
+The default CSC broker endpoint is `tcp://172.18.200.1:1883`. Override
+`CSC_BROKER_URL` only when testing a different reachable broker.
+
+Full benchmark targets can saturate local hosts because they drive thousands of
+MQTT clients through Kind, Envoy Gateway, NATS, and auth-callout. If a full run
+fails with EOFs or success-rate misses, check host CPU and pod metrics before
+treating it as a networking failure.
 
 ```bash
+# Verify host access to the CSC Envoy Gateway after NATS is deployed
+nc -vz 172.18.200.1 1883
+
 # Run functional tests against all candidates
 make test-functional
 
@@ -169,6 +183,6 @@ Run against the local Kind environment:
 make dummy-bms
 ```
 
-The dummy BMS target uses the same local e2e environment and gateway
-port-forward setup as the functional and performance tests. It publishes to the
-CSC broker URL exported by that wrapper.
+The dummy BMS target uses the same local e2e environment and Envoy Gateway
+LoadBalancer path as the functional and performance tests. It publishes to the
+CSC broker at `tcp://172.18.200.1:1883` unless `CSC_BROKER_URL` is overridden.

local/infra/README.md

@@ -15,6 +15,10 @@ The infrastructure consists of:
 
 ## Quick Start
 
+On macOS, install and start `docker-mac-net-connect` before running local tests
+from the host. Linux hosts normally reach the Docker bridge IPs directly. See
+[local/README.md](../README.md#macos-tweaks).
+
 ```bash
 # Setup complete infrastructure (clusters, MetalLB, Envoy Gateway, cert-manager, metrics-server, Keycloak)
 # All operations are parallelized across clusters for maximum speed:
@@ -29,6 +33,9 @@ make setup-infra
 # Verify everything is running
 make verify-infra
 
+# Verify the host can reach the CSC Keycloak HTTPRoute through Envoy Gateway
+curl http://172.18.200.1/realms/event-bus/.well-known/openid-configuration
+
 # Optional: Deploy full observability stack (Prometheus + Grafana)
 make setup-observability
 ```
@@ -171,12 +178,12 @@ Kubernetes Metrics Server provides resource metrics (CPU/memory) for nodes and p
 kubectl top nodes --context kind-csc
 
 # View pod metrics
-kubectl top pods -n event-bus-nats --context kind-csc
+kubectl top pods -n event-bus --context kind-csc
 ```
 
 ## Keycloak (OAuth2 Authentication)
 
-Keycloak provides OAuth2/OpenID Connect authentication for testing the event bus auth callout service. A single Keycloak instance runs in the CSC cluster, and all clusters (CSC, CPC-1, CPC-2) access it via the external MetalLB LoadBalancer IP (172.18.200.1). Host-side tests use localhost port-forwarding because Docker-network LoadBalancer IPs are not reachable from every workstation environment.
+Keycloak provides OAuth2/OpenID Connect authentication for testing the event bus auth callout service. A single Keycloak instance runs in the CSC cluster, and all clusters (CSC, CPC-1, CPC-2) access it via the external MetalLB LoadBalancer IP (172.18.200.1). Host-side local tests use the same Envoy Gateway path.
 
 **Deployment:**
 
@@ -199,18 +206,11 @@ make setup-keycloak
 
 **Access:**
 
-Keycloak is exposed via Envoy Gateway HTTPRoute on port 80 at the CSC cluster's MetalLB LoadBalancer IP: `172.18.200.1`. All clusters access Keycloak at this address. From the host, prefer `make test-functional` or `make test-performance`; those targets port-forward Keycloak to `http://127.0.0.1:18080` automatically.
+Keycloak is exposed via Envoy Gateway HTTPRoute on port 80 at the CSC cluster's MetalLB LoadBalancer IP: `172.18.200.1`. On macOS, keep `docker-mac-net-connect` running so the host can reach this address. Linux hosts normally reach the Docker bridge IPs directly.
 
 ```bash
-# Verify Keycloak from inside the Docker network
-curl http://172.18.200.1/realms/event-bus/.well-known/openid-configuration
-
 # Verify Keycloak from the host
-kubectl port-forward -n envoy-gateway-system svc/$(kubectl get svc \
-  --context kind-csc -n envoy-gateway-system \
-  -l gateway.envoyproxy.io/owning-gateway-name=shared-gateway \
-  -o jsonpath='{.items[0].metadata.name}') 18080:80 --context kind-csc
-curl http://127.0.0.1:18080/realms/event-bus/.well-known/openid-configuration
+curl http://172.18.200.1/realms/event-bus/.well-known/openid-configuration
 ```
 
 **Token Endpoint (all clusters):**
@@ -512,7 +512,7 @@ kubectl get servicemonitor -A --context kind-csc
 # Access Prometheus UI and check Status -> Targets
 
 # Verify service labels match ServiceMonitor selector
-kubectl get svc -n event-bus-nats -o yaml --context kind-csc
+kubectl get svc -n event-bus -o yaml --context kind-csc
 ```
 
 ## Cleanup