fix(automl,autorag): resolve MinIO storage access issues (#7221)

* fix(automl,autorag): add support for MinIO storage

Add S3 client support for in-cluster MinIO and other S3-compatible
object stores. Three configurable options control security behavior,
all defaulting to permissive for MinIO compatibility:

- S3_ALLOW_HTTP=true: permit plain HTTP S3 endpoints
- S3_INSECURE_SKIP_VERIFY=true: skip TLS cert verification
- S3_ALLOW_INTERNAL_IPS=true: allow RFC-1918 private IPs

Customers can set any of these to "false" via environment variables
to enforce stricter security policies. Loopback, link-local, and
reserved IP ranges remain always blocked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(automl,autorag): clone DefaultTransport for S3 client, add permissive mode tests

- Clone http.DefaultTransport instead of bare http.Transport to preserve
  default timeouts and connection pooling. Add 30s client timeout.
- Add tests for the permissive (production-default) code paths:
  HTTP accepted when AllowHTTP=true, private IPs accepted when
  AllowInternalIPs=true, loopback/link-local still blocked when
  permissive, HTTP+private IP combination works.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(automl,autorag): address code review feedback for S3 client

- Add CGNAT 100.64.0.0/10 to automl blocked ranges (was already in autorag)
- Use named blockedRange type instead of repeated anonymous struct literals
- Fix autorag to use c.options.InsecureSkipVerify (post-defaults) instead
  of raw opts
- Add CGNAT and IPv6 ULA test cases to permissive mode tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(automl,autorag): use CA bundles for S3 TLS instead of skipping verification

Replace InsecureSkipVerify, AllowHTTP, and AllowInternalIPs env vars
(which were not settable in production) with proper CA bundle-based TLS
verification. The RHOAI operator already mounts cluster and custom CA
bundles into the BFF pod via --bundle-paths, so self-signed MinIO
certificates are validated against these bundles rather than skipped.

- Remove S3_INSECURE_SKIP_VERIFY, S3_ALLOW_HTTP, S3_ALLOW_INTERNAL_IPS
- Add RootCAs to S3ClientOptions, populated from operator-mounted bundles
- HTTPS always required (no plain HTTP)
- Private IPs always allowed (MinIO runs in-cluster)
- Dev-mode fallback: skip TLS verification when no CA bundles provided
- Add BUNDLE_PATHS support to Makefiles for local development

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(automl,autorag): quote BUNDLE_PATHS in Makefile to prevent word-splitting

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(automl,autorag): guard DefaultTransport assertion, add TLS tests, document SSRF allowlist

- Extract cloneDefaultTransport() to safely handle non-standard
  http.DefaultTransport replacements (e.g. in test environments)
- Add TestNewRealS3Client_WithRootCAs and TestNewRealS3Client_DevModeFallback
  to verify both TLS transport configuration paths
- Document CGN (100.64/10, RFC 6598) alongside RFC-1918 and IPv6 ULA
  in validateIPAddress doc comments as permitted ranges

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nicholas Mazzitelli <nickmazz@ca.ibm.com>

Author: 3 people

packages/automl/bff/Makefile

@@ -10,6 +10,7 @@ AUTH_METHOD ?= internal
 AUTH_TOKEN_HEADER ?= Authorization
 AUTH_TOKEN_PREFIX ?= Bearer
 INSECURE_SKIP_VERIFY ?= false
+BUNDLE_PATHS ?=
 #frontend static assets root directory
 STATIC_ASSETS_DIR ?= ./static
 # ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
@@ -63,7 +64,7 @@ endif
 run: fmt vet envtest ## Runs the project.
 	trap 'exit 0' INT; \
 	ENVTEST_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" \
-	go run ./cmd --port=$(PORT) --auth-method=${AUTH_METHOD} --auth-token-header=$(AUTH_TOKEN_HEADER) --auth-token-prefix="$(AUTH_TOKEN_PREFIX)" --static-assets-dir=$(STATIC_ASSETS_DIR) --mock-k8s-client=$(MOCK_K8S_CLIENT) --mock-http-client=$(MOCK_HTTP_CLIENT) --mock-pipeline-server-client=$(MOCK_PIPELINE_SERVER_CLIENT) --pipeline-server-url="$(PIPELINE_SERVER_URL)" --dev-mode=$(DEV_MODE) --dev-mode-client-port=$(DEV_MODE_CLIENT_PORT) --deployment-mode=$(DEPLOYMENT_MODE) --log-level=$(LOG_LEVEL) --allowed-origins=$(ALLOWED_ORIGINS) --insecure-skip-verify=$(INSECURE_SKIP_VERIFY)
+	go run ./cmd --port=$(PORT) --auth-method=${AUTH_METHOD} --auth-token-header=$(AUTH_TOKEN_HEADER) --auth-token-prefix="$(AUTH_TOKEN_PREFIX)" --static-assets-dir=$(STATIC_ASSETS_DIR) --mock-k8s-client=$(MOCK_K8S_CLIENT) --mock-http-client=$(MOCK_HTTP_CLIENT) --mock-pipeline-server-client=$(MOCK_PIPELINE_SERVER_CLIENT) --pipeline-server-url="$(PIPELINE_SERVER_URL)" --dev-mode=$(DEV_MODE) --dev-mode-client-port=$(DEV_MODE_CLIENT_PORT) --deployment-mode=$(DEPLOYMENT_MODE) --log-level=$(LOG_LEVEL) --allowed-origins=$(ALLOWED_ORIGINS) --insecure-skip-verify=$(INSECURE_SKIP_VERIFY) $(if $(BUNDLE_PATHS),--bundle-paths="$(BUNDLE_PATHS)")
 
 ##@ Dependencies
 

packages/automl/bff/internal/api/app.go

@@ -152,7 +152,18 @@ func NewApp(cfg config.EnvConfig, logger *slog.Logger) (*App, error) {
 		s3ClientFactory = s3mocks.NewMockClientFactory()
 	} else {
 		logger.Info("Using real S3 client factory")
-		s3ClientFactory = s3int.NewRealClientFactory(s3int.S3ClientOptions{DevMode: cfg.DevMode})
+		// TLS verification uses the operator-mounted CA bundles (rootCAs) so that
+		// self-signed MinIO certificates are validated properly rather than skipped.
+		// The RHOAI operator passes --bundle-paths with cluster CA, service-ca, and
+		// odh-trusted-ca-bundle paths, which are loaded into rootCAs above.
+		// HTTPS is always required; plain HTTP is rejected to prevent credentials
+		// from being transmitted in cleartext.
+		// RFC-1918 private IPs are allowed (MinIO runs in-cluster); loopback,
+		// link-local, and reserved ranges are always blocked.
+		s3ClientFactory = s3int.NewRealClientFactory(s3int.S3ClientOptions{
+			DevMode: cfg.DevMode,
+			RootCAs: rootCAs,
+		})
 	}
 
 	app := &App{

packages/automl/bff/internal/integrations/s3/client.go

@@ -4,13 +4,15 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"crypto/tls"
 	"encoding/csv"
 	"errors"
 	"fmt"
 	"io"
 	"log/slog"
 	"math"
 	"net"
+	"net/http"
 	"net/url"
 	"os"
 	"strconv"
@@ -97,6 +99,37 @@ func NewRealS3Client(creds *S3Credentials, opts S3ClientOptions) (*RealS3Client,
 		Credentials: credentials.NewStaticCredentialsProvider(creds.AccessKeyID, creds.SecretAccessKey, ""),
 	}
 
+	if c.options.RootCAs != nil {
+		// Clone the default transport to preserve its timeouts and connection pooling,
+		// then supply the custom CA pool so that self-signed or cluster-issued
+		// certificates (e.g. MinIO) are verified against the operator-mounted
+		// CA bundles rather than the system default.
+		transport := cloneDefaultTransport()
+		transport.TLSClientConfig = &tls.Config{
+			RootCAs:    c.options.RootCAs,
+			MinVersion: tls.VersionTLS12,
+		}
+		cfg.HTTPClient = &http.Client{
+			Transport: transport,
+			Timeout:   30 * time.Second,
+		}
+	} else if c.options.DevMode {
+		// In dev mode without CA bundles, skip TLS verification so developers
+		// can test against clusters with self-signed certificates from their
+		// local machine. In production the operator always provides CA bundles
+		// via --bundle-paths, so this path is never reached.
+		slog.Warn("S3 TLS certificate verification disabled (dev mode, no CA bundles provided)")
+		transport := cloneDefaultTransport()
+		transport.TLSClientConfig = &tls.Config{
+			InsecureSkipVerify: true, //nolint:gosec // dev-mode only fallback
+			MinVersion:         tls.VersionTLS12,
+		}
+		cfg.HTTPClient = &http.Client{
+			Transport: transport,
+			Timeout:   30 * time.Second,
+		}
+	}
+
 	c.s3Client = awss3.NewFromConfig(cfg, func(o *awss3.Options) {
 		o.BaseEndpoint = aws.String(validatedEndpoint)
 		o.UsePathStyle = true
@@ -422,8 +455,26 @@ func (c *RealS3Client) GetCSVSchema(ctx context.Context, bucket, key string) (CS
 	}, nil
 }
 
+// cloneDefaultTransport returns a clone of http.DefaultTransport if it is an
+// *http.Transport, or a fresh *http.Transport otherwise. This avoids a panic
+// if DefaultTransport has been replaced with a non-standard implementation
+// (e.g. in test environments).
+func cloneDefaultTransport() *http.Transport {
+	if t, ok := http.DefaultTransport.(*http.Transport); ok {
+		return t.Clone()
+	}
+	return &http.Transport{}
+}
+
 // validateAndNormalizeEndpoint validates the S3 endpoint URL to prevent SSRF attacks.
-// Requires HTTPS and blocks private/reserved IP ranges.
+//
+// HTTPS is required — plain HTTP is rejected because S3 credentials would be
+// transmitted in cleartext. Self-signed or cluster-issued certificates are
+// supported via the RootCAs option (populated from operator-mounted CA bundles).
+//
+// RFC-1918 private IPs are permitted because MinIO commonly runs on the same
+// cluster using service IPs (e.g. 10.x). Loopback, link-local, and reserved
+// IP ranges are always blocked.
 func (c *RealS3Client) validateAndNormalizeEndpoint(endpoint string) (string, error) {
 	if endpoint == "" {
 		return "", fmt.Errorf("endpoint URL cannot be empty")
@@ -472,21 +523,26 @@ func (c *RealS3Client) validateAndNormalizeEndpoint(endpoint string) (string, er
 }
 
 // validateIPAddress checks if an IP address is in a blocked range.
+//
+// RFC-1918 private ranges (10/8, 172.16/12, 192.168/16), Carrier-Grade NAT
+// (100.64/10, RFC 6598), and IPv6 unique local addresses (fc00::/7) are
+// permitted because MinIO and other S3-compatible stores commonly run on the
+// same cluster using service or pod IPs in these ranges.
+// Loopback, link-local, and reserved ranges are always blocked to prevent
+// SSRF targeting the node or cloud metadata services.
 func (c *RealS3Client) validateIPAddress(ip net.IP) error {
-	blockedRanges := []struct {
+	type blockedRange struct {
 		cidr        string
 		description string
-	}{
+	}
+
+	blockedRanges := []blockedRange{
 		{"0.0.0.0/8", "reserved 'this network' range (RFC 1122)"},
-		{"10.0.0.0/8", "RFC-1918 private range (10.0.0.0/8)"},
-		{"172.16.0.0/12", "RFC-1918 private range (172.16.0.0/12)"},
-		{"192.168.0.0/16", "RFC-1918 private range (192.168.0.0/16)"},
 		{"169.254.0.0/16", "link-local range (169.254.0.0/16)"},
 		{"127.0.0.0/8", "loopback range (127.0.0.0/8)"},
 		{"240.0.0.0/4", "reserved for future use (RFC 1112)"},
 		{"::1/128", "IPv6 loopback"},
 		{"fe80::/10", "IPv6 link-local"},
-		{"fc00::/7", "IPv6 unique local addresses"},
 	}
 
 	for _, blocked := range blockedRanges {

packages/automl/bff/internal/integrations/s3/client_factory.go

@@ -1,6 +1,7 @@
 package s3
 
 import (
+	"crypto/x509"
 	"log/slog"
 	"os"
 )
@@ -21,6 +22,15 @@ const (
 type S3ClientOptions struct {
 	DevMode bool // Controls whether ALLOW_UNRESOLVED_S3_ENDPOINTS is honoured
 
+	// RootCAs is the CA certificate pool used to verify S3 endpoint TLS certificates.
+	// In production the RHOAI operator mounts cluster and custom CA bundles into the
+	// pod (e.g. odh-trusted-ca-bundle, service-ca.crt) and passes them via
+	// --bundle-paths. When non-nil this pool is used instead of the system default,
+	// allowing connections to MinIO or other S3-compatible stores that use
+	// self-signed or cluster-issued certificates without skipping verification.
+	// When nil the system CA pool is used (suitable for public S3 endpoints).
+	RootCAs *x509.CertPool
+
 	// Transfer manager tuning for GetObject. Zero values fall back to defaults.
 	Concurrency        int
 	PartSizeBytes      int64

packages/automl/bff/internal/integrations/s3/client_test.go

@@ -1,6 +1,7 @@
 package s3
 
 import (
+	"crypto/x509"
 	"errors"
 	"fmt"
 	"testing"
@@ -28,6 +29,29 @@ func TestNewRealS3Client_WrapsErrEndpointValidation(t *testing.T) {
 // validateAndNormalizeEndpoint — SSRF protection tests
 // ---------------------------------------------------------------------------
 
+func TestNewRealS3Client_WithRootCAs(t *testing.T) {
+	t.Parallel()
+	pool := x509.NewCertPool()
+	_, err := NewRealS3Client(&S3Credentials{
+		AccessKeyID:     "a",
+		SecretAccessKey: "b",
+		Region:          "us-east-1",
+		EndpointURL:     "https://s3.amazonaws.com",
+	}, S3ClientOptions{RootCAs: pool})
+	assert.NoError(t, err)
+}
+
+func TestNewRealS3Client_DevModeFallback(t *testing.T) {
+	t.Parallel()
+	_, err := NewRealS3Client(&S3Credentials{
+		AccessKeyID:     "a",
+		SecretAccessKey: "b",
+		Region:          "us-east-1",
+		EndpointURL:     "https://s3.amazonaws.com",
+	}, S3ClientOptions{DevMode: true})
+	assert.NoError(t, err)
+}
+
 func TestValidateAndNormalizeEndpoint_AcceptsValidHTTPS(t *testing.T) {
 	c := newTestClient()
 	result, err := c.validateAndNormalizeEndpoint("https://s3.us-east-1.amazonaws.com")
@@ -56,25 +80,19 @@ func TestValidateAndNormalizeEndpoint_RejectsEmpty(t *testing.T) {
 	assert.Contains(t, err.Error(), "empty")
 }
 
-func TestValidateAndNormalizeEndpoint_RejectsPrivateIP_10(t *testing.T) {
-	c := newTestClient()
-	_, err := c.validateAndNormalizeEndpoint("https://10.0.0.1:9000")
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "RFC-1918")
-}
-
-func TestValidateAndNormalizeEndpoint_RejectsPrivateIP_172(t *testing.T) {
-	c := newTestClient()
-	_, err := c.validateAndNormalizeEndpoint("https://172.16.0.1:9000")
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "RFC-1918")
-}
-
-func TestValidateAndNormalizeEndpoint_RejectsPrivateIP_192(t *testing.T) {
+func TestValidateAndNormalizeEndpoint_AcceptsPrivateIPs(t *testing.T) {
 	c := newTestClient()
-	_, err := c.validateAndNormalizeEndpoint("https://192.168.1.1:9000")
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "RFC-1918")
+	for _, endpoint := range []string{
+		"https://10.0.0.1:9000",
+		"https://100.64.0.1:9000",
+		"https://172.16.0.1:9000",
+		"https://192.168.1.1:9000",
+		"https://[fd00::1]:9000",
+	} {
+		result, err := c.validateAndNormalizeEndpoint(endpoint)
+		assert.NoError(t, err, "should accept %s", endpoint)
+		assert.Equal(t, endpoint, result)
+	}
 }
 
 func TestValidateAndNormalizeEndpoint_RejectsLoopback(t *testing.T) {
@@ -119,11 +137,11 @@ func TestValidateAndNormalizeEndpoint_RejectsIPv6LinkLocal(t *testing.T) {
 	assert.Contains(t, err.Error(), "IPv6 link-local")
 }
 
-func TestValidateAndNormalizeEndpoint_RejectsIPv6UniqueLocal(t *testing.T) {
+func TestValidateAndNormalizeEndpoint_AcceptsIPv6UniqueLocal(t *testing.T) {
 	c := newTestClient()
-	_, err := c.validateAndNormalizeEndpoint("https://[fc00::1]:9000")
-	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "IPv6 unique local")
+	result, err := c.validateAndNormalizeEndpoint("https://[fc00::1]:9000")
+	assert.NoError(t, err)
+	assert.Equal(t, "https://[fc00::1]:9000", result)
 }
 
 // ---------------------------------------------------------------------------

packages/autorag/bff/Makefile

@@ -12,6 +12,7 @@ AUTH_METHOD ?= user_token
 AUTH_TOKEN_HEADER ?= Authorization
 AUTH_TOKEN_PREFIX ?= Bearer
 INSECURE_SKIP_VERIFY ?= false
+BUNDLE_PATHS ?=
 LLAMA_STACK_URL ?= ""
 #frontend static assets root directory
 STATIC_ASSETS_DIR ?= ./static
@@ -89,7 +90,8 @@ run: fmt vet envtest ## Runs the project.
 		--deployment-mode="$(DEPLOYMENT_MODE)" \
 		--log-level="$(LOG_LEVEL)" \
 		--allowed-origins="$(ALLOWED_ORIGINS)" \
-		--insecure-skip-verify="$(INSECURE_SKIP_VERIFY)"
+		--insecure-skip-verify="$(INSECURE_SKIP_VERIFY)" \
+		$(if $(BUNDLE_PATHS),--bundle-paths="$(BUNDLE_PATHS)")
 
 ##@ Dependencies
 

packages/autorag/bff/internal/api/app.go

@@ -157,8 +157,18 @@ func NewApp(cfg config.EnvConfig, logger *slog.Logger) (*App, error) {
 		s3ClientFactory = s3mocks.NewMockClientFactory()
 	} else {
 		logger.Info("Using real S3 client factory")
-		s3ClientOptions := s3int.S3ClientOptions{DevMode: cfg.DevMode}
-		s3ClientFactory = s3int.NewRealClientFactory(s3ClientOptions)
+		// TLS verification uses the operator-mounted CA bundles (rootCAs) so that
+		// self-signed MinIO certificates are validated properly rather than skipped.
+		// The RHOAI operator passes --bundle-paths with cluster CA, service-ca, and
+		// odh-trusted-ca-bundle paths, which are loaded into rootCAs above.
+		// HTTPS is always required; plain HTTP is rejected to prevent credentials
+		// from being transmitted in cleartext.
+		// RFC-1918 private IPs are allowed (MinIO runs in-cluster); loopback,
+		// link-local, and reserved ranges are always blocked.
+		s3ClientFactory = s3int.NewRealClientFactory(s3int.S3ClientOptions{
+			DevMode: cfg.DevMode,
+			RootCAs: rootCAs,
+		})
 	}
 
 	app := &App{