Skip to content

Add init containers and sidecars support for predictor topology #15

Description

@tartkri-bot

Problem

Users cannot add custom init containers or sidecars to InferenceService pods created by the predictor topology. This limits several advanced deployment patterns:

  1. Custom model download: A team has a proprietary model distribution system (e.g., their own CDN, Artifactory, P2P download like Dragonfly) and needs a custom init container to fetch the model before KServe's serving runtime starts.
  2. Model validation: Run a checksum / signature validation init container before serving.
  3. Certificate injection: A sidecar to inject or rotate mTLS certs.
  4. Service mesh proxy: While Istio sidecar injection is usually automatic, some teams need to explicitly add a proxy sidecar or customize the sidecar container spec.
  5. Observability agent: A sidecar that collects custom metrics/logs (e.g., Fluent Bit, Vector, custom Prometheus exporter).

KServe supports init containers and sidecars in the InferenceService predictor spec via containers.

Suggested approach

Expose initContainers and sidecars on ModelServerCustomSpec (predictor) and the equivalent on the LLM topology parameters. Validate that sidecars don't conflict with the main serving container.

Acceptance criteria

  • Predictor topology supports initContainers and containers (sidecars) fields.
  • Init containers run before the storage initializer / main serving container.
  • Duplicate port or volume mount conflicts between main container and sidecars are caught during validation.
  • Chainsaw tests verify init container execution order.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions