Trying to move VLLMInferenceService to deployment/ not everything works, but simulator should work

Author: dmytro-zaharnytskyi

.gitignore

@@ -37,5 +37,4 @@ pip-delete-this-directory.txt
 htmlcov/
 apps/frontend/.env.local
 apps/backend/.env
-CLAUDE.md
-temp/
+CLAUDE.md

deployment/README.md

@@ -31,7 +31,9 @@ deployment/
 │   └── kubernetes/          # Standard Kubernetes deployment (experimental)
 ├── samples/                 # Example model deployments
 │   └── models/
-│       ├── simulator/       # CPU-based test model
+│       ├── rbac/            # Shared RBAC for LLMInferenceService
+│       ├── simulator/       # CPU-based test model (llm-d-inference-sim)
+│       ├── facebook-opt-125m-cpu/  # CPU-based OPT-125M model
 │       └── qwen3/           # GPU-based Qwen3 model
 └── scripts/                 # Installation utilities
 ```
@@ -103,9 +105,17 @@ kustomize build deployment/overlays/kubernetes | envsubst | kubectl apply -f -
 
 ### Step 4: Deploy Sample Models (Optional)
 
+> [!NOTE]
+> These models use KServe's `LLMInferenceService` custom resource, which requires ODH/RHOAI with KServe enabled.
+
 #### Simulator Model (CPU)
 ```bash
-kustomize build deployment/samples/models/simulator | kubectl apply -f -
+kubectl apply -k deployment/samples/models/simulator/
+```
+
+#### Facebook OPT-125M Model (CPU)
+```bash
+kubectl apply -k deployment/samples/models/facebook-opt-125m-cpu/
 ```
 
 #### Qwen3 Model (GPU Required)
@@ -114,7 +124,16 @@ kustomize build deployment/samples/models/simulator | kubectl apply -f -
 > This model requires GPU nodes with `nvidia.com/gpu` resources available in your cluster.
 
 ```bash
-kustomize build deployment/samples/models/qwen3 | kubectl apply -f -
+kubectl apply -k deployment/samples/models/qwen3/
+```
+
+#### Verify Model Deployment
+```bash
+# Check LLMInferenceService status
+kubectl get llminferenceservices -n llm
+
+# Check pods
+kubectl get pods -n llm
 ```
 
 ## Platform-Specific Configuration
@@ -327,8 +346,8 @@ Check that policies are enforced:
 kubectl get authpolicy -A
 kubectl get tokenratelimitpolicy -A
 
-# Check InferenceServices are ready
-kubectl get inferenceservice -n llm
+# Check LLMInferenceServices are ready
+kubectl get llminferenceservices -n llm
 ```
 
 ## Services Exposed

deployment/components/odh/README.md

@@ -0,0 +1,174 @@
+# OpenDataHub Installation and Configuration
+
+This directory contains the configuration files for installing OpenDataHub (ODH) with KServe support for the MaaS platform.
+
+## Key Features
+
+- **RawDeployment Mode**: Direct pod deployments without Knative/Serverless overhead
+- **NVIDIA NIM Support**: GPU-accelerated inference with NVIDIA Inference Microservices
+- **Headless Services**: Direct pod-to-pod communication for low latency
+- **OpenShift Integration**: Uses OpenShift's default ingress and certificates
+
+## Installation Methods
+
+### Method 1: Automated Installation (Recommended)
+
+Use the provided installation script which handles all steps in the correct order:
+
+```bash
+# From the project root
+./deployment/scripts/installers/install-odh.sh
+```
+
+### Method 2: Manual Installation
+
+1. **Install ODH Operator** from OperatorHub:
+   ```bash
+   # Via OpenShift Console:
+   # 1. Navigate to Operators → OperatorHub
+   # 2. Search for "OpenDataHub"
+   # 3. Install with default settings
+   
+   # Or via CLI:
+   oc create -f - <<EOF
+   apiVersion: operators.coreos.com/v1alpha1
+   kind: Subscription
+   metadata:
+     name: opendatahub-operator
+     namespace: openshift-operators
+   spec:
+     channel: fast
+     name: opendatahub-operator
+     source: community-operators
+     sourceNamespace: openshift-marketplace
+   EOF
+   ```
+
+2. **Create namespace**:
+   ```bash
+   kubectl create namespace opendatahub
+   ```
+
+3. **Wait for CRDs to be registered**:
+   ```bash
+   # Wait for the operator to create the CRDs
+   kubectl wait --for condition=established --timeout=300s \
+     crd/dscinitializations.dscinitialization.opendatahub.io \
+     crd/datascienceclusters.datasciencecluster.opendatahub.io
+   ```
+
+4. **Apply the configuration**:
+   ```bash
+   # IMPORTANT: DSCInitialization MUST be created before DataScienceCluster
+   kubectl apply -f dscinitialisation.yaml
+   
+   # Wait for DSCInitialization to be ready
+   kubectl wait --for=jsonpath='{.status.phase}'=Ready \
+     dscinitializations.dscinitialization.opendatahub.io/default-dsci \
+     -n opendatahub --timeout=300s
+   
+   # Now create the DataScienceCluster
+   kubectl apply -f datasciencecluster.yaml
+   ```
+
+   Or use kustomize:
+   ```bash
+   kubectl apply -k deployment/components/odh/
+   ```
+
+## Troubleshooting
+
+### Error: "dscinitializations.dscinitialization.opendatahub.io not found"
+
+This is the most common error when creating a DataScienceCluster. It occurs when:
+1. The ODH operator is not installed
+2. The DSCInitialization resource hasn't been created yet
+3. The CRDs haven't been registered yet
+
+**Solution**: Run the fix script:
+```bash
+./deployment/scripts/installers/fix-odh-dsci.sh
+```
+
+This script will:
+- Check if the ODH operator is installed
+- Wait for CRDs to be registered
+- Create the DSCInitialization if missing
+- Provide next steps for creating the DataScienceCluster
+
+### Manual Troubleshooting Steps
+
+1. **Check operator status**:
+   ```bash
+   kubectl get csv -n openshift-operators | grep opendatahub
+   kubectl logs -n openshift-operators deployment/opendatahub-operator-controller-manager
+   ```
+
+2. **Check CRDs**:
+   ```bash
+   kubectl get crd | grep opendatahub
+   ```
+
+3. **Check existing resources**:
+   ```bash
+   kubectl get dscinitializations -A
+   kubectl get datasciencecluster -A
+   ```
+
+4. **Check pod status**:
+   ```bash
+   kubectl get pods -n opendatahub
+   kubectl get pods -n kserve
+   ```
+
+## Configuration Details
+
+### DSCInitialization
+- Configures the foundational settings for ODH
+- Sets up Service Mesh integration
+- Configures monitoring and trusted CA bundles
+- **MUST be created before DataScienceCluster**
+
+### DataScienceCluster
+- Deploys the actual ODH components
+- Configured for KServe with:
+  - **RawDeployment mode**: No Knative/Serverless overhead
+  - **NIM support**: For NVIDIA GPU inference
+  - **Headless services**: For direct pod communication
+  - **OpenShift ingress**: Native OpenShift routing
+
+### Components Status
+- ✅ **Enabled**: Dashboard, Workbenches, KServe (with NIM)
+- ❌ **Disabled**: ModelMesh, Pipelines, Ray, Kueue, Model Registry, TrustyAI, Training Operator
+
+## Verification
+
+After installation, verify the deployment:
+
+```bash
+# Check DSCInitialization status
+kubectl get dscinitializations -n opendatahub
+
+# Check DataScienceCluster status
+kubectl get datasciencecluster -n opendatahub
+
+# Check KServe components
+kubectl get pods -n kserve
+
+# Check if InferenceService CRD is available
+kubectl get crd inferenceservices.serving.kserve.io
+```
+
+## Integration with MaaS
+
+Once ODH is installed with KServe, you can:
+1. Deploy models using KServe InferenceService
+2. Use the MaaS API for model management
+3. Apply rate limiting and authentication policies
+4. Monitor model performance through the ODH dashboard
+
+## Additional Resources
+
+- [OpenDataHub Documentation](https://opendatahub.io/docs/)
+- [KServe Documentation](https://kserve.github.io/website/)
+- [NVIDIA NIM Documentation](https://docs.nvidia.com/nim/) 

deployment/components/odh/datasciencecluster.yaml

@@ -0,0 +1,55 @@
+apiVersion: datasciencecluster.opendatahub.io/v1
+kind: DataScienceCluster
+metadata:
+  name: default-dsc
+  namespace: opendatahub
+spec:
+  components:
+    # Core dashboard component for managing notebooks and models
+    dashboard:
+      managementState: Managed
+    
+    # Notebook controller for JupyterLab workbenches
+    workbenches:
+      managementState: Managed
+    
+    # Model serving with KServe configured for RawDeployment mode
+    # This configuration:
+    # - Uses RawDeployment mode instead of Knative/Serverless
+    # - Enables NVIDIA NIM support for GPU-accelerated inference
+    # - Uses headless services for direct pod communication
+    # - Integrates with OpenShift's default ingress
+    kserve:
+      managementState: Managed
+      defaultDeploymentMode: RawDeployment  # Direct pod deployments without Knative
+      nim:
+        managementState: Managed  # Enable NVIDIA NIM (NVIDIA Inference Microservices) support
+      rawDeploymentServiceConfig: Headless  # Use headless services for raw deployments
+      serving:
+        ingressGateway:
+          certificate:
+            type: OpenshiftDefaultIngress  # Use OpenShift's default ingress certificate
+        managementState: Removed  # Disable Knative serving (incompatible with RawDeployment)
+        name: knative-serving
+    
+    # Other components disabled for MaaS-focused deployment
+    modelmeshserving:
+      managementState: Removed  # Use KServe instead
+    
+    datasciencepipelines:
+      managementState: Removed  # Not needed for MaaS
+    
+    ray:
+      managementState: Removed  # Not needed for MaaS
+    
+    kueue:
+      managementState: Removed  # Not needed for MaaS
+    
+    modelregistry:
+      managementState: Removed  # Not needed for MaaS
+    
+    trustyai:
+      managementState: Removed  # Not needed for MaaS
+    
+    trainingoperator:
+      managementState: Removed  # Not needed for MaaS 

deployment/components/odh/dscinitialisation.yaml

@@ -0,0 +1,27 @@
+apiVersion: dscinitialization.opendatahub.io/v1
+kind: DSCInitialization
+metadata:
+  name: default-dsci
+  namespace: opendatahub
+spec:
+  # Namespace where ODH applications will be deployed
+  applicationsNamespace: opendatahub
+  
+  # Monitoring configuration for ODH components
+  monitoring:
+    managementState: Managed
+    namespace: opendatahub
+  
+  # Service Mesh configuration for secure communication
+  serviceMesh:
+    managementState: Managed
+    auth:
+      audiences:
+        - "https://kubernetes.default.svc"
+    controlPlane:
+      name: data-science-smcp
+      namespace: istio-system
+  
+  # Trusted CA bundle for TLS certificates
+  trustedCABundle:
+    managementState: Managed 

deployment/components/odh/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+metadata:
+  name: odh-installation
+
+namespace: opendatahub
+
+resources:
+  # DSCInitialization must be created before DataScienceCluster
+  - dscinitialisation.yaml
+  - datasciencecluster.yaml