RHOAIENG-50060: Create accelerator-specific LLMInferenceServiceConfig templates with distinguishing labels#1130
Open
KillianGolds wants to merge 4 commits intoopendatahub-io:masterfrom
Conversation
… and Spyre - Add conditional ibm-aiu-setup.sh sourcing to base template (wrapped in bash command with forwarding; no-op on non-Spyre nodes) - Add 5 accelerator LLMInferenceServiceConfigs under config/overlays/odh/accelerators/: - NVIDIA CUDA (image only) - AMD ROCm (image only) - IBM Spyre s390x, x86, ppc64le (image + arch-specific env) - All accelerator configs carry label opendatahub.io/config-type: accelerator - Inject accelerator images from params.env via kustomize replacements - No args overrides in accelerator configs -- preserves user arg customization per RHOAIENG-49005 / kserve#5049 Signed-off-by: Killian Golds <kgolds@redhat.com>
|
@KillianGolds: This pull request references RHOAIENG-50060 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "4.22." or "openshift-4.22.", but it targets "rhoai-3.4" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: KillianGolds The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
9 tasks
📝 WalkthroughWalkthroughAdds accelerator-specific KServe LLMInferenceServiceConfig templates and kustomize wiring (NVIDIA, AMD ROCm, IBM Spyre variants). Replaces direct vllm container start with a /bin/bash -c wrapper that conditionally sources /etc/profile.d/ibm-aiu-setup.sh before exec'ing vllm serve /mnt/models, preserving argument passthrough. Changes
Sequence Diagram(s)sequenceDiagram
actor Kubelet
participant ContainerShell as "/bin/bash -c (wrapper)"
participant Profile as "/etc/profile.d/ibm-aiu-setup.sh (optional)"
participant vLLM as "vllm serve /mnt/models"
Kubelet->>ContainerShell: start container (entrypoint -> bash -c)
ContainerShell->>ContainerShell: set -e
ContainerShell->>Profile: test -f /etc/profile.d/ibm-aiu-setup.sh
alt profile exists
ContainerShell->>Profile: source /etc/profile.d/ibm-aiu-setup.sh
end
ContainerShell->>vLLM: exec vllm serve /mnt/models "$@"
vLLM-->>Kubelet: serve requests (listening on port)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@KillianGolds: This pull request references RHOAIENG-50060 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "4.22." or "openshift-4.22.", but it targets "rhoai-3.4" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@config/llmisvcconfig/config-llm-template.yaml`:
- Around line 17-19: The silent `source` call may fail and still allow `exec
vllm serve /mnt/models "$@"` to run; update the shell snippet so `vllm` only
executes when sourcing succeeds (for example, enable strict failure with `set
-e` at the top of the snippet or chain the commands so the `source
/etc/profile.d/ibm-aiu-setup.sh` must return zero before `exec vllm` runs, or
add an explicit `|| { echo "IBM AIU setup failed"; exit 1; }` after the source).
Ensure the change targets the lines containing `/etc/profile.d/ibm-aiu-setup.sh`
and `exec vllm serve /mnt/models "$@"` so a broken IBM environment aborts
startup instead of launching vLLM.
In `@config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml`:
- Around line 1-15: The ppc64le LLMInferenceServiceConfig template (kind
LLMInferenceServiceConfig, spec.template.containers[name: main]) currently only
sets HF_HOME while s390x/x86 variants set Spyre-specific env vars (FLEX_DEVICE,
TOKENIZERS_PARALLELISM, DTLOG_LEVEL, TORCH_SENDNN_LOG, VLLM_SPYRE_USE_CB,
VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS, TORCH_SENDNN_CACHE_ENABLE, etc.);
update this template to either include the same Spyre-specific environment
variables with appropriate ppc64le values or add a clear inline comment/TODO in
the template explaining that the reduced env set is intentional or awaiting IBM
Power team input so the omission is not merged silently (modify
spec.template.containers -> name: main to add the comment/TODO or the env
entries).
In `@config/overlays/odh/kustomization.yaml`:
- Around line 81-100: The kustomization currently replaces image for three
LLMInferenceServiceConfig resources from a single ConfigMap key (ConfigMap name
kserve-parameters, fieldpath data.kserve-llm-d-ibm-spyre), but the three targets
(LLMInferenceServiceConfig names kserve-config-llm-template-ibm-spyre-s390x,
kserve-config-llm-template-ibm-spyre-x86,
kserve-config-llm-template-ibm-spyre-ppc64le) require distinct
architecture-specific images; update params.env to define three separate keys
(kserve-llm-d-ibm-spyre-s390x, kserve-llm-d-ibm-spyre-x86,
kserve-llm-d-ibm-spyre-ppc64le) and modify this kustomization replacement block
so each target uses the corresponding ConfigMap key instead of the single
data.kserve-llm-d-ibm-spyre entry to ensure the correct arch-specific image is
injected for each LLMInferenceServiceConfig.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to data retention organization setting
📒 Files selected for processing (9)
config/llmisvcconfig/config-llm-template.yamlconfig/overlays/odh/accelerators/amd-rocm-config-llm-template.yamlconfig/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yamlconfig/overlays/odh/accelerators/ibm-spyre-s390x-config-llm-template.yamlconfig/overlays/odh/accelerators/ibm-spyre-x86-config-llm-template.yamlconfig/overlays/odh/accelerators/kustomization.yamlconfig/overlays/odh/accelerators/nvidia-cuda-config-llm-template.yamlconfig/overlays/odh/kustomization.yamlconfig/overlays/odh/params.env
Comment on lines
+1
to
+15
| apiVersion: serving.kserve.io/v1alpha2 | ||
| kind: LLMInferenceServiceConfig | ||
| metadata: | ||
| name: kserve-config-llm-template-ibm-spyre-ppc64le | ||
| annotations: | ||
| openshift.io/display-name: vLLM IBM Spyre ppc64le LLMInferenceServiceConfig | ||
| description: vLLM IBM Spyre ppc64le LLMInferenceServiceConfig for LLMInferenceService. | ||
| spec: | ||
| template: | ||
| containers: | ||
| - name: main | ||
| image: placeholder | ||
| env: | ||
| - name: HF_HOME | ||
| value: /tmp/hf_home |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find all IBM Spyre LLM template config files
fd ".*spyre.*config.*template.*yaml" --type fRepository: opendatahub-io/kserve
Length of output: 286
🏁 Script executed:
#!/bin/bash
# Read each variant file to compare env var configurations
echo "=== ppc64le ===" && cat config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml
echo -e "\n=== s390x ===" && cat config/overlays/odh/accelerators/ibm-spyre-s390x-config-llm-template.yaml
echo -e "\n=== x86 ===" && cat config/overlays/odh/accelerators/ibm-spyre-x86-config-llm-template.yamlRepository: opendatahub-io/kserve
Length of output: 2543
🏁 Script executed:
#!/bin/bash
# Check for any TODO/FIXME/comments about ppc64le config or IBM Power
rg -i "ppc64le|ibm.*power" --type yaml -B 2 -A 2 config/overlays/odh/accelerators/
rg -i "TODO|FIXME" config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yamlRepository: opendatahub-io/kserve
Length of output: 1259
🏁 Script executed:
#!/bin/bash
# Check for README or documentation about accelerators configs
fd "README|ACCELERATOR" --type f config/overlays/odh/accelerators/
# Also search for any documentation directory
fd "doc|docs" --type d config/overlays/odh/accelerators/
# Check if there are any comments or docs at top-level about accelerators
rg -i "accelerator|spyre" README.md --type markdown -A 5 -B 5 2>/dev/null || echo "No match in README"Repository: opendatahub-io/kserve
Length of output: 83
ppc64le template lacks Spyre-specific env vars defined in s390x and x86 variants—confirm if intentional or document as incomplete.
The s390x variant defines 8 environment variables (FLEX_DEVICE, TOKENIZERS_PARALLELISM, DTLOG_LEVEL, TORCH_SENDNN_LOG, VLLM_SPYRE_USE_CB, VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS, TORCH_SENDNN_CACHE_ENABLE) and x86 defines 10; ppc64le only has HF_HOME. If this sparse configuration is intentional for ppc64le hardware, add a comment explaining why these knobs are not needed. If it is a placeholder pending IBM Power team input, add a # TODO: annotation to prevent the gap from being silently merged as final.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml`
around lines 1 - 15, The ppc64le LLMInferenceServiceConfig template (kind
LLMInferenceServiceConfig, spec.template.containers[name: main]) currently only
sets HF_HOME while s390x/x86 variants set Spyre-specific env vars (FLEX_DEVICE,
TOKENIZERS_PARALLELISM, DTLOG_LEVEL, TORCH_SENDNN_LOG, VLLM_SPYRE_USE_CB,
VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS, TORCH_SENDNN_CACHE_ENABLE, etc.);
update this template to either include the same Spyre-specific environment
variables with appropriate ppc64le values or add a clear inline comment/TODO in
the template explaining that the reduced env set is intentional or awaiting IBM
Power team input so the omission is not merged silently (modify
spec.template.containers -> name: main to add the comment/TODO or the env
entries).
Comment on lines
+81
to
+100
| - source: | ||
| kind: ConfigMap | ||
| name: kserve-parameters | ||
| fieldpath: data.kserve-llm-d-ibm-spyre | ||
| targets: | ||
| - select: | ||
| kind: LLMInferenceServiceConfig | ||
| name: kserve-config-llm-template-ibm-spyre-s390x | ||
| fieldPaths: | ||
| - spec.template.containers.[name=main].image | ||
| - select: | ||
| kind: LLMInferenceServiceConfig | ||
| name: kserve-config-llm-template-ibm-spyre-x86 | ||
| fieldPaths: | ||
| - spec.template.containers.[name=main].image | ||
| - select: | ||
| kind: LLMInferenceServiceConfig | ||
| name: kserve-config-llm-template-ibm-spyre-ppc64le | ||
| fieldPaths: | ||
| - spec.template.containers.[name=main].image |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -g 'params.env' --exec grep -n 'kserve-llm-d-ibm-spyre\|kserve-llm-d-nvidia-cuda\|kserve-llm-d-amd-rocm' {}Repository: opendatahub-io/kserve
Length of output: 483
🏁 Script executed:
# First, verify the file structure and see the context around lines 81-100
fd -g 'kustomization.yaml' | grep odh | head -5Repository: opendatahub-io/kserve
Length of output: 199
🏁 Script executed:
# Search for any README or documentation about image architecture support
fd -g 'README*' -o -g '*.md' | xargs grep -l 'spyre\|s390x\|ppc64le' 2>/dev/null | head -10Repository: opendatahub-io/kserve
Length of output: 181
🏁 Script executed:
# Look for any other references to architecture-specific images or configs
rg 's390x|ppc64le' -t yaml -t md | head -20Repository: opendatahub-io/kserve
Length of output: 1338
🏁 Script executed:
cat -n config/overlays/odh/kustomization.yaml | sed -n '75,105p'Repository: opendatahub-io/kserve
Length of output: 1137
🏁 Script executed:
# Check if there's an x86 variant
fd -g '*ibm-spyre*' -o -g '*x86*' config/overlays/odh/accelerators/Repository: opendatahub-io/kserve
Length of output: 181
🏁 Script executed:
# Also check what's in the accelerators directory
ls -la config/overlays/odh/accelerators/ | grep ibmRepository: opendatahub-io/kserve
Length of output: 334
🏁 Script executed:
cat config/overlays/odh/accelerators/ibm-spyre-s390x-config-llm-template.yamlRepository: opendatahub-io/kserve
Length of output: 974
🏁 Script executed:
cat config/overlays/odh/accelerators/ibm-spyre-x86-config-llm-template.yamlRepository: opendatahub-io/kserve
Length of output: 1118
Verify that a single Spyre image is correct for all three architectures.
All three Spyre variants (s390x, x86, ppc64le) pull their container image from the same data.kserve-llm-d-ibm-spyre key, but their LLMInferenceServiceConfig resources define completely different environment variables per architecture. For example, s390x sets FLEX_DEVICE=VF, VLLM_SPYRE_USE_CB=1, and VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS=1, while x86 sets FLEX_COMPUTE=SENTIENT, FLEX_DEVICE=PF, and VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS=0. These architectures (s390x is IBM mainframe, x86 is Intel/AMD) require different CPU instruction sets and cannot share a single binary image. The params.env file defines only one Spyre image key with a specific digest (vllm-spyre-rhel9@sha256:...), which is a single-architecture image, not a multi-arch manifest list. This will cause deployment failures on s390x and ppc64le systems. Define separate image keys for each architecture (kserve-llm-d-ibm-spyre-s390x, kserve-llm-d-ibm-spyre-x86, kserve-llm-d-ibm-spyre-ppc64le) in params.env and update the kustomization.yaml replacement sources accordingly.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@config/overlays/odh/kustomization.yaml` around lines 81 - 100, The
kustomization currently replaces image for three LLMInferenceServiceConfig
resources from a single ConfigMap key (ConfigMap name kserve-parameters,
fieldpath data.kserve-llm-d-ibm-spyre), but the three targets
(LLMInferenceServiceConfig names kserve-config-llm-template-ibm-spyre-s390x,
kserve-config-llm-template-ibm-spyre-x86,
kserve-config-llm-template-ibm-spyre-ppc64le) require distinct
architecture-specific images; update params.env to define three separate keys
(kserve-llm-d-ibm-spyre-s390x, kserve-llm-d-ibm-spyre-x86,
kserve-llm-d-ibm-spyre-ppc64le) and modify this kustomization replacement block
so each target uses the corresponding ConfigMap key instead of the single
data.kserve-llm-d-ibm-spyre entry to ensure the correct arch-specific image is
injected for each LLMInferenceServiceConfig.
…ange make precommit regenerates quick-install scripts which embed kustomize build output -- updated to reflect the new bash command wrapper in config/llmisvcconfig/config-llm-template.yaml. Signed-off-by: Killian Golds <kgolds@redhat.com>
|
@KillianGolds: This pull request references RHOAIENG-50060 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "4.22." or "openshift-4.22.", but it targets "rhoai-3.4" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@hack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.sh`:
- Around line 120263-120265: The current two-line pattern "[ -f
/etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh"
followed by "exec vllm serve /mnt/models \"$@\"" can silently continue if
sourcing fails; change it to an explicit check that exits on source failure
(e.g., if the file exists then attempt to source it and if sourcing returns
non-zero, log an error and exit with non-zero) before running exec vllm serve;
apply the same guarded-sourcing pattern to the other quick-install scripts
(kserve-standard-mode-full-install-with-manifests.sh,
llmisvc-full-install-with-manifests.sh) and the base template
(config/llmisvcconfig/config-llm-template.yaml) so that failing to source
/etc/profile.d/ibm-aiu-setup.sh prevents starting vllm in a misconfigured
environment.
In
`@hack/setup/quick-install/kserve-standard-mode-full-install-with-manifests.sh`:
- Around line 120532-120533: Ensure the script fails fast if sourcing
/etc/profile.d/ibm-aiu-setup.sh fails: after attempting to source that file,
check the return status and if non-zero print an error and exit with non-zero
status instead of proceeding to the exec vllm serve /mnt/models "$@". In
practice, replace the bare "[ -f /etc/profile.d/ibm-aiu-setup.sh ] && source
/etc/profile.d/ibm-aiu-setup.sh" with an explicit conditional that sources the
file and on failure logs a clear message and exits (or use "source
/etc/profile.d/ibm-aiu-setup.sh || { echo 'failed to source ...' >&2; exit 1;
}") so exec vllm only runs when the IBM AIU environment was successfully loaded.
In `@hack/setup/quick-install/llmisvc-full-install-with-manifests.sh`:
- Around line 85774-85780: The current startup command uses "[ -f
/etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh"
which will silently continue if the file exists but sourcing fails; change this
to an explicit if-check that only skips when the file is absent and otherwise
attempts to source and, on any sourcing error, prints a clear error to stderr
and exits non‑zero before reaching "exec vllm serve /mnt/models \"$@\""; ensure
the new logic preserves the skip-on-absent behavior but fail-fast on source
errors so Spyre nodes don't start vllm in an unconfigured environment.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to data retention organization setting
📒 Files selected for processing (3)
hack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.shhack/setup/quick-install/kserve-standard-mode-full-install-with-manifests.shhack/setup/quick-install/llmisvc-full-install-with-manifests.sh
Comment on lines
120263
to
120265
| - | | ||
| [ -f /etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh | ||
| exec vllm serve /mnt/models "$@" |
There was a problem hiding this comment.
Silent source failure leaves Spyre nodes in a misconfigured state.
The two script lines are independent statements — if ibm-aiu-setup.sh exists but errors during sourcing, the && suppresses the failure and exec vllm serve still runs on the next line without the IBM AIU environment configured. On Spyre nodes, this would silently start vllm in a broken state. Prefer an explicit if/|| guard:
🛡️ Proposed fix
- - |
- [ -f /etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh
- exec vllm serve /mnt/models "$@"
+ - |
+ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then
+ source /etc/profile.d/ibm-aiu-setup.sh || exit 1
+ fi
+ exec vllm serve /mnt/models "$@"This fix ensures that if the AIU setup script is present but fails, the container exits immediately instead of running vllm in an undefined environment. The same pattern should be applied to the other quick-install scripts (kserve-standard-mode-full-install-with-manifests.sh, llmisvc-full-install-with-manifests.sh) and the base template config/llmisvcconfig/config-llm-template.yaml from which these are generated.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - | | |
| [ -f /etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh | |
| exec vllm serve /mnt/models "$@" | |
| - | | |
| if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then | |
| source /etc/profile.d/ibm-aiu-setup.sh || exit 1 | |
| fi | |
| exec vllm serve /mnt/models "$@" |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@hack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.sh`
around lines 120263 - 120265, The current two-line pattern "[ -f
/etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh"
followed by "exec vllm serve /mnt/models \"$@\"" can silently continue if
sourcing fails; change it to an explicit check that exits on source failure
(e.g., if the file exists then attempt to source it and if sourcing returns
non-zero, log an error and exit with non-zero) before running exec vllm serve;
apply the same guarded-sourcing pattern to the other quick-install scripts
(kserve-standard-mode-full-install-with-manifests.sh,
llmisvc-full-install-with-manifests.sh) and the base template
(config/llmisvcconfig/config-llm-template.yaml) so that failing to source
/etc/profile.d/ibm-aiu-setup.sh prevents starting vllm in a misconfigured
environment.
hack/setup/quick-install/kserve-standard-mode-full-install-with-manifests.sh
Outdated
Show resolved
Hide resolved
Comment on lines
85774
to
85780
| command: | ||
| - vllm | ||
| - serve | ||
| - /mnt/models | ||
| - /bin/bash | ||
| - -c | ||
| - | | ||
| [ -f /etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh | ||
| exec vllm serve /mnt/models "$@" | ||
| - -- |
There was a problem hiding this comment.
Source failure is silently ignored — consider fail-fast behavior for IBM AIU environments
If /etc/profile.d/ibm-aiu-setup.sh exists but fails to source (e.g., due to a syntax error or a failed export inside the script), the && short-circuit ends and execution falls through to exec vllm serve without the required IBM AIU environment — silently. On non-Spyre nodes this is fine (file absent → skip), but on Spyre nodes a broken setup script would cause vllm to launch in an unconfigured environment, likely producing hard-to-diagnose runtime failures rather than an immediate startup error.
If the intent is fail-open (best-effort setup, always start vllm), the current logic is correct. If Spyre nodes require the setup to succeed, consider:
🛡️ Proposed fail-fast variant
- [ -f /etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh
+ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then
+ source /etc/profile.d/ibm-aiu-setup.sh || { echo "ERROR: failed to source ibm-aiu-setup.sh" >&2; exit 1; }
+ fi🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@hack/setup/quick-install/llmisvc-full-install-with-manifests.sh` around lines
85774 - 85780, The current startup command uses "[ -f
/etc/profile.d/ibm-aiu-setup.sh ] && source /etc/profile.d/ibm-aiu-setup.sh"
which will silently continue if the file exists but sourcing fails; change this
to an explicit if-check that only skips when the file is absent and otherwise
attempts to source and, on any sourcing error, prints a clear error to stderr
and exits non‑zero before reaching "exec vllm serve /mnt/models \"$@\""; ensure
the new logic preserves the skip-on-absent behavior but fail-fast on source
errors so Spyre nodes don't start vllm in an unconfigured environment.
- Base template: use set -e + if block for ibm-aiu-setup.sh sourcing so a broken IBM environment aborts startup instead of silently launching vLLM with a failed setup - ppc64le Spyre config: add missing arch-specific env vars (FLEX_COMPUTE, FLEX_DEVICE, warmup params, etc.) aligned with odh-model-controller reference template - Regenerate install scripts Signed-off-by: Killian Golds <kgolds@redhat.com>
|
@KillianGolds: This pull request references RHOAIENG-50060 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "4.22." or "openshift-4.22.", but it targets "rhoai-3.4" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml (1)
13-33: Add inline comment explaining architecture-specific env var differences.The ppc64le config (like x86) intentionally omits
VLLM_SPYRE_USE_CBandTORCH_SENDNN_CACHE_ENABLE, which are present in s390x. This reflects different acceleration strategies: ppc64le/x86 use warmup batching (VLLM_SPYRE_WARMUP_*) while s390x uses continuous batching (VLLM_SPYRE_USE_CB). Adding a brief inline comment (e.g.,# ppc64le uses warmup batching instead of continuous batching) would clarify this architectural choice and prevent future confusion about the gap.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml` around lines 13 - 33, Add a one-line inline YAML comment near the env block (e.g., above the VLLM_SPYRE_WARMUP_* entries) explaining that ppc64le/x86 intentionally omit VLLM_SPYRE_USE_CB and TORCH_SENDNN_CACHE_ENABLE (present in s390x) because these architectures use warmup batching (VLLM_SPYRE_WARMUP_BATCH_SIZES, VLLM_SPYRE_WARMUP_PROMPT_LENS, VLLM_SPYRE_WARMUP_NEW_TOKENS) rather than continuous batching; reference the variable names VLLM_SPYRE_USE_CB, TORCH_SENDNN_CACHE_ENABLE and the VLLM_SPYRE_WARMUP_* series in the comment for clarity.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@config/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yaml`:
- Around line 13-33: Add a one-line inline YAML comment near the env block
(e.g., above the VLLM_SPYRE_WARMUP_* entries) explaining that ppc64le/x86
intentionally omit VLLM_SPYRE_USE_CB and TORCH_SENDNN_CACHE_ENABLE (present in
s390x) because these architectures use warmup batching
(VLLM_SPYRE_WARMUP_BATCH_SIZES, VLLM_SPYRE_WARMUP_PROMPT_LENS,
VLLM_SPYRE_WARMUP_NEW_TOKENS) rather than continuous batching; reference the
variable names VLLM_SPYRE_USE_CB, TORCH_SENDNN_CACHE_ENABLE and the
VLLM_SPYRE_WARMUP_* series in the comment for clarity.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to data retention organization setting
📒 Files selected for processing (5)
config/llmisvcconfig/config-llm-template.yamlconfig/overlays/odh/accelerators/ibm-spyre-ppc64le-config-llm-template.yamlhack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.shhack/setup/quick-install/kserve-standard-mode-full-install-with-manifests.shhack/setup/quick-install/llmisvc-full-install-with-manifests.sh
Regenerated via 📦 Extracting versions from go.mod... ✅ Updated kserve-deps.env ISTIO_VERSION=1.27.1 KEDA_VERSION=2.17.3 GATEWAY_API_VERSION=v1.3.1-0.20251106052652-079e4774d76b GIE_VERSION=v1.2.0 LWS_VERSION=v0.7.0 OPENTELEMETRY_OPERATOR_VERSION=0.74.3 Generated /Users/killiangolds/Desktop/Dev-Environment/kserve-fork/kserve-images.sh go vet ./pkg/... ./cmd/... && cd qpext && go vet ./... go mod tidy cd qpext && go mod tidy /Users/killiangolds/Desktop/Dev-Environment/kserve-fork/bin/.venv/bin/black --config python/pyproject.toml ./python ./docs /Users/killiangolds/Desktop/Dev-Environment/kserve-fork/bin/.venv/bin/flake8 --config=.flake8 . hack/update-codegen.sh Codegen version v0.34.2 Generating defaulter code for 3 targets Generating conversion code for 2 targets Generating client code for 3 targets Generating lister code for 3 targets Generating informer code for 3 targets hack/update-openapigen.sh You are not in the target directory: /Users/killiangolds/go/src/github.com/kserve/kserve Creating a symbolic link for the target directory ... Changing to the target directory: /Users/killiangolds/go/src/github.com/kserve/kserve Returning to the original directory: /Users/killiangolds/Desktop/Dev-Environment/kserve-fork hack/python-sdk/client-gen.sh Downloading the swagger-codegen JAR package ... Generating Python SDK for KServe ... [main] INFO o.o.codegen.DefaultGenerator - Generating with dryRun=false [main] INFO o.o.codegen.DefaultGenerator - OpenAPI Generator: python (client) [main] INFO o.o.codegen.DefaultGenerator - Generator 'python' is considered stable. [main] INFO o.o.c.languages.PythonClientCodegen - Environment variable PYTHON_POST_PROCESS_FILE not defined so the Python code may not be properly formatted. To define it, try 'export PYTHON_POST_PROCESS_FILE="/usr/local/bin/yapf -i"' (Linux/Mac) [main] INFO o.o.c.languages.PythonClientCodegen - NOTE: To enable file post-processing, 'enablePostProcessFile' must be set to `true` (--enable-post-process-file for CLI). [main] WARN o.o.codegen.utils.URLPathUtils - 'host' (OAS 2.0) or 'servers' (OAS 3.0) not defined in the spec. Default to [http://localhost] for server URL [http://localhost/] [main] WARN o.o.codegen.utils.URLPathUtils - 'host' (OAS 2.0) or 'servers' (OAS 3.0) not defined in the spec. Default to [http://localhost] for server URL [http://localhost/] [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Condition is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LLMInferenceServiceSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelCacheStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelNodeGroupStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeClaimSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PersistentVolumeSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.LocalModelInfo is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.WorkerSpec is not defined [main] INFO o.o.codegen.DefaultGenerator - Model v1alpha1.ServingRuntimeStatus not generated since it's a free-form object [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ObjectMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1alpha1.TrainedModelStatus is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ListMeta is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvVar is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EnvFromSource is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Lifecycle is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerPort is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerResizePolicy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ContainerRestartRule is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.SecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Probe is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeDevice is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.VolumeMount is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.autoscaling.v2.HorizontalPodAutoscalerBehavior is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/k8s.io.api.apps.v1.DeploymentStrategy is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.Addressable is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.dev.serving.pkg.apis.serving.v1.TrafficTarget is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/knative.URL is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/resource.Quantity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodReadinessGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodResourceClaim is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSchedulingGate is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Toleration is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.TopologySpreadConstraint is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Volume is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodOS is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.ResourceRequirements is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodSecurityContext is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Affinity is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.PodDNSConfig is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.EphemeralContainer is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.HostAlias is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.LocalObjectReference is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.ModelUtils - #/components/schemas/v1.Container is not defined [main] WARN o.o.codegen.utils.Model…
|
@KillianGolds: This pull request references RHOAIENG-50060 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target either version "4.22." or "openshift-4.22.", but it targets "rhoai-3.4" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
hack/setup/quick-install/llmisvc-dependency-install.sh (1)
520-520: Same unusedKEDA_VERSIONissue asllmisvc-full-install-helm.sh.No KEDA component is installed by this script, so
KEDA_VERSIONis dead. This is the same generator-level fix described in the comment onllmisvc-full-install-helm.sh.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@hack/setup/quick-install/llmisvc-dependency-install.sh` at line 520, Remove the unused KEDA_VERSION variable from this script: delete the KEDA_VERSION=2.17.3 declaration in hack/setup/quick-install/llmisvc-dependency-install.sh (symbol: KEDA_VERSION) and instead apply the same generator-level fix used for llmisvc-full-install-helm.sh so the generator no longer emits KEDA_VERSION into scripts that do not install KEDA; if KEDA is intended to be installed here, instead add the corresponding installation logic referencing KEDA_VERSION and ensure the variable is used.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@hack/setup/quick-install/llmisvc-full-install-helm.sh`:
- Line 520: The KEDA_VERSION variable is dead code in
llmisvc-full-install-helm.sh (and also appears in
llmisvc-dependency-install.sh); update the generator to only emit version
variables that are actually used by a script instead of blindly copying all
entries from kserve-deps.env, or remove the unused KEDA_VERSION assignment from
these scripts. Specifically, modify the generator logic that reads
kserve-deps.env so it filters for only the component names referenced by the
target script (or template) before emitting assignments, ensuring KEDA_VERSION
is not output for llmisvc-full-install-helm.sh and
llmisvc-dependency-install.sh; alternatively, delete the KEDA_VERSION line from
those scripts if you prefer the quick fix.
---
Duplicate comments:
In `@hack/setup/quick-install/llmisvc-dependency-install.sh`:
- Line 520: Remove the unused KEDA_VERSION variable from this script: delete the
KEDA_VERSION=2.17.3 declaration in
hack/setup/quick-install/llmisvc-dependency-install.sh (symbol: KEDA_VERSION)
and instead apply the same generator-level fix used for
llmisvc-full-install-helm.sh so the generator no longer emits KEDA_VERSION into
scripts that do not install KEDA; if KEDA is intended to be installed here,
instead add the corresponding installation logic referencing KEDA_VERSION and
ensure the variable is used.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to data retention organization setting
📒 Files selected for processing (10)
hack/setup/quick-install/kserve-knative-mode-dependency-install.shhack/setup/quick-install/kserve-knative-mode-full-install-helm.shhack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.shhack/setup/quick-install/kserve-standard-mode-dependency-install.shhack/setup/quick-install/kserve-standard-mode-full-install-helm.shhack/setup/quick-install/kserve-standard-mode-full-install-with-manifests.shhack/setup/quick-install/llmisvc-dependency-install.shhack/setup/quick-install/llmisvc-full-install-helm.shhack/setup/quick-install/llmisvc-full-install-with-manifests.shkserve-deps.env
✅ Files skipped from review due to trivial changes (1)
- hack/setup/quick-install/kserve-standard-mode-dependency-install.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- hack/setup/quick-install/kserve-knative-mode-full-install-with-manifests.sh
| KSERVE_VERSION=v0.16.0 | ||
| ISTIO_VERSION=1.27.1 | ||
| KEDA_VERSION=2.17.2 | ||
| KEDA_VERSION=2.17.3 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
rg -n 'KEDA_VERSION' \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.shRepository: opendatahub-io/kserve
Length of output: 221
🏁 Script executed:
# Check for KEDA-related installation functions or steps
rg -n -i 'install.*keda|keda.*install|keda|_keda' \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.sh | head -20Repository: opendatahub-io/kserve
Length of output: 599
🏁 Script executed:
# Get line count to understand file size
wc -l hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.sh
# Search for usage of KEDA_VERSION (excluding the definition line)
rg 'KEDA_VERSION' --no-filename -B 2 -A 2 \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.shRepository: opendatahub-io/kserve
Length of output: 442
🏁 Script executed:
# Search for function definitions related to KEDA or any install functions
rg -n 'install_|^[a-z_]+\(\)' \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.sh | grep -i keda
# Also check for any invocations of install functions
rg -n 'install_' \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.sh | grep -v ':#.*install_' | head -20Repository: opendatahub-io/kserve
Length of output: 1866
🏁 Script executed:
# List all install/uninstall functions defined in both scripts
rg -n '^(un)?install_' \
hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.shRepository: opendatahub-io/kserve
Length of output: 3342
🏁 Script executed:
# Check the top of the files for any auto-generation markers
head -20 hack/setup/quick-install/llmisvc-full-install-helm.sh \
hack/setup/quick-install/llmisvc-dependency-install.sh
# Search for generator script
find hack -name "*generat*" -o -name "*template*" | head -20Repository: opendatahub-io/kserve
Length of output: 2278
🏁 Script executed:
# Find and check the definition files
find hack -name "*.definition" | xargs ls -la
# Check if KEDA appears in any definition files
rg -l 'keda|KEDA' hack/setup/scripts/install-script-generator/ hack/setup/quick-install/ 2>/dev/nullRepository: opendatahub-io/kserve
Length of output: 2832
🏁 Script executed:
# Check the llmisvc definition files
cat hack/setup/quick-install/definitions/llmisvc/llmisvc-dependency-install.definition
cat hack/setup/quick-install/definitions/llmisvc/llmisvc-full-install.definition
# Also check what kserve-deps.env contains
cat hack/kserve-deps.env | grep -i kedaRepository: opendatahub-io/kserve
Length of output: 879
🏁 Script executed:
# Find version definition files
find hack -name "*deps*" -o -name "*version*" | grep -E "\.(env|yaml|json|definition)$" | head -20
# Check the generator to see how it includes versions
cat hack/setup/scripts/install-script-generator/generator.py | head -100Repository: opendatahub-io/kserve
Length of output: 3734
🏁 Script executed:
# Search for where versions are loaded
rg -n 'KEDA_VERSION|version.*env|\.env' hack/setup/scripts/install-script-generator/ | head -30
# Check for component definition directories
find hack/setup/infra -type d | head -20Repository: opendatahub-io/kserve
Length of output: 1556
🏁 Script executed:
# Check script_builder.py around line 228
sed -n '220,250p' hack/setup/scripts/install-script-generator/pkg/script_builder.py
# Also find the kserve-deps.env file
find . -name "kserve-deps.env" 2>/dev/null
# Look for it in the repo root
ls -la kserve-deps.env 2>/dev/null || echo "Not found in root"Repository: opendatahub-io/kserve
Length of output: 1589
🏁 Script executed:
# Check the content of kserve-deps.env
cat kserve-deps.env
# Also check the template to see how KSERVE_DEPS_CONTENT is inserted
sed -n '95,115p' hack/setup/scripts/install-script-generator/templates/generated-script.templateRepository: opendatahub-io/kserve
Length of output: 1577
KEDA_VERSION is unused—remove dead version variable from generator.
This script has no KEDA installation step, yet KEDA_VERSION is defined at line 520. The same issue appears in llmisvc-dependency-install.sh. The generator currently includes all version variables from kserve-deps.env unconditionally; it should emit only versions for components that are actually referenced in each script's definition.
🧰 Tools
🪛 Shellcheck (0.11.0)
[warning] 520-520: KEDA_VERSION appears unused. Verify use (or export if used externally).
(SC2034)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@hack/setup/quick-install/llmisvc-full-install-helm.sh` at line 520, The
KEDA_VERSION variable is dead code in llmisvc-full-install-helm.sh (and also
appears in llmisvc-dependency-install.sh); update the generator to only emit
version variables that are actually used by a script instead of blindly copying
all entries from kserve-deps.env, or remove the unused KEDA_VERSION assignment
from these scripts. Specifically, modify the generator logic that reads
kserve-deps.env so it filters for only the component names referenced by the
target script (or template) before emitting assignments, ensuring KEDA_VERSION
is not output for llmisvc-full-install-helm.sh and
llmisvc-dependency-install.sh; alternatively, delete the KEDA_VERSION line from
those scripts if you prefer the quick fix.
|
@KillianGolds: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds accelerator-specific
LLMInferenceServiceConfigresources for NVIDIA CUDA, AMD ROCm, and IBM Spyre (s390x, x86, ppc64le) underconfig/overlays/odh/accelerators/, and modifies the baseLLMInferenceServiceConfigtemplate to support Spyre without requiringcommand/argsoverrides in the accelerator configs.Changes:
config/llmisvcconfig/config-llm-template.yaml): wraps the vLLM launch in a bash command that conditionally sources/etc/profile.d/ibm-aiu-setup.shif present (no-op on non-Spyre nodes). Uses$@forwarding so all args - model name, port, and any user-specified flags — pass through correctly.config/overlays/odh/accelerators/: NVIDIA CUDA and AMD ROCm override image only; IBM Spyre configs (s390x, x86, ppc64le) override image and set arch-specific env vars required for Spyre startup.argsoverrides in any accelerator config - if an accelerator config overridesargs, user-specified args on theirLLMInferenceServicewould be silently discarded. This aligns with the fix landing via feat(llmisvc): move vLLM args to command field kserve/kserve#5049.replacementssourcing fromparams.envthrough thekserve-parametersConfigMap.opendatahub.io/config-type: acceleratorfor dashboard filtering.Which issue(s) this PR fixes:
Fixes # RHOAIENG-50060
Feature/Issue validation/testing:
kustomize build config/overlays/odh/- 5 acceleratorLLMInferenceServiceConfigresources rendered with correct images substituted fromparams.envandopendatahub.io/config-type: acceleratorlabel appliedTestMergeSpecs(27 sub-tests),TestReplaceVariables(10 sub-tests) - all passTestLLMInferenceServiceController(92 specs via envtest) - all passibm-aiu-setup.shabsent, vLLM starts as before)ibm-aiu-setup.shsourced correctly, vLLM starts successfullySpecial notes for your reviewer:
The base template change (bash wrapper + conditional ibm-aiu-setup.sh sourcing) will interact with the upcoming PR syncing kserve#5049 which is landing in #1123 - that PR will also modify the base template command. The change here is structurally compatible: both add a bash wrapper with $@ forwarding. Merge conflict resolution will be straightforward - integrate the conditional source line into that version.
IBM Power team has confirmed availability to test on P (ppc64le) and Z (s390x) as per comments on RHOAIENG-50350, will be coordinating for full e2e testing.
Checklist:
Release note:
Summary by CodeRabbit
New Features
Chores