Skip to content

Use NIM instead of Models through NGC Containers #6

@TharunSivamani

Description

@TharunSivamani

Is there a way to use NIM API Calls instead of having a model deployed on the GPU?

Would changing the values.yaml file while deploying the helm chart should do.
Is there any alternate for this, and the other models like reranker and embedding model?

nemollm-embedding-embedding-deployment-6bdc968784-9v2mm
nemollm-inference-nemollm-infer-deployment-5b88bf7bc-cmxsz  
ranking-ms-ranking-deployment-5c7768d88b-zvtt5

These pods would associate to the models by NVIDIA, so changing to NIM can also save costs.

Any help would be appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions