feat: Add blueprint TensorRT-LLM + Triton #65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
omrishiv
requested changes
May 13, 2025
There was a problem hiding this comment.
This is incredibly thorough, thank you so much for the contribution! I have a few small requests/questions. @vara-bonthu would you also mind taking a look as this infrastructure is the only one I haven't consolidated and you may be a little more familiar with it.
| blueprints/inference/trtllm-nvidia-triton-server-gpu/triton_model_files/* | ||
| blueprints/inference/trtllm-nvidia-triton-server-gpu/benchmark-grpc/results.txt | ||
| blueprints/inference/trtllm-nvidia-triton-server-gpu/benchmark-http/results/* | ||
| blueprints/inference/trtllm-nvidia-triton-server-gpu/.ecr_repo_uri |
There was a problem hiding this comment.
As .ecr_repo_uri and .eks_region seem to be a common pattern, can we gitignore them with
**/.ecr_repo_uri
**/.eks_region
| COPY start.sh /start.sh | ||
| RUN chmod +x /start.sh | ||
|
|
||
| ENTRYPOINT ["/bin/bash", "/start.sh"] No newline at end of file |
| echo -e "\nBuilding llama finetuning trn1 docker image" \ | ||
| && docker build . --no-cache -t $ECR_REPO_URI:latest \ | ||
| && docker push $ECR_REPO_URI:latest \ | ||
| && echo -e "\nImage successfully pushed to ECR" No newline at end of file |
| print(f"* {region}: {region_long_name}") | ||
| for instance_type in instance_types: | ||
| print(f" - {instance_type}") | ||
| print("\n") No newline at end of file |
| @@ -0,0 +1,3 @@ | |||
| #!/bin/bash | |||
|
|
|||
| python3 /tensorrtllm_backend/scripts/launch_triton_server.py --world_size=1 --model_repo=/triton_model_files No newline at end of file | |||
| instanceStorePolicy: RAID0 | ||
| nodePool: | ||
| labels: | ||
| - type: karpenter |
There was a problem hiding this comment.
can we align these to the current labels:
- instanceType: g6e-gpu-karpenter
- type: karpenter
- accelerator: nvidia
- gpuType: l40s
| values: ["amd64"] | ||
| - key: "karpenter.sh/capacity-type" | ||
| operator: In | ||
| values: ["on-demand"] |
| # --------------------------------------------------------------------------------------------------------- # | ||
| # NOTE: this is a reminder to modify "aws s3 sync command" within provisioner "local-exec", before deploying | ||
| # --------------------------------------------------------------------------------------------------------- # | ||
| # module "triton_server_trtllm" { |
There was a problem hiding this comment.
why is all of this commented out?
| bucket_name = module.s3_bucket[count.index].s3_bucket_id | ||
| } | ||
|
|
||
| provisioner "local-exec" { |
There was a problem hiding this comment.
is this possible to toggle base on which one you're using?
| @@ -0,0 +1,658 @@ | |||
| --- | |||
| title: NVIDIA Triton Server with TensorRT LLM | |||
| sidebar_position: 2 | |||
There was a problem hiding this comment.
please remove the fixed positioning
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #53!
Adds a new blueprint at
blueprints/inference/trtllm-nvidia-triton-server-gpufor deploying and serving LLMs using TensorRT-LLM with NVIDIA Triton Inference Server, demonstrated with a 1B parameter LLaMA model for ultra-low latency.Includes:
Motivation
No existing blueprint demonstrates optimized LLM inference with TensorRT-LLM + Triton. This fills that gap with a performant, scalable GPU-based solution.
More
website/docsorwebsite/blogsection for this featurepre-commit run -awith this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes