add a new ray service for Graviton based model inference with llamacpp

[ddynwzh1992]

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction. When we triage the issues, we will add labels to the issue like "Enhancement", "Bug" which should indicate to you that this issue can be worked on and we are looking forward to your PR. We would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

add a new ray service for Graviton based model inference with llamacpp

Motivation

provide a new cost effective solution for CPU based model inference on EKS

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
• Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
• Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

[vara-bonthu]

Thanks for raising the PR! Could you please create an issue first before we move forward with this?
This will need the following changes:

1/ Use the JARK stack as a prerequisite for deploying this model
2/ Ensure the Karpenter Nodepool is available to Graviton instances and use these labels in the Ray deployment. See this x86 example

ai-on-eks/infra/base/terraform/addons.tf

Line 483 in e77964b

x86-cpu-karpenter = {

3/ Create a folder called "Graviton" here: https://github.com/awslabs/ai-on-eks/tree/main/website/docs/blueprints/inference
4/ Move the README.md to this folder and make sure the documentation follows the same format as other examples like: https://awslabs.github.io/ai-on-eks/docs/blueprints/inference/GPUs/vLLM-rayserve

Let me know once you've got the issue created and we can track the progress there!

[omrishiv]

@vara-bonthu this was missing from the split from DoEKS, it's already in there, but agree with the comments

[ddynwzh1992]

@vara-bonthu I have created the issue , and move the doc to a new folder named Graviton

[vara-bonthu]

Thanks for updating the PR with comments. I would need your help to fix a few more things before approving the PR.

[vara-bonthu]

I don't think we have any Karpenter nodepool that creates this node for you. You may need to create a new nodepool for ARM. See this example for x86 https://github.com/awslabs/ai-on-eks/blob/e45a5a71702c5a04c27ac9cca72a510001580a7b/infra/base/terraform/addons.tf#L483C1-L537C6

and create a new one for ARM.

[vara-bonthu]

Just a question: Does this blueprint require custom benchmarking code or have we explored if we can use open source benchmarking tools? I am only thinking about the maintainability of this Go code in the longer term.

[vara-bonthu]

Use this doc as an example to write this doc. You need to add sidebar_label etc.

You can test deploy this website locally with simple steps and modify the doc. Checkout the instructions https://github.com/awslabs/ai-on-eks/blob/main/website/README.md

[ddynwzh1992]

Removed the sidebar_position

[omrishiv]

@vara-bonthu this looks good to me, can you re-review please? thank you

[omrishiv]

I know this was started way before we had the inference charts, but ultimately, that's where this deployment will move to. I will +1 the comment #88 (comment) that when we have benchmarking available for the inference charts, this will probably go away.