Skip to content

README.md

Latest commit

 

History

History
89 lines (59 loc) · 4.17 KB

README.md

File metadata and controls

89 lines (59 loc) · 4.17 KB

Client Setup Prerequisites

llm-d guides use a standard set of client tools on Linux and Mac OSX. The provided install-deps.sh script will download and install the tools below.

Supported Development Platforms

Currently llm-d community only supports OSX and Linux development.

Required Tools

Binary Minimum Required Version Download / Installation Instructions
yq v4+ yq (mikefarah) – installation
git v2.30.0+ git – installation guide
helm v3.12.0+ Helm – quick-start install
helmfile v1.1.0+ Helmfile - installation
kubectl v1.28.0+ kubectl – install & setup

Optional Tools

Binary Recommended Version Download / Installation Instructions
stern 1.30+ stern - installation
helm diff plugin v3.10.0+ helm diff installation docs

HuggingFace Token

Most guides download their model from Huggingface directly in the llm-d image. There are exceptions to this like the simulated-accelerators guide that uses no model, or the wide-ep-lws guide which uses a model loaded from storage directly on the nodes for faster development cycle iterations.

For the rest you will need to create a Kubernetes secret in your deployment namespace containing your HuggingFace Token. For more information on getting a token, see the huggingface docs.

The following script will create the token in the current namespace using the name llm-d-hf-token, which is used in all guides:

export HF_TOKEN=<from Huggingface>
export HF_TOKEN_NAME=${HF_TOKEN_NAME:-llm-d-hf-token}
kubectl create secret generic ${HF_TOKEN_NAME} \
    --from-literal="HF_TOKEN=${HF_TOKEN}" \
    --namespace "${NAMESPACE}" \
    --dry-run=client -o yaml | kubectl apply -f -

Choose an llm-d Version {#llm-d-version}

Check for known issues first.

Choose Main Branch

Choose the main branch if you want the latest llm-d.

branch="main"
# Clone the repo and switch to the desired branch
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

Choose the Latest Release

Choose the latest release of llm-d with a high qualification standard.

branch=$(curl -s https://api.github.com/repos/llm-d/llm-d/releases/latest | jq -r '.tag_name')
# Clone the repo and switch to the desired branch
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

Choose a Commit

Choose this when you need a specific feature/bug fix, and lock your environment to a specific commit.

branch=<commit_sha>
# Clone the repo and switch to the desired branch
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

Pulling llm-d Images from GitHub Container Registry (GHCR)

All of the container images in the llm-d organization are public on GitHub and require no authentication to pull.