Skip to content

Commit 4d50ebe

Browse files
nathan-weinbergderekhiggins
nathan-weinberg
and
derekhiggins
authored
ci: update distro container action with smoke test (#16)
* ci: update distro container action with smoke test also add CI linter to pre-commit config Assisted-by: coderabbitai Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * chore: bump trustyai_fms to 0.2.1 to mirror d/s change and unblock CI Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * chore: bump lmeval to 0.2.4 to mirror d/s change and unblock CI Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * ci: add vllm action and integration to image test Co-authored-by: Derek Higgins <derekh@redhat.com> Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * ci: standardize concurrency rules Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * fix: add env var to disable K8S usage for LM Eval Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * refactor: move smoke tests into a seperate shell script also add shell linter to pre-commit config Assisted-by: coderabbitai Signed-off-by: Nathan Weinberg <nweinber@redhat.com> * fix: modify VLLM_URL for containerized environments Signed-off-by: Nathan Weinberg <nweinber@redhat.com> --------- Signed-off-by: Nathan Weinberg <nweinber@redhat.com> Co-authored-by: Derek Higgins <derekh@redhat.com>
1 parent 9448807 commit 4d50ebe

File tree

10 files changed

+205
-63
lines changed

10 files changed

+205
-63
lines changed

‎.github/actions/setup-vllm/action.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: Setup VLLM
2+
description: Start VLLM
3+
runs:
4+
using: "composite"
5+
steps:
6+
- name: Start VLLM
7+
shell: bash
8+
run: |
9+
# Start vllm container
10+
docker run -d \
11+
--name vllm \
12+
-p 8000:8000 \
13+
--privileged=true \
14+
--net=host \
15+
quay.io/higginsd/vllm-cpu:65393ee064 \
16+
--host 0.0.0.0 \
17+
--port 8000 \
18+
--enable-auto-tool-choice \
19+
--tool-call-parser llama3_json \
20+
--model /root/.cache/Llama-3.2-1B-Instruct \
21+
--served-model-name meta-llama/Llama-3.2-1B-Instruct
22+
23+
# Wait for vllm to be ready
24+
echo "Waiting for vllm to be ready..."
25+
timeout 900 bash -c 'until curl -fsS http://localhost:8000/health >/dev/null; do
26+
echo "Waiting for vllm..."
27+
sleep 5
28+
done'

‎.github/workflows/pre-commit.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ on:
66
branches: [main]
77

88
concurrency:
9-
group: ${{ github.workflow }}-${{ github.ref }}
9+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
1010
cancel-in-progress: true
1111

1212
jobs:

‎.github/workflows/redhat-distro-container-build.yml‎

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎.github/workflows/redhat-distro-container.yml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
name: Build, test, and publish Red Hat Distribution Containers
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
- rhoai-v*
8+
types:
9+
- opened
10+
- synchronize
11+
push:
12+
branches:
13+
- main
14+
- rhoai-v*
15+
16+
concurrency:
17+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
18+
cancel-in-progress: true
19+
20+
env:
21+
REGISTRY: quay.io
22+
IMAGE_NAME: quay.io/opendatahub/llama-stack # tags for the image will be added dynamically
23+
24+
jobs:
25+
build-test-push:
26+
runs-on: ubuntu-latest
27+
strategy:
28+
matrix:
29+
platform: [linux/amd64] # TODO: enable other arch once all pip packages are available.
30+
31+
steps:
32+
- name: Checkout repository
33+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
34+
35+
- name: Set up QEMU
36+
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
37+
38+
- name: Set up Docker Buildx
39+
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
40+
41+
- name: Build image
42+
id: build
43+
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
44+
with:
45+
context: .
46+
file: distribution/Containerfile
47+
platforms: ${{ matrix.platform }}
48+
push: false
49+
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
50+
load: true # needed to load for smoke test
51+
cache-from: type=gha
52+
cache-to: type=gha,mode=max
53+
54+
- name: Setup vllm for image test
55+
id: vllm
56+
uses: ./.github/actions/setup-vllm
57+
58+
- name: Smoke test image
59+
id: smoke-test
60+
shell: bash
61+
env:
62+
INFERENCE_MODEL: meta-llama/Llama-3.2-1B-Instruct
63+
VLLM_URL: http://localhost:8000/v1
64+
run: ./tests/smoke.sh
65+
66+
- name: Log in to Quay.io
67+
id: login
68+
if: github.event_name == 'push'
69+
uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
70+
with:
71+
registry: ${{ env.REGISTRY }}
72+
username: ${{ secrets.QUAY_USERNAME }}
73+
password: ${{ secrets.QUAY_PASSWORD }}
74+
75+
- name: Publish image to Quay.io
76+
id: publish
77+
if: github.event_name == 'push'
78+
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
79+
with:
80+
context: .
81+
file: distribution/Containerfile
82+
platforms: ${{ matrix.platform }}
83+
push: true
84+
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}${{ github.ref == 'refs/heads/main' && format(',{0}:latest', env.IMAGE_NAME) || '' }} # only update 'latest' tag if push is to the 'main' branch
85+
cache-from: type=gha
86+
cache-to: type=gha,mode=max

‎.github/workflows/semantic-pr.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ on:
99
- synchronize
1010

1111
concurrency:
12-
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
12+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
1313
cancel-in-progress: true
1414

1515
permissions:

‎.pre-commit-config.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,16 @@ repos:
3535
args: [ --fix ]
3636
- id: ruff-format
3737

38+
- repo: https://github.com/rhysd/actionlint
39+
rev: v1.7.7
40+
hooks:
41+
- id: actionlint
42+
43+
- repo: https://github.com/koalaman/shellcheck-precommit
44+
rev: v0.11.0
45+
hooks:
46+
- id: shellcheck
47+
3848
- repo: local
3949
hooks:
4050
- id: pkg-gen

‎distribution/Containerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ RUN pip install \
1414
fire \
1515
httpx \
1616
kubernetes \
17-
llama_stack_provider_lmeval==0.1.7 \
18-
llama_stack_provider_trustyai_fms==0.1.2 \
17+
llama_stack_provider_lmeval==0.2.4 \
18+
llama_stack_provider_trustyai_fms==0.2.1 \
1919
matplotlib \
2020
mcp>=1.8.1 \
2121
nltk \

‎distribution/providers.d/remote/eval/trustyai_lmeval.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
adapter:
22
adapter_type: trustyai_lmeval
3-
pip_packages: ["kubernetes", "llama_stack_provider_lmeval==0.1.7"]
3+
pip_packages: ["kubernetes", "llama_stack_provider_lmeval==0.2.4"]
44
config_class: llama_stack_provider_lmeval.config.LMEvalEvalProviderConfig
55
module: llama_stack_provider_lmeval
66
api_dependencies: ["inference"]

‎distribution/providers.d/remote/safety/trustyai_fms.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
adapter:
22
adapter_type: trustyai_fms
3-
pip_packages: ["llama_stack_provider_trustyai_fms==0.1.2"]
3+
pip_packages: ["llama_stack_provider_trustyai_fms==0.2.1"]
44
config_class: llama_stack_provider_trustyai_fms.config.FMSSafetyProviderConfig
55
module: llama_stack_provider_trustyai_fms
66
api_dependencies: ["safety"]

‎tests/smoke.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
3+
set -uo pipefail
4+
5+
function start_and_wait_for_llama_stack_container {
6+
# Start llama stack
7+
docker run \
8+
-d \
9+
--pull=never \
10+
--net=host \
11+
-p 8321:8321 \
12+
--env INFERENCE_MODEL="$INFERENCE_MODEL" \
13+
--env VLLM_URL="$VLLM_URL" \
14+
--env TRUSTYAI_LMEVAL_USE_K8S=False \
15+
--env TRUSTYAI_LM_EVAL_NAMESPACE=dummy \
16+
--name llama-stack \
17+
"$IMAGE_NAME:$GITHUB_SHA"
18+
echo "Started Llama Stack container..."
19+
20+
# Wait for llama stack to be ready by doing a health check
21+
echo "Waiting for Llama Stack server..."
22+
for i in {1..60}; do
23+
echo "Attempt $i to connect to Llama Stack..."
24+
resp=$(curl -fsS http://127.0.0.1:8321/v1/health)
25+
if [ "$resp" == '{"status":"OK"}' ]; then
26+
echo "Llama Stack server is up!"
27+
return
28+
fi
29+
sleep 1
30+
done
31+
echo "Llama Stack server failed to start :("
32+
echo "Container logs:"
33+
docker logs llama-stack || true
34+
exit 1
35+
}
36+
37+
function test_model_list {
38+
echo "===> Looking for model $INFERENCE_MODEL..."
39+
resp=$(curl -fsS http://127.0.0.1:8321/v1/models)
40+
if echo "$resp" | grep -q "$INFERENCE_MODEL"; then
41+
echo "Model $INFERENCE_MODEL was found :)"
42+
return
43+
else
44+
echo "Model $INFERENCE_MODEL was not found :("
45+
echo "Container logs:"
46+
docker logs llama-stack || true
47+
exit 1
48+
fi
49+
}
50+
51+
function test_model_openai_inference {
52+
echo "===> Attempting to chat with model $INFERENCE_MODEL..."
53+
resp=$(curl -fsS http://127.0.0.1:8321/v1/openai/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"$INFERENCE_MODEL\",\"messages\": [{\"role\": \"user\", \"content\": \"What color is grass?\"}], \"max_tokens\": 10, \"temperature\": 0.0}")
54+
if echo "$resp" | grep -q "green"; then
55+
echo "===> Inference is working :)"
56+
return
57+
else
58+
echo "===> Inference is not working :("
59+
echo "Container logs:"
60+
docker logs llama-stack || true
61+
exit 1
62+
fi
63+
}
64+
65+
main() {
66+
echo "===> Starting smoke test..."
67+
start_and_wait_for_llama_stack_container
68+
test_model_list
69+
test_model_openai_inference
70+
echo "===> Smoke test completed successfully!"
71+
}
72+
73+
trap 'docker rm -f -v llama-stack >/dev/null 2>&1 || true' EXIT
74+
main "$@"
75+
exit 0

0 commit comments

Comments
 (0)