feat: enhance integration testing workflow for Vertex AI by Artemon-line · Pull Request #136 · opendatahub-io/llama-stack-distribution

Artemon-line · 2025-11-25T14:43:03Z

Added a step to check for Vertex AI recordings in the integration tests workflow, allowing conditional execution of Vertex AI tests based on the presence of recordings.
Updated the integration tests script to dynamically select the text model based on the environment, supporting both Vertex AI and vllm-inference providers.

This improves the testing process by enabling the use of recorded data for Vertex AI, ensuring more reliable and efficient test runs.

What does this PR do?

Test Plan

- Added a step to check for Vertex AI recordings in the integration tests workflow, allowing conditional execution of Vertex AI tests based on the presence of recordings. - Updated the integration tests script to dynamically select the text model based on the environment, supporting both Vertex AI and vllm-inference providers. This improves the testing process by enabling the use of recorded data for Vertex AI, ensuring more reliable and efficient test runs. Signed-off-by: Artemy <ahladenk@redhat.com>

coderabbitai · 2025-11-25T14:43:24Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

Provide your own instructions using the high_level_summary_instructions setting.
Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

📝 Description — Summarize the main change in 50–60 words, explaining what was done.

📓 References — List relevant issues, discussions, documentation, or related PRs.

📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.

📊 Contributor Summary — Include a Markdown table showing contributions:
| Contributor | Lines Added | Lines Removed | Files Changed |

✔️ Additional Notes — Add any extra reviewer context.
Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

nathan-weinberg · 2025-11-25T14:58:34Z

@Artemon-line can you refrain from adding reviewers until the PR is out of draft?

ktdreyer · 2025-11-25T15:16:22Z

.github/workflows/live-tests.yml

+        run: |
+          docker run -d --net=host -p 8321:8321 \
+            -v "$HOME/.config/gcloud:/root/.config/gcloud:ro" \
+            -e VERTEX_AI_PROJECT="${{ env.VERTEX_AI_PROJECT }}" \


Why don't we use secrets.VERTEX_AI_PROJECT directly here? There is a level of indirection that makes this more difficult to follow than necessary IMHO

ktdreyer · 2025-11-25T15:17:22Z

.github/workflows/live-tests.yml

+      - name: Set Vertex AI environment variables
+        run: |
+          echo "VERTEX_AI_PROJECT=${{ secrets.VERTEX_AI_PROJECT }}" >> $GITHUB_ENV
+          echo "VERTEX_AI_LOCATION=${{ secrets.VERTEX_AI_LOCATION || 'us-central1' }}" >> $GITHUB_ENV


VERTEX_AI_LOCATION is not a secret. We could just hardcode it to us-central1 for simplicity.

ktdreyer · 2025-11-25T15:19:58Z

.github/workflows/live-tests.yml

+          **Source**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+          **Date**: $(date -u +%Y-%m-%d)
+
+          Auto-generated by the live CI tests workflow." \


We're going to have a lot of commits with this message. You can delete this Auto-generated line - it's just repetitive noise since we're recording Source and Date.

ktdreyer · 2025-11-25T15:22:12Z

docs/github-actions-testing-guide.md

+
+## Quick Start
+
+### 1. Install `act`


This section is duplicated with above.

ktdreyer · 2025-11-25T15:22:52Z

docs/github-actions-testing-guide.md

+podman ps
+```
+
+**Note**: The helper script (`scripts/test-with-act.sh`) will automatically detect and use Podman if available, falling back to Docker if needed. GitHub Actions workflows use Docker, but Podman is preferred for local testing.


This sentence "GitHub Actions workflows use Docker, but Podman is preferred for local testing." is duplicated several times.

ktdreyer · 2025-11-25T15:28:01Z

docs/github-actions-testing-guide.md

+
+### 2. Self-Hosted Actions
+
+Actions in `.github/actions/` need to be available. `act` should handle this automatically.


I don't understand the purpose of this section

ktdreyer · 2025-11-25T15:29:50Z

docs/live-tests-guide.md

+
+1. **Podman** - Required for building and running containers (including vllm container)
+2. **gcloud CLI** - For GCP authentication (if using Vertex AI provider)
+3. **Git** - For cloning repositories


These documents should assume basic things like Git or Python are installed.

ktdreyer · 2025-11-25T15:30:23Z

docs/live-tests-guide.md

+#### Vertex AI
+- `VERTEX_AI_PROJECT` - GCP project ID (required)
+- `VERTEX_AI_LOCATION` - GCP region (optional, defaults to us-central1)
+- GCP authentication via `gcloud auth application-default login`


This information is duplicated with above.

ktdreyer · 2025-11-25T15:30:47Z

docs/live-tests-guide.md

+gcloud auth application-default login
+
+# Verify project is set
+echo $VERTEX_AI_PROJECT


The shell script can't do this automatically for us? Can we take this out of the .md file that humans read?

ktdreyer · 2025-11-25T15:38:36Z

.github/workflows/live-tests.yml

+        uses: google-github-actions/auth@v2
+        with:
+          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}


We're using the procedure documented in the section "(Preferred) Direct Workload Identity Federation". There is no service account - the service_account parameter here is unnecessary.

ktdreyer · 2025-11-25T15:40:22Z

@Artemon-line I suggest that you push your changes to a work-in-progress branch in this opendatahub-io repo (instead of your fork) so that 1) we can access this repository's secrets, and 2) GCP will authorize the GH Actions.

When we're happy with how this is working on that branch, then we can create the PR to main.

Artemon-line · 2025-11-25T16:17:02Z

@Artemon-line I suggest that you push your changes to a work-in-progress branch in this opendatahub-io repo (instead of your fork) so that 1) we can access this repository's secrets, and 2) GCP will authorize the GH Actions.

When we're happy with how this is working on that branch, then we can create the PR to main.

Thanks a lot @ktdreyer, I will follow up on all the findings you highlighted and also redo some stuff to make it less robust. I understand your point about using the repo branch, which is my next step; this one was just for an overall introduction to the idea of reusing recordings.

Artemon-line requested review from ktdreyer, leseb and nathan-weinberg November 25, 2025 14:43

ktdreyer requested changes Nov 25, 2025

View reviewed changes

Artemon-line closed this Nov 25, 2025


		### 2. Self-Hosted Actions

		Actions in `.github/actions/` need to be available. `act` should handle this automatically.

Conversation

Artemon-line commented Nov 25, 2025

What does this PR do?

Test Plan

Uh oh!

coderabbitai bot commented Nov 25, 2025

Review skipped

Uh oh!

nathan-weinberg commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ktdreyer commented Nov 25, 2025

Uh oh!

Artemon-line commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants