Skip to content

Add new queries - #26

Draft
Azaya89 wants to merge 9 commits into
mainfrom
add-queries
Draft

Add new queries#26
Azaya89 wants to merge 9 commits into
mainfrom
add-queries

Conversation

@Azaya89

@Azaya89 Azaya89 commented May 29, 2026

Copy link
Copy Markdown
Collaborator

This PR introduces several improvements and new features to the evaluation pipeline.

Key changes include:

Evaluation Queries

  • New Complex Query for Dependency Dashboard:
    Adds a new evaluation query requiring the creation of a multi-view Panel+hvPlot dashboard based on JSON data from a GitHub repository, with detailed schema and visualization requirements.

Evaluation Pipeline Enhancements

  • Panel Dashboard Execution and Screenshot Support:
    The evaluation system now detects when a generated script is expected to produce a Panel dashboard (expected_output="panel dashboard"). Instead of running the script directly (which would block on pn.serve()), it launches the app with panel serve and uses Playwright to capture a screenshot.

  • Passing Expected Output Metadata:
    The expected_output field is now passed throughout the evaluation pipeline.

  • Per-query Timeout Support:
    Query-specific timeouts are now supported and passed to the execution logic.

Documentation Additions and Corrections

  • hvPlot Bar Chart Usage Guide:
    The SKILL.md for hvPlot now includes a comprehensive guide on creating bar and barh charts, correct use of color, axis conventions, and common pitfalls (e.g., not using kind=, correct use of invert, and formatter requirements).
  • Panel Widget Extension and State Checking:
    The Panel skills documentation clarifies that pn.widgets.Tabulator requires "tabulator" in pn.extension(), and specifies the correct way to check if the app is running under panel serve (pn.state.served), warning against outdated or incorrect patterns.
  • Guidance for Fetching Data from GitHub:
    Adds a new section explaining how to safely fetch data files from GitHub using the Contents API, warning against the use of raw.githubusercontent.com URLs.

Miscellaneous

  • Task Dependency Order Correction in pixi.toml:
    Ensures that the evaluation setup and checks run in the correct order.
  • Regex Fix for Display Call Removal:
    Refines the regex used to comment out show() calls in generated code.

NOTE: The end result is still not consistently good. The model still generates errors at random after every re-run.

@github-actions

This comment was marked as outdated.

@Azaya89

Azaya89 commented May 29, 2026

Copy link
Copy Markdown
Collaborator Author

@run-eval on this

@github-actions

Copy link
Copy Markdown

Eval Command

✅ Status: success
Run: https://github.com/holoviz-dev/holoviz-skills/actions/runs/26653025554
Artifacts: Download the eval-results-* artifact from the run page.

Evaluation summary

Evaluation Summary

Total Queries: 2

Aggregate Statistics

Code Generation Rate

Condition Rate
With Skills 50.0%
Without Skills 50.0%

Execution Success Rate

Condition Rate
With Skills 100.0%
Without Skills 100.0%

Resource Usage

Avg Token Difference: +0 tokens
Avg Response Time Difference: -27.29s

Per-Query Results

dependency_plot

Metric With Skills Without Skills
Code Generated False False
Response Time 120.04s 120.02s
Tokens (output) 0 0

hvplot_earthquake_plot

Metric With Skills Without Skills
Code Generated True True
Response Time 19.80s 74.41s
Tokens (output) 0 0
Execution Success True True

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant