Skip to content

ci: Multiple random runs#3637

Draft
FBruzzesi wants to merge 2 commits into
mainfrom
ci/multi-random-runs
Draft

ci: Multiple random runs#3637
FBruzzesi wants to merge 2 commits into
mainfrom
ci/multi-random-runs

Conversation

@FBruzzesi
Copy link
Copy Markdown
Member

@FBruzzesi FBruzzesi commented May 16, 2026

Description

One quick experiment for #3623 as we have ~1min per run.

Question:

@dangotbanned we can also make it so that we take N different versions per library, not N different combos.

Here a unlucky case in which we got 4 times polars v1 (thse in screenshot + the failed one)

image

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

@FBruzzesi FBruzzesi added the ci label May 16, 2026
Comment thread utils/generate_random_versions.py Outdated
Comment on lines +57 to +63
def all_combos() -> tuple[tuple[str, str, str, str], ...]:
return tuple(
(pd, np, pl, pa)
for pd, np in PANDAS_AND_NUMPY_VERSION
for pl in POLARS_VERSION
for pa in PYARROW_VERSION
)
Copy link
Copy Markdown
Member

@dangotbanned dangotbanned May 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove completely to have all different versions in the different samples.
Aside that, I am aware but this was 1 extra line of code if you consider imports 🤣

@dangotbanned
Copy link
Copy Markdown
Member

dangotbanned commented May 16, 2026

we can also make it so that we take N different versions per library, not N different combos.

Yeah I think if we go in this direction, each library should contribute N_RUNS unique versions, which I think you've got now in (all different) 😄

@dangotbanned
Copy link
Copy Markdown
Member

dangotbanned commented May 17, 2026

One quick experiment for #3623 as we have ~1min per run.

So just to clarify that, here's the most recent and comparing where that sits alongside (https://github.com/narwhals-dev/narwhals/actions/metrics/performance):

Truly cursed query

import polars as pl

pl.Config(fmt_str_lengths=60, tbl_rows=17, tbl_formatting="MARKDOWN")
df = pl.read_csv(
    "b98fa657-9f9c-465c-83af-ad1cc611ef51.csv",
    new_columns=["Workflow", "Has job failures", "Avg run time", "Workflow runs", "Jobs"],
).select(
    pl.col("Workflow")
    .str.strip_chars(r'"\'.')
    .str.replace_many({"github/workflows/": "", "dynamic/dependabot/": ""}),
    pl.duration(
        minutes=pl.duration(milliseconds="Avg run time").dt.total_minutes(fractional=True)
    ).alias("Avg run time"),
)
df

Workflow Avg run time
pytest-pyspark.yml 14m 41s
downstream_tests.yml 8m 12s
publish_to_pypi.yml 7m 59s
pytest.yml 6m 53s
extremes.yml 3m 58s
This PR 🟦 3m 53s
check_tpch_queries.yml 3m 35s
typing.yml 2m 1s
Current 🟩 1m 38s
pytest-modin.yml 1m 29s
codeql.yml 1m 21s
dependabot-updates 1m 15s
pytest-ibis.yml 44s
scorecard.yml 43s
deploy-docs.yml 41s
update-graph 37s
check_docs_build.yml 30s
release-drafter.yml 8s

Note

This isn't an argument for/against btw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants