[Bug]: using `--shard` with `--url` does not distribute tests evenly, contains duplicates

[matt-halliday]

Describe the bug

I'm seeing an issue where using --shard with --url is not testing all components, and between shards, many tests are duplicated/overlapped.

I've not deployed a reproduction because a volume of components is required, but I have included some very basic steps to reproduce against an already hosted Storybook deployment using --url.

This is how we're running in CI where we are encountering the issue currently - deploying a static storybook and running visual snapshot tests with playwright against it spread over shards.

The result is that we have far fewer snapshots than we should, due to shards running the same tests and missing some out.

It's random, and different on each run.

To Reproduce

npm create storybook@latest

Choose React

npm install @storybook/test-runner --save-dev
npx playwright install

Now baseline:

npx test-storybook -u --browsers chromium --url=https://drei.pmnd.rs

Results:

Test Suites: 112 passed, 112 total
Tests:       180 passed, 180 total
Snapshots:   0 total
Time:        19.126 s

Total tests 180

Now shard 1:

npx test-storybook -u --browsers chromium --url=https://drei.pmnd.rs --shard=1/2

Shard 1 Results:

Test Suites: 56 passed, 56 total
Tests:       80 passed, 80 total
Snapshots:   0 total
Time:        10.681 s

Now shard 2:

npx test-storybook -u --browsers chromium --url=https://drei.pmnd.rs --shard=2/2

Shard 2 Results:

Test Suites: 56 passed, 56 total
Tests:       88 passed, 88 total
Snapshots:   0 total
Time:        11.249 s

Total tests 168

Across the two shards are many duplicates - not all components have been tested.

System

Storybook Environment Info:

  System:
    OS: macOS 15.6
    CPU: (12) arm64 Apple M2 Pro
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.16.0 - ~/.nvm/versions/node/v22.16.0/bin/node
    Yarn: 1.22.22 - ~/.nvm/versions/node/v22.16.0/bin/yarn
    npm: 10.9.2 - ~/.nvm/versions/node/v22.16.0/bin/npm <----- active
  Browsers:
    Chrome: 139.0.7258.128
    Safari: 18.6
  npmPackages:
    @storybook/react-vite: ^9.1.2 => 9.1.2 
    @storybook/test-runner: ^0.23.0 => 0.23.0 
    eslint-plugin-storybook: ^9.1.2 => 9.1.2 
    storybook: ^9.1.2 => 9.1.2

Additional context

No response

[matt-halliday]

I think the issue is that the jest default sequencer hashes the path of the files to sort the tests, and because it's creating a temp directory here, that is different each time.

I might play with a custom sequencer in a PR?

[jtbandes]

Yep I'm seeing the same issue. Can confirm that the custom sequencer in #578 fixed the issue when I copied it into my project.