fix: ensure csv.parse generates a single shared array per file+options

[oleiade]

What

The csv.parse function was creating a new SharedArray with a random name (based on time.Now().Nanosecond()) for each VU, causing memory usage to scale linearly with VU count instead of sharing data across VUs. For a 40MB CSV file with 100 VUs, this resulted in ~6.5GB memory usage instead of the expected ~200MB.

This commit fixes the issue by:

1. Replacing random SharedArray name generation with deterministic hash-based naming. The name is now derived from the file path and parser options (delimiter, skipFirstLine, fromLine, toLine, asObjects) using SHA256, ensuring the same file+options combination always produces the same SharedArray name.
2. Refactoring the internal sharedArrays structure to use a slot-based system with sync.Once guarantees. This ensures the underlying data is initialized exactly once per unique name, even when multiple VUs call csv.parse concurrently.
3. Adding loadOrStore method that uses double-checked locking to minimize contention while ensuring thread-safe single initialization of shared arrays.

Closes #5493

Demo

Given a 40MB csv file, and the following script:

import { open } from 'k6/experimental/fs'
import csv from 'k6/experimental/csv'
import { scenario } from 'k6/execution'

export const options = {
    vus: 20,
	duration: '10m',
}

// Open the csv file, and parse it ahead of time.
const file = await open('data.csv');
// The `csv.parse` function consumes the entire file at once, and returns
// the parsed records as a SharedArray object.
const csvRecords = await csv.parse(file, { delimiter: ',' })

export default async function() {
	// The csvRecords a SharedArray. Each element is a record from the CSV file, represented as an array
	// where each element is a field from the CSV record.
	//
	// Thus, `csvRecords[scenario.iterationInTest]` will give us the record for the current iteration.
	console.log(csvRecords[scenario.iterationInTest])
}

Before this PR

After this PR

Checklist

• I have performed a self-review of my code.
• I have commented on my code, particularly in hard-to-understand areas.
• I have added tests for my changes.
• I have run linter and tests locally (make check) and all pass.

Checklist: Documentation (only for k6 maintainers and if relevant)

Please do not merge this PR until the following items are filled out.

• I have added the correct milestone and labels to the PR.
• I have updated the release notes: link
• I have updated or added an issue to the k6-documentation: grafana/k6-docs#NUMBER if applicable
• I have updated or added an issue to the TypeScript definitions: grafana/k6-DefinitelyTyped#NUMBER if applicable

Related PR(s)/Issue(s)

[joanlopez]

Great job @oleiade!

Note that none of the comments I left are blocking, but I'd really appreciate if we can "merge" the two "duplicated" methods 🙏🏻

[inancgumus]

I've moved this to the 1.6.0 milestone, as we're in the release process.

[oleiade]

I've made a pass on the underlying SharedArray code as per your concerns @codebien and found that it's already pretty well tests. However, I've added a bunch of further tests to the data package, in order to limit the risks of breaking anything with our internal changes in this PR. Hope that's satisfying 🙇🏻

[codebien]

Hey @oleiade,
thanks for let me know.

If I understand the changes correctly, the ones in the data module are orthogonal to the csv part but not really required for the fix of the specific related issue.

The data module is very sensible, so I'd appreciate it if you could split the pull requests by their domains to keep the things clean and easy enough to be reviewed in the future without too much effort. I mean to have one pull request for csv fix and one for the data module.

[oleiade]

Happy to split the PR.

However, most of the changes in the data module are dependencies for the CSV fix. The standalone parts (mostly increased test coverage in data_test.go and share_test.go to safeguard SharedArray) are isolated in my last commit.

I realize the data module is sensitive. But this was the best solution I could come up with at the time.

An alternative to the current approach I've been tinkering with might be to extract the 'shared memory' logic into a dedicated package that provides shared data structure used by csv, and that we could extend to further modules that might need that too in the future (fs), to tackle this need to initialize once, reuse multiple times that we tried to address here.. This would decouple it from the data module's specific implementation.

Plus, we could, if we so choose, migrate the data module to the new package down the road too. I think it could make sense, what do you folks think @codebien @joanlopez any other alternatives?

[oleiade]

Hey folks 👋🏻

Thanks a lot for the constructive feedback. I went back back to a less intrusive set of changes, leveraging the mutex as opposed to introduce a new sync.Once+slots based system.

I took the liberty to rewrite the history for clarity too, as I think it serves the review in this specific case.

Also removed the commit adding test coverage to SharedArray, and will open a separate PR 🙇🏻

[codebien]

LGTM 🚀 I've just one request.

[mstoykov]

LGTM apart from the if @codebien pointed out