csv.parse do not share the array between VUs #5493
Description
Brief summary
I need to load a CSV to have some random data for my VUs. My CSV is around ~40MB large.
When using csv.parse it seems to create one SharedArray per VU since my RAM usage scales linearly with the VU number.
k6 version
1.4.0
OS
Ubuntu 22.04
Docker version and image (if applicable)
No response
Steps to reproduce the problem
- Generate a big-ish CSV file (40MB
- Run k6 with this file
import {open} from 'k6/experimental/fs';
import csv from 'k6/experimental/csv';
import {sleep} from "k6";
const csvFile = 'users.csv';
const users = await csv.parse(await open(csvFile), {delimiter: ';', asObjects: true});
export const options = {
vus: 100,
duration: '10m',
};
export default function () {
const user = users[Math.floor(Math.random() * users.length)];
sleep(10)
}
- compare the result with less VUs
Expected behaviour
The memory footprint should stay around the same regardless of the number of VUs
Actual behaviour
The memory footprint explodes (in my case around 6.5Go used for 100VU and scales with the number of VU)
Start of analysis
My guess but I'm not a go expert is that we create the shared array with a "random" name each time and so it's loading the CSV for each VU instead of being a real SharedArray. Code in question:
Maybe we need to be able to pass a name to the csv.parse method
Used workaround
For reference I did the same thing with the same file and the "old" way of loading a csv with papaparse and the other "open" function as a workaround. The parsing is obviously slower since it's done in javascript but the memory footprint is around 0.2Go instead which is way better.
"Old" way script
import {sleep} from "k6";
import papaparse from "./papaparse.js"
import {SharedArray} from "k6/data";
const csvFile = 'users.csv';
const users = new SharedArray('users', function () {
return papaparse.parse(open(csvFile), {header: true, delimiter: ';'}).data;
});
export const options = {
vus: 100,
duration: '10m',
};
export default function () {
const user = users[Math.floor(Math.random() * users.length)];
sleep(10)
}