How to automatically split a large CSV file into smaller files?

[herbst010]

Hello everyone, I have a single .csv file with 8 million rows, and I'd like to split it into multiple .csv files of 1 million rows each, automatically saving them locally on .csv format. Is it possible to do this with a Python script on Visidata? If so, could you help me create it?

[saulpw]

There are better tools than VisiData to do this for csv, assuming none of the fields have newlines in them. Use the POSIX tools head and split.

[frosencrantz]

Miller can do this: https://miller.readthedocs.io/en/latest/reference-verbs/#split

[techinmay]

To automatically split a large CSV file into smaller files, you have a few options depending on your comfort level. If you prefer coding, you can use Python (with pandas) to divide the file into chunks based on row count. However, these methods may require some technical setup and can be tricky when handling headers, formatting, or very large datasets. For a simpler and fully automated approach, you can use a dedicated tool like the SysTools CSV Splitter. It allows you to split large CSV files in just a few clicks-no coding required. You can choose to divide files by row count, file size, column value, or date, depending on your needs. The tool also ensures that data integrity and formatting remain intact, making it especially useful for handling large or complex CSV files efficiently.