What types of variables can Luau scripts create using qsv luau?

[derekmahar]

Can qsv lua or qsv luajit scripts

• Create list or array variables?
• Create map, dictionary, or associative array variables? (I'm not familiar with Lua, so I don't know whether Lua supports variables of these types or even what it calls them.)
• Create variables which are not columns in the output CSV file?
• Refer to columns in a previous row of a CSV file, including those which the script calculates?

[jqnatividad]

@derekmahar , your question inspired me to expand the Lua functionality of qsv:

• replaced Lua with Luau. Luau is an expanded version of Lua extended by Roblox with type checking, sandboxing, additional language operators, and improved performance.
• added --prologue and --epilogue options. The prologue allows you to run a Luau/LuaJIT script before the CSV is processed, typically for initializing variables. The epilogue is ran after the CSV is processed, typically for doing complex aggregations.
• added special variables _idx and _rowcount. _idx contains the current row number while processing the CSV. _rowcount is set to zero while the CSV is being processed, and to the row count during the epilogue.

With these changes, going to your questions:

• Create list or array variables - by using the --prologue to initialize variables, including arrays.
• Create map, dictionary or associative array variables - by using Lua Tables, and perhaps, taking advantage of the _idx variable to store columns in a Table, using _idx as index.
• Create variables which are not columns in the output CSV file - by using the --prologue option to initialize additional variables, and perhaps, even the --epilogue to create additional variables to facilitate computing complex aggregations that you can capture from stderr.
• Refer to columns in a previous row of a CSV file, including those which the script calculates - by making judicious use of the prologue to declare arrays, into which you can store each column selectively as each row is processed by using _idx.

To see an example of these extended capabilities, check out this test:

https://github.com/jqnatividad/qsv/blob/c7aa6e656d9bd7fa04339b7a28c11c5121728878/tests/test_luau.rs#L165-L204

[derekmahar]

@jqnatividad, thank you for so quickly and cleverly implementing these features so that you could answer "yes" to all of my questions! I'm pleased and impressed that I inspired you to make qsv more closely resemble AWK for CSV files! (Have you ever considered creating an AWK-like tool similar to Hawk that uses Lua or Luau as the embedded programming language?)

I must spend some time learning more about Lua tables before I can try these new features.

[derekmahar]

@jqnatividad, how difficult was it to replace the Lau interpreter with Luau?

[jqnatividad]

@derekmahar , I'm glad you liked it!

TBH, I got tired of fighting python's distribution and GIL limitations with the py command, and was really pleasantly surprised myself to find how performant and problem-free lua is in contrast.

I'm humbled by the "AWK for CSV" comparison. As for creating an AWK-like tool using Lua/Luau, I have enough on my plate with qsv and my CKAN-related projects 😄 - but I'm always open to new idears to make data-wrangling/data management better.

And it took me about a day to pull together the --prologue and --epilogue functionality. Actually, switching to luau was a late decision as I was putting together the CI tests for prologue/epilogue, as the test I wrote that was informed by your question was not working as I hoped.

As I was troubleshooting the test, I found out about luau's += operator and if-then-else expression, and decided to switch after learning more about luau.

Switching from Lua to Luau was not that big an effort thanks in large part to the wonderful mlua crate.

[jqnatividad]

CKAN, as you can see from the different example sites, is more of a Data Management System (DMS) framework with which you can build all kinds of DMS solutions. It has a very active community with hundreds of community-built extensions. Dataverse is more of a turn-key system targeted to academia.

[pdurbin]

Yes, Dataverse is primary used in academia but non-profits, governments and a few companies use it as well. Yes, it's turn-key but extensible! 🚀

For a little context, I found qsv and this discussion from slides for a talk entitled "Powering curation tooling, including statistical disclosure risk analysis, with Dataverse" at https://barcelo.eventsair.com/QuickEventWebsitePortal/dcm2026/programadcm26/Agenda/AgendaItemDetail?id=2e3ee111-3d70-41d8-b495-0053e318e24c

[derekmahar]

Out of curiosity, can CKAN or Dataverse publish data sets in Parquet format?

[jqnatividad]

CKAN is format-agnostic, and can just publish about any file format.

FYI, qsv started as a side project when we were working with a hedge fund to catalog their disparate data assets and they wanted us to auto-FAIRify the metadata from their various data collections on a nightly basis.

Python tooling was just too slow, so that's how I found xsv and forked qsv from it to keep adding more features.

It now powers our 5th generation data-ingestion with Datapusher+, and we can certainly create parquet files if required using qsv.

You may also want to check out the qsv Cowork Plugin. With it, Claude Cowork becomes a Data Analyst Workbench, and it transparently converts files to parquet so it can process data more efficiently.

Here is the plugin in action - https://dathere.github.io/peoples-api-demos/NYC-Housing-Policy-SOD2026/

[pdurbin]

Same, Dataverse is format-agnostic. Please feel free to join the discussion about Parquet support in Dataverse! Here's the issue:

• incorporate parquet files? IQSS/dataverse#9897