Currently, skims are read from OMX files, but based on our experience, storing skims in Parquet format can reduce file size by ~30%. Further, code for reading parquet format is extremely efficient as it is heavily optimized and widely used. This would significantly reduce storage requirements and improve efficiency.
We already support reading Parquet instead of CSV for the population synthesizer inputs, which has proven very valuable for reducing the size of model inputs. Extending this capability to skims would provide similar benefits.
Proposed enhancement:
Add support for reading skims directly from Parquet format.
Maintain backward compatibility with existing OMX inputs.
Background:
The original request and some discussion are available here.
Matrix data is currently handled as "openmatrix" or OMX format, which is documented here.
Instructions:
The new code should accept skims input data in Parquet format, with the first two columns giving origin and destination (or row and column labels, respectively), and subsequent columns giving named matrix array tables, named in exactly the same way as they would be with OMX files, including double underscore delimited time periods as appropriate. Some users report that the matrix files are always "dense" and ordered, but the code should NOT assume the matrix is dense. It should check the values of the origin and destination columns to determine whether the input is dense, and also if it is row-major or column major, adjusting the read process as needed. If the input parquet file is found to be dense and in row major or column major format, it should use an optimized pathway to read the data into memory. However, the code should accept sparse format parquet inputs as well. If dense data is found but the inputs are not appropriately sorted such that the data column is not in either row major or column major formats, an error should be raised. Sparse data should be allowed to be either sorted or unsorted.
The code should accept matrix data spread over multiple parquet files, similar to how the current code accepts data in multiple OMX files. It should NOT assume that all files have origins and destinations in exactly the same order -- each file should be checked to make sure of data formatting.
The code should auto-detect whether the input files are OMX or parquet format, and not rely on the user to specify.
Begin work by making a plan, then execute the plan. Ask questions as needed in this issue or in the PR that you create.
Currently, skims are read from OMX files, but based on our experience, storing skims in Parquet format can reduce file size by ~30%. Further, code for reading parquet format is extremely efficient as it is heavily optimized and widely used. This would significantly reduce storage requirements and improve efficiency.
We already support reading Parquet instead of CSV for the population synthesizer inputs, which has proven very valuable for reducing the size of model inputs. Extending this capability to skims would provide similar benefits.
Proposed enhancement:
Add support for reading skims directly from Parquet format.
Maintain backward compatibility with existing OMX inputs.
Background:
The original request and some discussion are available here.
Matrix data is currently handled as "openmatrix" or OMX format, which is documented here.
Instructions:
The new code should accept skims input data in Parquet format, with the first two columns giving origin and destination (or row and column labels, respectively), and subsequent columns giving named matrix array tables, named in exactly the same way as they would be with OMX files, including double underscore delimited time periods as appropriate. Some users report that the matrix files are always "dense" and ordered, but the code should NOT assume the matrix is dense. It should check the values of the origin and destination columns to determine whether the input is dense, and also if it is row-major or column major, adjusting the read process as needed. If the input parquet file is found to be dense and in row major or column major format, it should use an optimized pathway to read the data into memory. However, the code should accept sparse format parquet inputs as well. If dense data is found but the inputs are not appropriately sorted such that the data column is not in either row major or column major formats, an error should be raised. Sparse data should be allowed to be either sorted or unsorted.
The code should accept matrix data spread over multiple parquet files, similar to how the current code accepts data in multiple OMX files. It should NOT assume that all files have origins and destinations in exactly the same order -- each file should be checked to make sure of data formatting.
The code should auto-detect whether the input files are OMX or parquet format, and not rely on the user to specify.
Begin work by making a plan, then execute the plan. Ask questions as needed in this issue or in the PR that you create.