Problem
Commit a281c0dd (replace Arrow CSV/JSON IO with native reader framework) introduced a per-field std::string heap allocation + copy that didn't exist in the Arrow-based v0.1.3 code.
Root Cause
In append_csv_field_to_builder() (src/storages/loader/loader_utils.cc:234-266), every CSV field goes through:
auto token = field.get<std::string>(); // ← heap alloc + copy
builder->push_back_elem(
parse_value_by_type(token, ...)); // ← re-parse string to target typeFor numeric types (int64, double, etc.), the flow is:
CSV bytes → std::string (heap alloc) → string-to-type parse → Value → push_back_elem (virtual)
v0.1.3 comparison
Arrow's CSV reader performed type conversion during parsing, directly from the byte buffer to typed Arrow arrays (e.g., using std::from_chars on the raw bytes). No intermediate std::string was allocated.
For SF1000 scale, the total number of CSV fields is in the hundreds of millions. A heap allocation + copy per field is very expensive.
Impact
Estimated ~2000s of the ~6461s total regression on SF1000.
Proposed Fix
Options:
- Use
field.get<csv::string_view>() instead of field.get<std::string>() to avoid heap allocation, then parse directly from the string_view.
- Add a
push_back_opt(const T& val) path to ValueColumnBuilder for typed direct insertion, bypassing the Value intermediate and the push_back_elem virtual dispatch.
- For numeric types, parse directly from the underlying
csv::CSVField's buffer without going through std::string.
Benchmark
v0.1.3: 15166.20s
main (c8009e4): 21627.64s (+6461s regression)
main (b0905a1): 19273.97s (+4108s remaining)
References
Problem
Commit
a281c0dd(replace Arrow CSV/JSON IO with native reader framework) introduced a per-fieldstd::stringheap allocation + copy that didn't exist in the Arrow-based v0.1.3 code.Root Cause
In
append_csv_field_to_builder()(src/storages/loader/loader_utils.cc:234-266), every CSV field goes through:For numeric types (int64, double, etc.), the flow is:
v0.1.3 comparison
Arrow's CSV reader performed type conversion during parsing, directly from the byte buffer to typed Arrow arrays (e.g., using
std::from_charson the raw bytes). No intermediatestd::stringwas allocated.For SF1000 scale, the total number of CSV fields is in the hundreds of millions. A heap allocation + copy per field is very expensive.
Impact
Estimated ~2000s of the ~6461s total regression on SF1000.
Proposed Fix
Options:
field.get<csv::string_view>()instead offield.get<std::string>()to avoid heap allocation, then parse directly from the string_view.push_back_opt(const T& val)path toValueColumnBuilderfor typed direct insertion, bypassing theValueintermediate and thepush_back_elemvirtual dispatch.csv::CSVField's buffer without going throughstd::string.Benchmark
References
a281c0dd1d467cc2ca9a584dd90fad9ecfd34242src/storages/loader/loader_utils.cc