feat(docs): rough draft v2 ideas

[nyurik]

Work in progress, in part based on #1201

Some areas are a bit rough and need to be cleaned up

TBD

• shared presence with shared dictionaries - referencing a presence column from inside a shared dict is troublesome because it requires some complex searching - i.e. it would have to be a resolved column index rather than clustered column index (mix of scalars and shared dicts). Perhaps a better path is to define a new unnamed shared presence column type - and reference that.

[github-actions]

Performance Comparison main → v2

Total Elapsed Time: 5.16s → 5.08s (-1.5%)
CPU Baseline: 81.23µs → 80.94µs (-0.4%)
Benchmark ID: mlt-convert

timing - Function execution time metrics.

+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| Function                             | Calls                        | Avg                            | P95                            | Total                          | % Total                      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode                    | 11544 → 11292 (-2.2%)        | 949.40µs → 954.80µs (+0.6%)    | 4.11ms → 4.19ms (+1.9%)        | 10.96s → 10.78s (-1.6%)        | 212.42% → 212.11% (-0.1%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode_into               | 46179 → 45173 (-2.2%)        | 166.18µs → 168.73µs (+1.5%)    | 555.01µs → 559.62µs (+0.8%)    | 7.67s → 7.62s (-0.7%)          | 148.73% → 149.95% (+0.8%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| writer::with                         | 3240194 → 3167491 (-2.2%)    | 1.85µs → 1.86µs (+0.5%)        | 4.44µs → 4.47µs (+0.7%)        | 5.99s → 5.91s (-1.3%)          | 116.05% → 116.21% (+0.1%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_to                     | 46182 → 45174 (-2.2%)        | 101.30µs → 103.38µs (+2.1%)    | 393.47µs → 398.85µs (+1.4%)    | 4.68s → 4.67s (-0.2%)          | 90.67% → 91.88% (+1.3%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| physical::encode_u32s                | 2931637 → 2866058 (-2.2%)    | 988.00ns → 995.00ns (+0.7%)    | 321.00ns → 330.00ns (+2.8%)    | 2.90s → 2.85s (-1.7%)          | 56.15% → 56.13% (-0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_properties             | 46179 → 45173 (-2.2%)        | 60.93µs → 61.46µs (+0.9%)      | 293.12µs → 293.89µs (+0.3%)    | 2.81s → 2.78s (-1.1%)          | 54.53% → 54.62% (+0.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_prop                   | 167650 → 163955 (-2.2%)      | 16.55µs → 16.70µs (+0.9%)      | 32.70µs → 32.83µs (+0.4%)      | 2.77s → 2.74s (-1.1%)          | 53.77% → 53.86% (+0.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| tile::from_tile                      | 46183 → 45177 (-2.2%)        | 31.07µs → 30.86µs (-0.7%)      | 135.55µs → 130.88µs (-3.4%)    | 1.43s → 1.39s (-2.8%)          | 27.81% → 27.43% (-1.4%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::write_shared_dict       | 31712 → 30956 (-2.4%)        | 41.74µs → 41.76µs (+0.0%)      | 111.17µs → 112.89µs (+1.5%)    | 1.32s → 1.29s (-2.3%)          | 25.65% → 25.43% (-0.9%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::write_str_col               | 84211 → 82334 (-2.2%)        | 15.21µs → 15.37µs (+1.1%)      | 22.50µs → 22.72µs (+1.0%)      | 1.28s → 1.27s (-0.8%)          | 24.82% → 24.90% (+0.3%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::group_string_properties | 11548 → 11296 (-2.2%)        | 115.64µs → 110.43µs (-4.5%)    | 502.01µs → 478.98µs (-4.6%)    | 1.34s → 1.25s (-6.7%)          | 25.88% → 24.54% (-5.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::select_vertex_strategy       | 46182 → 45174 (-2.2%)        | 10.96µs → 11.56µs (+5.5%)      | 41.95µs → 41.09µs (-2.1%)      | 506.38ms → 522.16ms (+3.1%)    | 9.81% → 10.27% (+4.7%)       |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::write_str_plain             | 84212 → 82334 (-2.2%)        | 4.59µs → 4.92µs (+7.2%)        | 8.64µs → 8.72µs (+0.9%)        | 386.40ms → 405.47ms (+4.9%)    | 7.49% → 7.98% (+6.5%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| sort::sort                           | 46183 → 45177 (-2.2%)        | 8.25µs → 7.80µs (-5.5%)        | 37.15µs → 31.18µs (-16.1%)     | 381.00ms → 352.30ms (-7.5%)    | 7.38% → 6.93% (-6.1%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| logical::encode_u32s                 | 2832382 → 2768912 (-2.2%)    | 83.00ns → 79.00ns (-4.8%)      | 230.00ns → 221.00ns (-3.9%)    | 237.68ms → 220.77ms (-7.1%)    | 4.61% → 4.34% (-5.9%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| Function                             | Calls                        | Avg                            | P95                            | Total                           | % Total                    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| optimizer::encode                    | 11544 → 11292 (-2.2%)        | 187.0 KB → 186.5 KB (-0.3%)    | 385.7 KB → 385.7 KB (+0.0%)    | 2.1 GB → 2.0 GB (-4.8%)         | 36.56% → 36.84% (+0.8%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| tile::from_tile                      | 46183 → 45177 (-2.2%)        | 36.5 KB → 36.3 KB (-0.5%)      | 142.2 KB → 144.6 KB (+1.7%)    | 1.6 GB → 1.6 GB (+0.0%)         | 28.53% → 28.71% (+0.6%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| writer::with                         | 3240194 → 3167491 (-2.2%)    | 188 B → 187 B (-0.5%)          | 9.1 KB → 9.0 KB (-1.1%)        | 583.1 MB → 566.2 MB (-2.9%)     | 10.11% → 10.14% (+0.3%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| strings::fsst_try_train              | 27133 → 26515 (-2.3%)        | 14.7 KB → 13.5 KB (-8.2%)      | 3.4 MB → 3.4 MB (+0.0%)        | 389.9 MB → 349.9 MB (-10.3%)    | 6.76% → 6.27% (-7.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| physical::encode_u32s                | 2931637 → 2866058 (-2.2%)    | 123 B → 123 B (+0.0%)          | 34.5 KB → 34.1 KB (-1.2%)      | 346.4 MB → 336.6 MB (-2.8%)     | 6.01% → 6.03% (+0.3%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| shared_dict::group_string_properties | 11548 → 11296 (-2.2%)        | 24.9 KB → 24.7 KB (-0.8%)      | 142.6 KB → 142.1 KB (-0.4%)    | 281.4 MB → 272.7 MB (-3.1%)     | 4.88% → 4.88% (+0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| encode::write_to                     | 46182 → 45174 (-2.2%)        | 3.3 KB → 3.3 KB (+0.0%)        | 12.9 KB → 12.7 KB (-1.6%)      | 148.2 MB → 144.1 MB (-2.8%)     | 2.57% → 2.58% (+0.4%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| shared_dict::write_shared_dict       | 31712 → 30956 (-2.4%)        | 3.3 KB → 3.2 KB (-3.0%)        | 12.1 KB → 11.8 KB (-2.5%)      | 101.3 MB → 96.5 MB (-4.7%)      | 1.76% → 1.73% (-1.7%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| strings::write_str_col               | 84211 → 82334 (-2.2%)        | 750 B → 745 B (-0.7%)          | 2.5 KB → 2.5 KB (+0.0%)        | 60.3 MB → 58.6 MB (-2.8%)       | 1.04% → 1.05% (+1.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| logical::encode_u32s                 | 2832382 → 2768912 (-2.2%)    | 10 B → 10 B (+0.0%)            | 224 B → 224 B (+0.0%)          | 29.1 MB → 28.3 MB (-2.7%)       | 0.50% → 0.51% (+2.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| strings::write_raw_str_data          | 199499 → 195040 (-2.2%)      | 117 B → 114 B (-2.6%)          | 18.9 KB → 18.0 KB (-4.8%)      | 22.4 MB → 21.3 MB (-4.9%)       | 0.39% → 0.38% (-2.6%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| encode::build_morton_dict            | 504 → 476 (-5.6%)            | 40.5 KB → 41.3 KB (+2.0%)      | 88.9 KB → 89.9 KB (+1.1%)      | 19.9 MB → 19.2 MB (-3.5%)       | 0.35% → 0.34% (-2.9%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| encode::write_id_value_stream        | 47777 → 46578 (-2.5%)        | 155 B → 154 B (-0.6%)          | 595 B → 595 B (+0.0%)          | 7.1 MB → 6.8 MB (-4.2%)         | 0.12% → 0.12% (+0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| strings::write_str_plain             | 84212 → 82334 (-2.2%)        | 61 B → 61 B (+0.0%)            | 256 B → 256 B (+0.0%)          | 5.0 MB → 4.9 MB (-2.0%)         | 0.09% → 0.09% (+0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+
| encode::write_prop                   | 167650 → 163955 (-2.2%)      | 24 B → 24 B (+0.0%)            | 144 B → 144 B (+0.0%)          | 3.9 MB → 3.8 MB (-2.6%)         | 0.07% → 0.07% (+0.0%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------+

Threads

No threads to compare

---

Generated with hotpath-rs

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 62.10%. Comparing base (34ae1ae) to head (0ed213f).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/mlt/src/convert.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1322      +/-   ##
==========================================
+ Coverage   58.76%   62.10%   +3.33%     
==========================================
  Files          83       83              
  Lines       13775    13779       +4     
==========================================
+ Hits         8095     8557     +462     
+ Misses       5680     5222     -458     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.