Implement statistics recomputation with updates #2640
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2640 +/- ##
==========================================
+ Coverage 91.52% 91.54% +0.02%
==========================================
Files 479 480 +1
Lines 41177 41259 +82
Branches 5474 5483 +9
==========================================
+ Hits 37688 37772 +84
+ Misses 1910 1909 -1
+ Partials 1579 1578 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
joka921
left a comment
joka921
left a comment
There was a problem hiding this comment.
Mostly small comments, but I have a serious question about blank nodes.
| if (id.getDatatype() == Datatype::BlankNodeIndex) { | ||
| nextBlankNode = | ||
| std::max(nextBlankNode, id.getBlankNodeIndex().get() + 1); |
There was a problem hiding this comment.
I think this doesn't work if you in the located triples have randomly allocated blank nodes, as you will have a really large maximum. There is a precondition, that those blank nodes from the main index are dense. Let's discuss this subtlety.
| // queries the statistics which are never updated. Consider calling | ||
| // `IndexImpl::recomputeStatistics` and storing the result somewhere in this | ||
| // case. It also doesn't return the correct result for internal | ||
| // permutations. |
There was a problem hiding this comment.
Don't we have functions in the CompressedRelation that correctly computes distinct bla (currently i think col1, but maybe we can extend this? And then we can only use this expensive computation if there are any updates at all. At least extend the comment.
There was a problem hiding this comment.
Yes, and these are also used by group by (but obviously they take longer to compute than just reading the stats)
Overview
Conformance check passed ✅No test result changes. |
|
joka921
left a comment
joka921
left a comment
There was a problem hiding this comment.
Only a very small additional comment:)
2 participants
This is a preparation PR for #2408. It implements a function that allows to recompute statistics.
This is currently not used to correct outdated information (though ideally it should be), but that is a potential option for the future.