vef: add stateful encoder to Field/Item#66
Merged
malone-at-work merged 6 commits intomainfrom Mar 3, 2026
Merged
Conversation
malone-at-work
changed the title
malone-at-work
changed the title
malone-at-work
changed the title
malone-at-work
force-pushed
the
maloney/vdf/2
branch
3 times, most recently
from
d1f7997 to
b00a94c
Compare
Member
Author
|
I added another commit this morning, because for temp tables memory needs to be allocated on the TableShare's mem root not on the Table's. |
tomas-villagesql
approved these changes
Mar 2, 2026
Member
tomas-villagesql
left a comment
tomas-villagesql
left a comment
There was a problem hiding this comment.
approved, but would update one test as commented
| PREPARE stmt FROM 'SELECT id FROM t1 WHERE val = ? ORDER BY id'; | ||
|
|
||
| SET @p = '(1.0,2.0)'; | ||
| EXECUTE stmt USING @p; |
Member
There was a problem hiding this comment.
I would make these without decimal, as the test would work equally well if everything was encoded as a string. I.e make the strings not match, but complex match
Member
Author
There was a problem hiding this comment.
This is the same feedback I always give, but missed in my own PR, thanks!
malone-at-work
force-pushed
the
maloney/vdf/2
branch
from
33807da to
228a04d
Compare
|
|
||
| private: | ||
| const villagesql::TypeContext *custom_type{nullptr}; | ||
| villagesql::TypeEncoder *type_encoder_{nullptr}; |
Member
There was a problem hiding this comment.
This is unfortunate, because we will also need a decoder, and maybe other pointers. It would be nice to put them into a container, or ideally with the TypeContext, but I think that is a much larger change. Should we put a TODO here (villagesql, not beta).
Member
Author
There was a problem hiding this comment.
Yes, let's clean up in a future PR. It cannot go in the TypeContext, but maybe it can be packaged with it.
malone-at-work
force-pushed
the
maloney/vdf/2
branch
from
228a04d to
79fa736
Compare
TypeEncoder ia a per-Field or Item object that holds encoding state across rows. Previously, every encode call allocated a new output buffer from mem_root; now a single scratch buffer (sized to persisted_length) is allocated once and reused for the lifetime of the Field's TABLE or the Item's query execution. For VDF-based encode ops, the VDF call structures (vef_invalue_t, vef_vdf_args_t, vef_vdf_result_t) are pre-filled at construction time, so only the per-row fields (input pointer/length, result type) are touched on each call. A rarely-used overflow path handles the case where VDF output exceeds the pre-allocated buffer size. TypeEncoder is lazily created via GetTypeEncoderFor and cached on the Field or Item. Item::cleanup() nulls the pointer between prepared-statement re-executions so a fresh encoder is allocated on the new thd->mem_root next execution. The public encode API in util.h is simplified: callers no longer pass MEM_ROOT, field_name, or TypeContext — the encoder resolves these from the Field or Item itself.
malone-at-work
force-pushed
the
maloney/vdf/2
branch
from
79fa736 to
125ea81
Compare
|
|
||
| private: | ||
| const villagesql::TypeContext *custom_type{nullptr}; | ||
| villagesql::TypeEncoder *type_encoder_{nullptr}; |
malone-at-work
force-pushed
the
maloney/vdf/2
branch
from
9f6121f to
a40c233
Compare
villagestevers
approved these changes
Mar 3, 2026
|
|
||
| private: | ||
| const villagesql::TypeContext *custom_type{nullptr}; | ||
| villagesql::TypeEncoder *type_encoder_{nullptr}; |
malone-at-work
force-pushed
the
maloney/vdf/2
branch
from
a40c233 to
634846d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TypeEncoder is a per-Field or Item object that holds encoding state across rows. Previously, every encode call allocated a new output buffer from mem_root; now a single scratch buffer (sized to persisted_length) is allocated once and reused for the lifetime of the Field's TABLE or the Item's query execution.
For VDF-based encode ops, the VDF call structures (vef_invalue_t, vef_vdf_args_t, vef_vdf_result_t) are pre-filled at construction time, so only the per-row fields (input pointer/length, result type) are touched on each call. A rarely-used overflow path handles the case where VDF output exceeds the pre-allocated buffer size.
TypeEncoder is lazily created via GetTypeEncoderFor and cached on the Field or Item. Item::cleanup() nulls the pointer between prepared-statement re-executions so a fresh encoder is allocated on the new thd->mem_root next execution.
The public encode API in util.h is simplified: callers no longer pass MEM_ROOT, field_name, or TypeContext — the encoder resolves these from the Field or Item itself.