sdk: prerun+varargs by dbentley-vsql · Pull Request #254 · villagesql/villagesql-server

dbentley-vsql · 2026-04-22T19:41:05Z

It appears we accidentally blocked varargs functions. Even though we could set a prerun, the calls (or lack of) to .params would still be checked and throw an error.

malone-at-work

I am going to think about this a little more; I think there may be better ways to guide people to use this correctly

malone-at-work · 2026-04-22T19:49:48Z

Why are we making changes to this file - this is only for older extensions to keep compiling.

Reverted. Thanks.

malone-at-work · 2026-04-22T20:10:15Z

  }
+
+ private:
+  static void config_error__param_type_and_varargs_are_mutually_exclusive();


Is there a reason we can't use static_assert for these?

Hah; I had done that but reverted it because it added a template parameter. Now it's static_assert again.

malone-at-work · 2026-04-22T20:13:08Z

 // FuncBuilder
 // =============================================================================

+enum class ParamMode { kUndeclared, kFixed, kVarargs };


s/kUndeclared/kUnset/?

malone-at-work · 2026-04-22T20:32:17Z

  }
+
+ private:
+  static void config_error__param_type_and_varargs_are_mutually_exclusive();


This way of inducing an error worries me, can you add a test that this fails, especially with the following cmake option:

target_link_options(foo_ext PRIVATE -undefined dynamic_lookup)

It's static_assert now.

malone-at-work · 2026-04-22T20:36:32Z

  for (unsigned int i = 0; i < value_count; i++) {
    Item *arg_item = args[i];
-    vef_type_id param_type =
-        (i < sig->param_count) ? sig->params[i].id : VEF_TYPE_STRING;


Did I write this? Is this intentional that we marshall excess parameters to functions? I think we should probably make this an error (except in the varargs) case.

Yeah, I think it was untested and was handled by a separate error. Commented.

malone-at-work · 2026-04-22T20:38:52Z

+  if (args->arg_count == 0) {
+    result->type = VEF_RESULT_ERROR;
+    snprintf(result->error_msg, VEF_MAX_ERROR_LEN,
+             "ba_concat_all requires at least one argument");


Please test these errors.

done (extension_simple_type_usage.result)

malone-at-work · 2026-04-22T20:39:11Z

+    vef_type_id id = args->arg_types[i].id;
+    if (id != VEF_TYPE_CUSTOM && id != VEF_TYPE_STRING) {
+      result->type = VEF_RESULT_ERROR;
+      snprintf(result->error_msg, VEF_MAX_ERROR_LEN,


Done (see above)

malone-at-work · 2026-04-22T20:40:37Z

+                  .param(BYTEARRAY)
+                  .build())
+        .func(make_func<&ba_len>("ba_len").returns(INT).param().build())
+        .func(make_func<&ba_concat_all>("ba_concat_all")


what would you think of a make_varargs_func() that doesn't allow you to specify parameters?

Why put varargs in the func name when everything else is methods on the builder?

malone-at-work · 2026-05-18T11:46:02Z


    FuncWithMetadata meta{};
-    meta.f = &Wrapper<Func, NumParams>::invoke;
+    if constexpr (std::is_same_v<decltype(Func), vef_vdf_func_t>) {


I believe you were planning on fixing this to not expose the ABI types?

malone-at-work

Please remove raw ABI structs from the API.

Introduce typed C++ wrappers for the per-statement lifecycle hooks (prerun and postrun). Extension authors no longer touch raw ABI structs to write these hooks; only the typed signatures are accepted. Why --- Extension authors writing prerun/postrun previously had to: void my_prerun(vef_context_t *, vef_prerun_args_t *args, vef_prerun_result_t *result) { if (args->arg_count == 0) { result->type = VEF_RESULT_ERROR; snprintf(result->error_msg, VEF_MAX_ERROR_LEN, "..."); return; } result->result_buffer_size = N; result->user_data = new MyState{}; } Two problems: raw ABI in the user-facing API (against the SDK's direction; see #548) and bespoke error-reporting plumbing. After this PR: void my_prerun(vsql::PrerunArgs args, vsql::PrerunResult out) { if (args.size() == 0) { out.error("at least one argument required"); return; } out.request_buffer_size(N); out.set_user_data(new MyState{}); } void my_postrun(vsql::PostrunArgs args) { args.delete_state<MyState>(); } void my_vdf(MyState &state, IntResult out) { ... } State lifetime stays explicit in this PR: prerun stashes the pointer, postrun frees it. A follow-up will add auto-cleanup (see "Not in this PR" below). Where to start reviewing ------------------------ Read the layers in this order: 1. villagesql/sdk/include/villagesql/vsql/pre_post_run.h New public types: PrerunArgs, PrerunArgType, PrerunResult, PostrunArgs. This is what extension authors see. 2. villagesql/examples/vsql-simple/src/extension.cc The ba_call_index demo: set_user_data in prerun, mutable State& in the VDF, delete_state in postrun. 3. mysql-test/suite/villagesql/extension/t/extension_simple_type_usage.test What it looks like at the SQL surface. 4. villagesql/sdk/include/villagesql/extension.h Refreshed user-facing docs for the typed hook shape. The remaining file is SDK plumbing: 5. villagesql/sdk/include/villagesql/vsql/func_builder.h - .prerun<>() / .postrun<>() are typed-only; static_assert rejects raw vef_prerun_func_t / vef_postrun_func_t. - WrapperTypedState / WrapperVoidState dispatch on the first param of the VDF: State& / const State& / void* selects how user_data is forwarded. - typed_prerun_thunk / typed_postrun_thunk adapt the user's typed signature to the raw ABI shape the server invokes. Not in this PR -------------- - Auto-cleanup for SDK-allocated state. Requires an ABI side channel (e.g. a user_data_deleter slot on vef_prerun_result_t) so the SDK can register a destructor independent of the user_data pointer that extensions also use for raw/custom allocations. Marked TODO(villagesql-beta) in pre_post_run.h. - A `.state<T>()` builder that records State at the template level and routes typed signatures of the form `void(T&, ...)`. Will land with the auto-cleanup mechanism above. - Typed VarArgs/AnyArg wrappers for varargs VDFs (the other raw-ABI place in extension code, only reached via PR #254). - vef_postrun_result_t is empty in the ABI today, so there is no PostrunResult wrapper. Will add when the struct grows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

dbentley-vsql · 2026-05-21T15:55:07Z

Superseded by #563

Introduce typed C++ wrappers for the per-statement lifecycle hooks (prerun and postrun). Extension authors no longer touch raw ABI structs to write these hooks; only the typed signatures are accepted. Why --- Extension authors writing prerun/postrun previously had to: void my_prerun(vef_context_t *, vef_prerun_args_t *args, vef_prerun_result_t *result) { if (args->arg_count == 0) { result->type = VEF_RESULT_ERROR; snprintf(result->error_msg, VEF_MAX_ERROR_LEN, "..."); return; } result->result_buffer_size = N; result->user_data = new MyState{}; } Two problems: raw ABI in the user-facing API (against the SDK's direction; see #548) and bespoke error-reporting plumbing. After this PR: void my_prerun(vsql::PrerunArgs args, vsql::PrerunResult out) { if (args.size() == 0) { out.error("at least one argument required"); return; } out.request_buffer_size(N); out.set_user_data(new MyState{}); } void my_postrun(vsql::PostrunArgs args) { args.delete_state<MyState>(); } void my_vdf(MyState &state, IntResult out) { ... } State lifetime stays explicit in this PR: prerun stashes the pointer, postrun frees it. A follow-up will add auto-cleanup (see "Not in this PR" below). Where to start reviewing ------------------------ Read the layers in this order: 1. villagesql/sdk/include/villagesql/vsql/pre_post_run.h New public types: PrerunArgs, PrerunArgType, PrerunResult, PostrunArgs. This is what extension authors see. 2. villagesql/examples/vsql-simple/src/extension.cc The ba_call_index demo: set_user_data in prerun, mutable State& in the VDF, delete_state in postrun. 3. mysql-test/suite/villagesql/extension/t/extension_simple_type_usage.test What it looks like at the SQL surface. 4. villagesql/sdk/include/villagesql/extension.h Refreshed user-facing docs for the typed hook shape. The remaining file is SDK plumbing: 5. villagesql/sdk/include/villagesql/vsql/func_builder.h - .prerun<>() / .postrun<>() are typed-only; static_assert rejects raw vef_prerun_func_t / vef_postrun_func_t. - WrapperTypedState / WrapperVoidState dispatch on the first param of the VDF: State& / const State& / void* selects how user_data is forwarded. - typed_prerun_thunk / typed_postrun_thunk adapt the user's typed signature to the raw ABI shape the server invokes. Not in this PR -------------- - Auto-cleanup for SDK-allocated state. Requires an ABI side channel (e.g. a user_data_deleter slot on vef_prerun_result_t) so the SDK can register a destructor independent of the user_data pointer that extensions also use for raw/custom allocations. Marked TODO(villagesql-beta) in pre_post_run.h. - A `.state<T>()` builder that records State at the template level and routes typed signatures of the form `void(T&, ...)`. Will land with the auto-cleanup mechanism above. - Typed VarArgs/AnyArg wrappers for varargs VDFs (the other raw-ABI place in extension code, only reached via PR #254). - vef_postrun_result_t is empty in the ABI today, so there is no PostrunResult wrapper. Will add when the struct grows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Introduce typed C++ wrappers for the per-statement lifecycle hooks (prerun and postrun). Extension authors no longer touch raw ABI structs to write these hooks; only the typed signatures are accepted. Why --- Extension authors writing prerun/postrun previously had to: void my_prerun(vef_context_t *, vef_prerun_args_t *args, vef_prerun_result_t *result) { if (args->arg_count == 0) { result->type = VEF_RESULT_ERROR; snprintf(result->error_msg, VEF_MAX_ERROR_LEN, "..."); return; } result->result_buffer_size = N; result->user_data = new MyState{}; } Two problems: raw ABI in the user-facing API (against the SDK's direction; see #548) and bespoke error-reporting plumbing. After this PR: void my_prerun(vsql::PrerunArgs args, vsql::PrerunResult out) { if (args.size() == 0) { out.error("at least one argument required"); return; } out.request_buffer_size(N); out.set_user_data(new MyState{}); } void my_postrun(vsql::PostrunArgs args) { args.delete_state<MyState>(); } void my_vdf(MyState &state, IntResult out) { ... } State lifetime stays explicit in this PR: prerun stashes the pointer, postrun frees it. A follow-up will add auto-cleanup via .state<T>() (implementable without ABI changes; see TODO in pre_post_run.h). Where to start reviewing ------------------------ Read the layers in this order: 1. villagesql/sdk/include/villagesql/vsql/pre_post_run.h New public types: PrerunArgs, PrerunArgType, PrerunResult, PostrunArgs. This is what extension authors see. 2. villagesql/examples/vsql-simple/src/extension.cc The ba_call_index demo: set_user_data in prerun, mutable State& in the VDF, delete_state in postrun. 3. mysql-test/suite/villagesql/extension/t/extension_simple_type_usage.test What it looks like at the SQL surface. 4. villagesql/sdk/include/villagesql/extension.h Refreshed user-facing docs for the typed hook shape. The remaining file is SDK plumbing: 5. villagesql/sdk/include/villagesql/vsql/func_builder.h - .prerun<>() / .postrun<>() are typed-only; static_assert rejects raw vef_prerun_func_t / vef_postrun_func_t. - FuncBuilder gained a HasPrerun template flag; .prerun<>() flips it true, and build() static_asserts that void(State&,...) and void(void*,...) signatures require it. Registering a state-style VDF without a prerun is a compile error, not a runtime null deref. - WrapperTypedState / WrapperVoidState dispatch on the first param of the VDF: State& / const State& / void* selects how user_data is forwarded. - typed_prerun_thunk / typed_postrun_thunk adapt the user's typed signature to the raw ABI shape the server invokes. Not in this PR -------------- - A `.state<T>()` builder that records State at the template level and routes typed signatures of the form `void(T&, ...)`. The SDK would install both prerun and postrun thunks to manage the typed state's lifetime via the existing user_data slot — no ABI side channel needed. Will land as a follow-up. - Typed VarArgs/AnyArg wrappers for varargs VDFs (the other raw-ABI place in extension code, only reached via PR #254). - vef_postrun_result_t is empty in the ABI today, so there is no PostrunResult wrapper. Will add when the struct grows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

NB: this is analogous to #254 , but rebased and simplified sdk: typed varargs + required explicit arity Introduce typed C++ wrappers for varargs VDFs (vsql::VarArgs, vsql::AnyArg) and the matching .varargs() / zero-arity .param() builder methods, so an extension author can write a variadic SQL function without touching raw ABI types. Also tightens the v2 builder to require an explicit arity declaration before .build(). What changes for users Before, an extension author who wanted a varargs SQL function had to drop out of the typed SDK and write a raw-ABI body — the typed wrappers covered fixed-arity only: // Old: raw vef_vdf_func_t signature, manual invalue extraction void ba_concat_all(vef_context_t *ctx, vef_vdf_args_t *args, vef_vdf_result_t *result) { for (unsigned int i = 0; i < args->value_count; i++) { vef_invalue_t v = vsql::func_builder::get_invalue(ctx, args, i); if (v.is_null) { result->type = VEF_RESULT_NULL; return; } memcpy(result->str_buf + i * N, v.bin_value, N); } result->type = VEF_RESULT_VALUE; result->actual_len = args->value_count * N; } After: // New: typed VarArgs + AnyArg, no raw ABI in the body void ba_concat_all(vsql::VarArgs args, vsql::StringResult out) { auto dst = out.buffer(); size_t off = 0; for (auto a : args) { if (a.is_null()) { out.set_null(); return; } auto bytes = a.as_custom(); memcpy(dst.data() + off, bytes.data(), bytes.size()); off += bytes.size(); } out.set_length(off); } Registration picks up .varargs() (and zero-arity .param()): .func(make_func<&ba_len>("ba_len") .returns(INT) .param() // explicit zero-arity .build()) .func(make_func<&ba_concat_all>("ba_concat_all") .returns(STRING) .varargs() .prerun<&ba_concat_all_prerun>() .build()) The prerun hook is still raw ABI in this PR; the typed variant lands with #551 (typed vsql::PrerunArgs / PrerunResult), and ba_concat_all_prerun will convert to typed form when that merges. Required-arity change The v2 builder now rejects make_func<&fn>("name").returns(...).build() without an arity call. You must pick exactly one of: .param(TYPE) (one or more, for fixed-arity typed args) .param() (zero-arity) .varargs() Eight in-tree test-extensions had implicit zero-arity functions and pick up an explicit .param() in this PR. The v1 builder (villagesql::func_builder, raw-ABI) is unchanged — that cow has left the barn.

Introduce typed C++ wrappers for the per-statement lifecycle hooks (prerun and postrun). Extension authors no longer touch raw ABI structs to write these hooks; only the typed signatures are accepted. Why --- Extension authors writing prerun/postrun previously had to: void my_prerun(vef_context_t *, vef_prerun_args_t *args, vef_prerun_result_t *result) { if (args->arg_count == 0) { result->type = VEF_RESULT_ERROR; snprintf(result->error_msg, VEF_MAX_ERROR_LEN, "..."); return; } result->result_buffer_size = N; result->user_data = new MyState{}; } Two problems: raw ABI in the user-facing API (against the SDK's direction; see #548) and bespoke error-reporting plumbing. After this PR: void my_prerun(vsql::PrerunArgs args, vsql::PrerunResult out) { if (args.size() == 0) { out.error("at least one argument required"); return; } out.request_buffer_size(N); out.set_user_data(new MyState{}); } void my_postrun(vsql::PostrunArgs args) { args.delete_state<MyState>(); } void my_vdf(MyState &state, IntResult out) { ... } State lifetime stays explicit in this PR: prerun stashes the pointer, postrun frees it. A follow-up will add auto-cleanup via .state<T>() (implementable without ABI changes; see TODO in pre_post_run.h). Where to start reviewing ------------------------ Read the layers in this order: 1. villagesql/sdk/include/villagesql/vsql/pre_post_run.h New public types: PrerunArgs, PrerunArgType, PrerunResult, PostrunArgs. This is what extension authors see. 2. villagesql/examples/vsql-simple/src/extension.cc The ba_call_index demo: set_user_data in prerun, mutable State& in the VDF, delete_state in postrun. 3. mysql-test/suite/villagesql/extension/t/extension_simple_type_usage.test What it looks like at the SQL surface. 4. villagesql/sdk/include/villagesql/extension.h Refreshed user-facing docs for the typed hook shape. The remaining file is SDK plumbing: 5. villagesql/sdk/include/villagesql/vsql/func_builder.h - .prerun<>() / .postrun<>() are typed-only; static_assert rejects raw vef_prerun_func_t / vef_postrun_func_t. - FuncBuilder gained a HasPrerun template flag; .prerun<>() flips it true, and build() static_asserts that void(State&,...) and void(void*,...) signatures require it. Registering a state-style VDF without a prerun is a compile error, not a runtime null deref. - WrapperTypedState / WrapperVoidState dispatch on the first param of the VDF: State& / const State& / void* selects how user_data is forwarded. - typed_prerun_thunk / typed_postrun_thunk adapt the user's typed signature to the raw ABI shape the server invokes. Not in this PR -------------- - A `.state<T>()` builder that records State at the template level and routes typed signatures of the form `void(T&, ...)`. The SDK would install both prerun and postrun thunks to manage the typed state's lifetime via the existing user_data slot — no ABI side channel needed. Will land as a follow-up. - Typed VarArgs/AnyArg wrappers for varargs VDFs (the other raw-ABI place in extension code, only reached via PR #254). - vef_postrun_result_t is empty in the ABI today, so there is no PostrunResult wrapper. Will add when the struct grows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* sdk: typed prerun/postrun hooks Introduce typed C++ wrappers for the per-statement lifecycle hooks (prerun and postrun). Extension authors no longer touch raw ABI structs to write these hooks; only the typed signatures are accepted. Why --- Extension authors writing prerun/postrun previously had to: void my_prerun(vef_context_t *, vef_prerun_args_t *args, vef_prerun_result_t *result) { if (args->arg_count == 0) { result->type = VEF_RESULT_ERROR; snprintf(result->error_msg, VEF_MAX_ERROR_LEN, "..."); return; } result->result_buffer_size = N; result->user_data = new MyState{}; } Two problems: raw ABI in the user-facing API (against the SDK's direction; see #548) and bespoke error-reporting plumbing. After this PR: void my_prerun(vsql::PrerunArgs args, vsql::PrerunResult out) { if (args.size() == 0) { out.error("at least one argument required"); return; } out.request_buffer_size(N); out.set_user_data(new MyState{}); } void my_postrun(vsql::PostrunArgs args) { args.delete_state<MyState>(); } void my_vdf(MyState &state, IntResult out) { ... } State lifetime stays explicit in this PR: prerun stashes the pointer, postrun frees it. A follow-up will add auto-cleanup via .state<T>() (implementable without ABI changes; see TODO in pre_post_run.h). Where to start reviewing ------------------------ Read the layers in this order: 1. villagesql/sdk/include/villagesql/vsql/pre_post_run.h New public types: PrerunArgs, PrerunArgType, PrerunResult, PostrunArgs. This is what extension authors see. 2. villagesql/examples/vsql-simple/src/extension.cc The ba_call_index demo: set_user_data in prerun, mutable State& in the VDF, delete_state in postrun. 3. mysql-test/suite/villagesql/extension/t/extension_simple_type_usage.test What it looks like at the SQL surface. 4. villagesql/sdk/include/villagesql/extension.h Refreshed user-facing docs for the typed hook shape. The remaining file is SDK plumbing: 5. villagesql/sdk/include/villagesql/vsql/func_builder.h - .prerun<>() / .postrun<>() are typed-only; static_assert rejects raw vef_prerun_func_t / vef_postrun_func_t. - FuncBuilder gained a HasPrerun template flag; .prerun<>() flips it true, and build() static_asserts that void(State&,...) and void(void*,...) signatures require it. Registering a state-style VDF without a prerun is a compile error, not a runtime null deref. - WrapperTypedState / WrapperVoidState dispatch on the first param of the VDF: State& / const State& / void* selects how user_data is forwarded. - typed_prerun_thunk / typed_postrun_thunk adapt the user's typed signature to the raw ABI shape the server invokes. Not in this PR -------------- - A `.state<T>()` builder that records State at the template level and routes typed signatures of the form `void(T&, ...)`. The SDK would install both prerun and postrun thunks to manage the typed state's lifetime via the existing user_data slot — no ABI side channel needed. Will land as a follow-up. - Typed VarArgs/AnyArg wrappers for varargs VDFs (the other raw-ABI place in extension code, only reached via PR #254). - vef_postrun_result_t is empty in the ABI today, so there is no PostrunResult wrapper. Will add when the struct grows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * demo: state_writeback test showing void* user_data writeback gap * fix previous VDF * vef: convert ba_concat_all_prerun to typed PrerunArgs/PrerunResult ---------

dbentley-vsql requested a review from malone-at-work April 22, 2026 19:41

malone-at-work requested changes Apr 22, 2026

View reviewed changes

dbentley-vsql force-pushed the dbentley/sdk_varargs branch from 5097983 to 832caae Compare May 15, 2026 14:36

sdk: prerun+varargs

74833e7

dbentley-vsql force-pushed the dbentley/sdk_varargs branch from 832caae to 74833e7 Compare May 15, 2026 14:45

malone-at-work self-requested a review May 15, 2026 16:38

malone-at-work reviewed May 18, 2026

View reviewed changes

malone-at-work requested changes May 18, 2026

View reviewed changes

This was referenced May 20, 2026

[Server]: C++-ify Prerun/postrun #541

Closed

sdk: typed prerun/postrun #551

Merged

dbentley-vsql mentioned this pull request May 21, 2026

sdk: typed varargs #563

Merged

dbentley-vsql closed this May 21, 2026

Uh oh!

Conversation

dbentley-vsql commented Apr 22, 2026

Uh oh!

malone-at-work left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malone-at-work left a comment

Choose a reason for hiding this comment

Uh oh!

dbentley-vsql commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants