calculate_payload_size causes a stack overflow in debug mode with very small stack

[romanholidaypancakes]

The function calculate_payload_size may call itself recursively, which can lead to a stack overflow and trigger a crash.

Related: #1081

[poor-circle]

I dont understand how it could happen. Are you trying serialize a huge link list?

[poor-circle]

Could you show me your overflow stack?

[poor-circle]

struct node {
  std::unique_ptr<node> n;
  int value;
};

Are you trying to serialize a huge link list like this (node count >= 10^4) ?

[romanholidaypancakes]

The entire MyAppProto struct is nearly 4,000 bytes in size and contains 48 struct members.
Below is the crash stack; I anonymized some struct names using ....

nt!KernelStackOverflow+0x2bd
kapp!struct_pack::detail::calculate_one_size<unsigned __int64,0>+0xf [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 41] 
kapp!struct_pack::detail::calculate_payload_size<0,unsigned __int64,unsigned __int64,unsigned __int64,unsigned int,unsigned __int64,unsigned __int64,unsigned int,unsigned __int64,unsigned int,unsigned __int64,unsigned int,unsigned __int64,unsigned int,unsigned int,std::basic_string<char,std::char_traits<char>,std::allocator<char> >, ....,bool>+0x32 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 188] 
kapp!`struct_pack::detail::calculate_one_size<....>+0x10a [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 175] 
kapp!ylt::reflection::internal::object_tuple_view_helper<...'::`5'::<lambda_1> >+0x18e [deps\yalantinglibs\include\ylt\reflection\internal\generate\member_macro.hpp @ 54] 
kapp!ylt::reflection::internal::tuple_view<17,...'::`5'::<lambda_1> >+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 97] 
kapp!ylt::reflection::visit_members<...>'::`5'::<lambda_1>,17>+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 134] 
kapp!struct_pack::detail::visit_members<...>'::`5'::<lambda_1> >+0x32 [deps\yalantinglibs\include\ylt\struct_pack\reflection.hpp @ 963] 
kapp!struct_pack::detail::calculate_one_size<...,0>+0x49 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 172] 
kapp!struct_pack::detail::calculate_payload_size<0,....>+0x4e [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 188] 
kapp!`struct_pack::detail::calculate_one_size<....,0>'::`5'::<lambda_1>::operator()<... const &>+0x37 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 175] 
kapp!ylt::reflection::internal::object_tuple_view_helper<... &,2>::tuple_view<`struct_pack::detail::calculate_one_size<...::Add,0>'::`5'::<lambda_1> >+0x44 [deps\yalantinglibs\include\ylt\reflection\internal\generate\member_macro.hpp @ 21] 
kapp!ylt::reflection::internal::tuple_view<2,...,0>'::`5'::<lambda_1> >+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 97] 
kapp!ylt::reflection::visit_members<...,0>'::`5'::<lambda_1>,2>+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 134] 
kapp!struct_pack::detail::visit_members<...,0>'::`5'::<lambda_1> >+0x32 [deps\yalantinglibs\include\ylt\struct_pack\reflection.hpp @ 963] 
kapp!struct_pack::detail::calculate_one_size<...>+0x49 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 172] 
kapp!struct_pack::detail::calculate_payload_size<0,...n>+0x32 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 188] 
kapp!`struct_pack::detail::calculate_one_size<... &>+0x52 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 175] 
kapp!ylt::reflection::internal::object_tuple_view_helper<... ::`5'::<lambda_1> >+0x66 [deps\yalantinglibs\include\ylt\reflection\internal\generate\member_macro.hpp @ 28] 
kapp!ylt::reflection::internal::tuple_view<4,...'::`5'::<lambda_1> >+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 97] 
kapp!ylt::reflection::visit_members<...ss2:<lambda_1>,4>+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 134] 
kapp!struct_pack::detail::visit_members<... '::<lambda_1> >+0x32 [deps\yalantinglibs\include\ylt\struct_pack\reflection.hpp @ 963] 
kapp!struct_pack::detail::calculate_one_size<...,0>+0x49 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 172] 
kapp!struct_pack::detail::calculate_payload_size<0,some structs>+0x350 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 188] 
kapp!`struct_pack::detail::calculate_one_size<MyAppProto,0>'::`5'::<lambda_1>::operator()<MyAppProtoBase const &, some structs>+0x2fa [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 175] 
kapp!ylt::reflection::internal::object_tuple_view_helper<MyAppProto const &,48>::tuple_view<`struct_pack::detail::calculate_one_size<MyAppProto,0>'::`5'::<lambda_1> >+0x665 [deps\yalantinglibs\include\ylt\reflection\internal\generate\member_macro.hpp @ 164] 
kapp!ylt::reflection::internal::tuple_view<48,MyAppProto const &,`struct_pack::detail::calculate_one_size<MyAppProto,0>'::`5'::<lambda_1> >+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 97] 
kapp!ylt::reflection::visit_members<MyAppProto const &,`struct_pack::detail::calculate_one_size<MyAppProto,0>'::`5'::<lambda_1>,48>+0x29 [deps\yalantinglibs\include\ylt\reflection\member_ptr.hpp @ 134] 
kapp!struct_pack::detail::visit_members<MyAppProto const &,`struct_pack::detail::calculate_one_size<MyAppProto,0>'::`5'::<lambda_1> >+0x32 [deps\yalantinglibs\include\ylt\struct_pack\reflection.hpp @ 963] 
kapp!struct_pack::detail::calculate_one_size<MyAppProto,0>+0x49 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 172] 
kapp!struct_pack::detail::calculate_one_size<std::unique_ptr<MyAppProto,std::default_delete<MyAppProto> >,0>+0x66 [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 124] 
kapp!struct_pack::detail::calculate_payload_size<0,std::unique_ptr<MyAppProto,std::default_delete<MyAppProto> > >+0x1f [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 188] 
kapp!struct_pack::detail::get_serialize_runtime_info<0,std::unique_ptr<MyAppProto,std::default_delete<MyAppProto> > >+0x5e [deps\yalantinglibs\include\ylt\struct_pack\calculate_size.hpp @ 418] 
kapp!struct_pack::serialize_to<0,std::vector<char,std::allocator<char> >,std::unique_ptr<MyAppProto,std::default_delete<MyAppProto> > >+0x2c [deps\yalantinglibs\include\ylt\struct_pack.hpp @ 135] 
kapp!struct_pack::serialize<std::vector<char,std::allocator<char> >,std::unique_ptr<MyAppProto,std::default_delete<MyAppProto> > >+0x2c [deps\yalantinglibs\include\ylt\struct_pack.hpp @ 193] 
...
...
...
my stacks
...
...
...

[poor-circle]

Are you sure this is all stack? I think the stack of struct_pack is not very huge.

[poor-circle]

I Try to create a huge struct but it still works well, struct hi8 is has more than 100000 bytes.

struct hi {
  int a1000;
  int a1001;
  int a1002;
  int a1003;
  int a1004;
  int a1005;
  int a1006;
  int a1007;
  int a1008;
  int a1009;
  int a1010;
  int a1011;
  int a1012;
  int a1013;
  int a1014;
  int a1015;
  int a1016;
  int a1017;
  int a1018;
  int a1019;
  int a1020;
  int a1021;
  int a1022;
  int a1023;
  int a1024;
  int a1025;
  int a1026;
  int a1027;
  int a1028;
  int a1029;
  int a1030;
  int a1031;
  int a1032;
  int a1033;
  int a1034;
  int a1035;
  int a1036;
  int a1037;
  int a1038;
  int a1039;
  std::string b1000;
  std::string b1001;
  std::string b1002;
  std::string b1003;
  std::string b1004;
  std::string b1005;
  std::string b1006;
  std::string b1007;
  std::string b1008;
  std::string b1009;
  std::string b1010;
  std::string b1011;
  std::string b1012;
  std::string b1013;
  std::string b1014;
  std::string b1015;
  std::string b1016;
  std::string b1017;
  std::string b1018;
  std::string b1019;
  std::string b1020;
  std::string b1021;
  std::string b1022;
  std::string b1023;
  std::string b1024;
  std::string b1025;
  std::string b1026;
  std::string b1027;
  std::string b1028;
  std::string b1029;
  std::string b1030;
  std::string b1031;
  std::string b1032;
  std::string b1033;
  std::string b1034;
  std::string b1035;
  std::string b1036;
  std::string b1037;
  std::string b1038;
  std::string b1039;
};

struct hi2 {
  std::pair<hi,hi> a;
};

struct hi3 {
  hi2 a,b;
};

struct hi4 {
  hi3 a,b;
};

struct hi5 {
  hi4 a,b,c,d,e,f,g,h,i,j,k,l,m,n;
};

struct hi6 {
  hi5 a,b;
};

struct hi7 {
  hi6 a,b;
};

struct hi8 {
 hi7 a,b;
};

struct hi9 {
  std::vector<std::unique_ptr<hi8>> data;
};

void test() {
  hi9 h;
  for (int i=0;i<10;++i) {
    h.data.push_back(std::make_unique<hi8>());
  }
  auto buffer = struct_pack::serialize(h);
  std::cout << "hi4:"<<buffer.size() << std::endl;
  hi9 h2;
  auto ec =struct_pack::deserialize_to(h2,buffer);
  assert(!ec);
}

[romanholidaypancakes]

Are you sure this is all stack? I think the stack of struct_pack is not very huge.

But these stack frames alone occupy 0x2af0 bytes. I'm running in debug mode(/Od) (the compiler may reserve extra stack space for debugging).

[poor-circle]

Are you sure this is all stack? I think the stack of struct_pack is not very huge.

But these stack frames alone occupy 0x2af0 bytes. I'm running in debug mode(/Od) (the compiler may reserve extra stack space for debugging).

10KB is small, normally a stack has 1MB+ space.

[romanholidaypancakes]

Are you sure this is all stack? I think the stack of struct_pack is not very huge.

But these stack frames alone occupy 0x2af0 bytes. I'm running in debug mode(/Od) (the compiler may reserve extra stack space for debugging).

10KB is small, normally a stack has 1MB+ space.

To restate: I'm running in a constrained thread-stack environment where the thread stack is limited to only 3 KB.

[poor-circle]

Does Release Mode works well?

Are you sure this is all stack? I think the stack of struct_pack is not very huge.

But these stack frames alone occupy 0x2af0 bytes. I'm running in debug mode(/Od) (the compiler may reserve extra stack space for debugging).

[romanholidaypancakes]

Yes — it works well in release mode (Maximum Optimization (Favor Size) (/O1)) because there are fewer unnecessary function calls and reduced stack usage.

[poor-circle]

For meta programming grammer limit, struct_pack have to use a recursively way to calculate size. There has also many wrapper functions for meta programming which could be optimized. Those cause it stack overflow in very small stack.

I afraid this problem is hard to resolve.

[poor-circle]

By the way, we use __forceinline in MSVC to reduce wrapper function, but it doesn't work in msvc /O0.

[romanholidaypancakes]

For meta programming grammer limit, struct_pack have to use a recursively way to calculate size. There has also many wrapper functions for meta programming which could be optimized. Those cause it stack overflow in very small stack.

I afraid this problem is hard to resolve.

I previously thought there were other ways to replace recursive calls to reduce stack usage.

By the way, we use __forceinline in MSVC to reduce wrapper function, but it doesn't work in msvc /O0.

forceinline in MSVC debug mode requires extra options.

[poor-circle]

OK, so it seems that /O0 /d2Obforceinline or /Od + /Ob1 could fix your problem.