Velox crash after using Jemalloc

[whutjs]

Velox starts to crash frequently after linking with jemalloc 5.3.0:

#0 GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007fbd59cfa535 in _ GI abort () at abort.c:79
#2 0x00007fbd577104bb in os::abort(bool, void*, vvoid const*) [clone .cold] () from /usr/local/byteopenjdk/jdk-17/lib/server/libjvm.so
#3 0x00007fbd583963fc in VMError::report_and_die(int, char const*, char const*, _va_list_tag*, Thresad*, unsigned char*, void*, void*, char const*, int, unsigned long)() from /usr/local/byteopenjdk/jdk-17/lib/server/libjvm.so
#4 0x00007fbd58396elb in VMError::report_and_die(Thrread*, unsigned int, unsigned char*, void*, void*, char const*, ...) () from /usr/local/byteopenjdk/jdk-17/lib/server/libjvm.so
#5 0x00007fbd58396e4e in VMError::report_and_die(Thrread*, unsigned int, unsigned char*, void*, void*)() from /usr/local/byteopenjdk/jdk-17/lib/server/libjvm.so
#6 0x00007fbd5823e8b2 in JVM_handle_linux_signal ()from /usr/local/byteopenjdk/jdk-17/lib/server/libjvm.so
#7 <signal handler called>
#8 0x00007fbada54b98e in _mm_load_si128(long long _vector(2) const*) (__P=0x7fbd3d5d8098) at /usr/1iib/gcc/x86_64-linux-gnu/9/include/emmintrin.h:697
#9้ folly::f14::detail::F14Chunk<std::pair<folly::dynamic const, folly::dynamic>*>::occupiedMa:sk (this=0x7fbd3d5d8098) at_../folly/container/detail/F14Table.h:494
#10 0x00007fbada54a49c in folly::f14::detail::F14Chunk<std::pair<folly::dynamic const, folly::dynaamic>*>::lastOccupied (this=0x7fbd3d5d8098) at ..folly/container/detail/F14Table.h:508
#11 0x00007fbada5451df in folly::f14::detail::Fl14Table<folly::f14::detail::NodeContainerPolicy<ifolly::dynamic, folly::dynamic, folly::detail::DyynamicHasher, folly::detail::DynamicKeyEqual, void> > :directBuildFrom<folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynaamicHasher, folly::detail::DynamicKeyEqual, void> > const&>(this=0x7fbb2c101c60, src=...) at ../folly/conitainer/detail/F14Table.h:1576
#12 0x00007fbada541cdc in folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> >  :buildFromF14Table<folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> > const&V (this=0x7fbb2c101c60, src=...) at ../folly/container/detail/F14Table.h:1699
#13 0x00007fbada53ed63 in folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamnicHasher, folly::detail::DynamicKeyEqual, void> >  :F14Table (this=0x7fbb2c101c60, rhs=...) at ../folly/container/detail/F14Table.h:910
#14 0x00007fbada53d7f5 in folly::f14::detail::F14BasicMap<folly::f14::detail::NodeContainer:Policy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> >::F14BasicMap (this=0x7fbb2c101c60, rhs=...) at../folly/container/F14Map.h:168
#15 0x00007fbada53d81b in folly: F14NodeMap<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, std::allocator<std::pair<folly::dynamic const, folly::dynam ic> > >::F14NodeMap (this=0x7fbb2c101c60) at ../folly/container/F14Map.h:990
#16 0x00007fbada53d841 in folly::dynamic::ObjectlImpl: ObjectImpl (this=0x7fbb2c101c60) at ../folly/dynamic-inl.h:179 #17 0x00007fbada53895c in folly::dynamic::dynamic(this=0x7fbb2c101c58, o=...) at ../folly/dynamic.cpp:136
#18 0x00007fbada54eceb in std::pair<folly::dynamiic const, folly::dynamic>::pair (this=0x7fbb2c101030) at /usr/include/c++/9/bits/stl_pair.h:303
#19 0x00007fbada54ed55 in __gnu_cxx::new_allocator<std::pair<folly::dynamic const, folly::dynamic>>::construct<std::pair<folly::dynamic const, folly::dynamic>, std::pair<folly::dynamic const, fol运 y::dynamic>_const&> (this=0x7fba415f9cb8,___p=0x7fbb2c101c30) at /usr/include/c++/9/ext/newallocator.h:146
#20 0x00007fbada54d051 in std::allocator_traitts<std::allocator<std::pair<folly::dynamic conast, folly::dynamic> > >::construct<std::pair<folly::dynamic const, folly::dynamic>, std::pair<folly::dynam ic const, folly::dynamic> const&> (
_a=...,__p=0x7fbb2c101c30) at /usr/include/c++/9/bits/alloc traits.h:483
#21 0x00007fbada54a3cb in folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dyynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void>::constructValueAtItem<int, std :pair<folly::dynamic const, folly::dynamic> const&> (this=0x7fba415f9cb8, itemAddr=0x7fbb3729f240)at ../folly/container/detail/F14Policy.h:835
#22 0x00007fbada545102 in folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamnicHasher, folly::detail::DynamicKeyEqual, void> >: :directBuildFrom<folly::f14::detail::F14Table<<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynaamicHasher, folly::detail::DynamicKeyEqual, void> > const&>
(this=0x7fba415f9cb8, src=...) at ../folly/conatainer/detail/F14Table.h:1560
#23 0x00007fbada541cdc in folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> >: :buildFromF14Table<folly: f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> > const&y (this=0x7fba415f9cb8, src=...) at ../folly/container/detail/F14Table.h:1699
#24 0x00007fbada53ed63 in folly::f14::detail::F14Table<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamnicHasher, folly::detail::DynamicKeyEqual, void> >: :F14Table (this=0x7fba415f9cb8, rhs=...) at ../folly/container/detail/F14Table.h:910
#25 0x00007fbada53d7f5 in folly::f14::detail::F14BasicMap<folly::f14::detail::NodeContainerPolicy<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, void> >::F14BasicMap (this=0x7fba415f9cb8, rhs=...) at../folly/container/F14Map.h:168
#26 0x00007fbada53d81b in folly: F14NodeMap<folly::dynamic, folly::dynamic, folly::detail::DynamicHasher, folly::detail::DynamicKeyEqual, std::alloocator<std::pair<folly::dynamic const, folly::dynaam ic> > >::F14NodeMap (this=0x7fba415f9cb8) at../folly/container/F14Map.h:990
#27 0x00007fbada53d841 in folly::dynamic::ObjectImpl::ObjectImpl (this=0x7fba415f9cb8) at ../folly/dynamic-inl.h:179 #28 0x00007fbada53895c in folly::dynamic::dynamic(this=0x7fba415f9cb0, o=...) at ../folly/dynamic.cpp:136
#29 0x00007fbad7b804dc in void std::vector<folly::dynamic, std::allocator<folly::dynamic> >::_M_realloc_insert<folly::dynamic const&>(__gnu_cxx::__normal_iterator<folly::dynamic*, std::vector<folly ::dynamic, std::allocator<folly::dynamic> > >,, folly::dynamic const&) () from /root/.javacpp/cache/bytetsd-query-server.jar/bridge/linux-x86_64/libvelox_es_plugin.so #30 0x00007fbad7b7fbe2 in facebook::velox::exec::toPlanStatsJson(facebook::velox::exxec::TaskStats const&)() Type <RET> for more, q to quit, c to continuewithout paging--

Does anyone succeed in using jemalloc in Velox?

[mbasmanova]

CC: @tanjialiang @amitkdutta @xiaoxmeng

We use jemalloc at Meta.

It is hard to work with screenshots. Would you mind updating the question to include stack trace in plain text? The crash seems to come from folly's F14 hash table.

[whutjs]

CC: @tanjialiang @amitkdutta @xiaoxmeng

We use jemalloc at Meta.

It is hard to work with screenshots. Would you mind updating the question to include stack trace in plain text? The crash seems to come from folly's F14 hash table.

Thanks for the reply, I have updated the question.