Skip to content

Fix OOM#82

Merged
wargio merged 5 commits intomainfrom
nodepool-impl
Mar 20, 2026
Merged

Fix OOM#82
wargio merged 5 commits intomainfrom
nodepool-impl

Conversation

@b1llow
Copy link
Member

@b1llow b1llow commented Mar 19, 2026

Removed all recursive node clone calls and added a simple fuzzer

fixes #81

Copilot AI review requested due to automatic review settings March 19, 2026 15:24
@b1llow
Copy link
Member Author

b1llow commented Mar 19, 2026

_ZNSt3__110__function6__funcIZZZN3RBXL10updateLoopENS_10shared_ptrINS2_18MemoryStoreHashMapEEERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS3_INS2
_3Lua13WeakObjectRefEEExNS_8functionIFvNS2_10Reflection7VariantEEEENSH_IFvSB_EEESJ_SD_ENK3$_0cl

Previously, this input would cause an OOM, but this does not appear to be a valid C++ obfuscation symbol.

The LLM believes it has been truncated; the complete symbol is likely this, but it was able to produce correct results previously.

_ZNSt3__110__function6__funcIZN3RBX10updateLoopENS_10shared_ptrINS2_18MemoryStoreHashMapEEERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS3_INS2_3L
ua13WeakObjectRefEEExNS_8functionIFvNS2_10Reflection7VariantEEEENSH_IFvSB_EEESJ_SD_E3$_0NS_9allocatorIcEEFvvEEclEv
std::__1::__function::__func<RBX::updateLoop(std::__1::shared_ptr<RBX::MemoryStoreHashMap>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<RBX::Lua::WeakObjectRef>, long long, std::__1::function<void (RBX::Reflection::Variant)>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>)>, RBX::Reflection::Variant, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_0, std::__1::allocator<char>, void ()>::operator()()

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets the crash/OOM reported in #81 by removing recursive AST node cloning in the Itanium (v3) C++ demangler and introducing a libFuzzer harness to stress those code paths.

Changes:

  • Reworked v3 AST ownership to use a DemContext node pool and NodeRef references instead of deep clones.
  • Updated v3 parser/pretty-printer plumbing to use the new NodeRef vectors and safer iteration/bounds checks.
  • Added an optional Meson fuzz build (enable_fuzz) and a fuzz_demangle libFuzzer target.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/test_subs.c Initializes DemContext for substitution table tests and adapts to NodeRef vectors.
test/data/cxx_lambda.csv Adds a new lambda-related demangling test vector.
src/microsoft_demangle.c Adjusts list element free callback signature to match container expectations.
src/cplusplus/vec.h Ensures element destructors run when vectors shrink (resize down).
src/cplusplus/v3/v3.c Large v3 refactor: switches to NodeRef, removes cloning, adds context/node pool init/deinit, adds parser call/recursion limits, and introduces multiple ownership/transfer fixes.
src/cplusplus/v3/v3_pp.h Updates ast_pp prototype to accept NodeRef.
src/cplusplus/v3/types.h Introduces NodeRef, VecNodeRef/VecVecNodeRef, and DemContext node pool; updates many AST fields to NodeRef.
src/cplusplus/v3/parser_combinator.c Updates match_many* to allocate nodes via DemNode_new(p->context) and store children as NodeRef.
src/cplusplus/v3/meta.c Updates parser metadata containers to NodeRef-based vectors and removes output destruction in DemResult_deinit.
src/cplusplus/v3/macros.h Refactors rule macros to allocate via node pool, adds recursion/call limits, and updates AST append helpers for NodeRef.
src/cplusplus/v3/ast.c Updates node construction to register nodes into the DemContext pool; simplifies deinit/copy semantics for the new ownership model.
src/cplusplus/demangle.c Adjusts dispatcher logic for when to try v2/v3-type fallbacks.
meson.build Adds enable_fuzz build flow for a sanitizer/coverage-instrumented fuzz target.
meson_options.txt Adds enable_fuzz option.
fuzz/fuzz_demangle.c New libFuzzer harness targeting v3 demangling entry points.
.gitignore Ignores fuzz/corpus.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@wargio
Copy link
Member

wargio commented Mar 19, 2026

you seems to have some issues with const

b1llow and others added 4 commits March 20, 2026 01:06
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@wargio wargio merged commit ba04881 into main Mar 20, 2026
9 checks passed
@wargio wargio deleted the nodepool-impl branch March 20, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Segfault Crash when opening large binary

3 participants