[BUG - Release] Crash in process_simple_actions from overlapping ExecuteInGameThread callbacks

[SSyl]

Branch or Release
v3.0.1-932-ge31aaaa6

Game and Engine Version
Abiotic Factor v1.2.x (UE 5.4)

Describe the bug
When multiple deferred actions overlap via ExecuteInGameThread, whether through ExecuteWithDelay chains, direct scheduling from within callbacks, or just multiple mods independently queuing work, the game crashes with get_function_ref "Ref was not function" followed by abort().

I first noticed this during normal gameplay (~20 minutes in, hosting a multiplayer session) with my standard mod loadout and another crash about 30 minutes in. After investigating the crash dumps, I created a minimal reproduction mod that simulates two mods scheduling independent deferred work.

The crash appears to come from process_simple_actions using std::erase_if to iterate m_engine_tick_actions, while callbacks can push new items to that same vector during iteration:
https://github.com/UE4SS-RE/RE-UE4SS/blob/main/UE4SS/src/Mod/LuaMod.cpp#L3650-L3671

Mods directory
Using only default mods and my minimal reproduction mod attached: EngineTickReentryBug.zip

Also available at https://github.com/SSyl/UE4SS-EngineTickReentryBug

The repro simulates two mods with overlapping deferred work:

-- Chain A: schedules next iteration after 50ms
local function ChainA()
    ExecuteWithDelay(50, function()
        ExecuteInGameThread(ChainA)
    end)
end

-- Chain B: schedules next iteration after 75ms
local function ChainB()
    ExecuteWithDelay(75, function()
        ExecuteInGameThread(ChainB)
    end)
end

Why the 50/75ms delays? I felt like they were lenient enough to prove this isn't just a torture test - a hook that runs per frame would be firing every ~8ms at 120fps. You can drop the delays to 25/50 or lower to crash faster, or just call ExecuteInGameThread several times without any delay and it crashes almost instantly. I used two chains to simulate a more realistic scenario where two separate mods are independently doing game-thread work.

To reproduce

1. Copy EngineTickReentryBug folder into your Mods directory
2. Launch the game (single or multiplayer, doesn't matter)
3. Watch the UE4SS console for iteration count logs
4. Wait for crash (typically 1-5 minutes, varies due to timing-dependent nature)

Expected behavior
Deferred callbacks should execute without crashing, even when they schedule additional deferred work.

Screenshots, UE4SS Log, and .dmp file
5 crash dumps collected, all showing the same call chain:

engine_tick_hook
  → process_simple_actions
    → std::erase_if
      → get_function_ref
        → abort

Iteration counts at crash varied significantly, which makes me think it may be a race condition/timing dependent bug:

• ~1,190 iterations
• ~2,058 iterations
• ~2,299 iterations
• ~4,300 iterations
• ~10,800 iterations

Sample log output before crash:

[Lua] [EngineTickReentryBug] Chain A - Iteration 2295
[Lua] [EngineTickReentryBug] Chain B - Iteration 2296
[Lua] [EngineTickReentryBug] Chain A - Iteration 2297
[Lua] [EngineTickReentryBug] Chain B - Iteration 2298
[Lua] [EngineTickReentryBug] Chain A - Iteration 2299
*crash*

Three minidumps are in the zip file, but I have several full dumps (7-8GB) available upon request.

EnginetickRentry-UEMinidumps.zip

GitHub wouldn't let me zip the pdb/dll with it, but I'm using UE4SS 3.0.1-932.

Desktop (please complete the following information):

• OS: Windows 11 25H2

Additional context

• I have hundreds of hours on UE4SS v3.0.1-553 (the version currently shipped on Nexus for Abiotic Factor) and never encountered this crash, so I think it was introduced (or at least exposed) via a commit afterwards
• The original crash during normal gameplay occurred while hosting multiplayer, but the repro crashes in single player as well
• The timing variance (1k-11k iterations) makes me think it's a race condition
• Additional context/speculation in my readme at my repro mod's repo: https://github.com/SSyl/UE4SS-EngineTickReentryBug

[UE4SS]

Thanks for the report.

We do protect both the engine tick callback, as well as the ExecuteInGameThread registration on the C++ side with std::recursive_mutex, so I'm not sure what's going on.

[UE4SS]

Can you try this dll ?
https://www.dropbox.com/scl/fi/c6jv1xvq6c34wg64h8e06/UE4SS_maybe_lua_fix.zip?rlkey=w44dur0tqqz6e1y3jzeibcnqb&dl=1

[emoose]

Can you try this dll ? https://www.dropbox.com/scl/fi/c6jv1xvq6c34wg64h8e06/UE4SS_maybe_lua_fix.zip?rlkey=w44dur0tqqz6e1y3jzeibcnqb&dl=1

Can't speak for OP but I think this fixed a similar crash issue I've been having, my code uses NotifyOnNewObject to hook an actor ctor and then ExecuteInGameThreadWithDelay inside the newobject callback so I can overwrite a value which gets set after ctor, previously this had chance of crashing inside erase_if (and sometimes in lua code(?)), which I guess was because the notify also runs in game thread.

Would the changes in that DLL have affected the delayed funcs? or maybe I'm just imagining things lol.

E: aw sadly still had a crash in some LUA code, haven't run into any erase_if crash though, hopefully OP has better luck with it.

(E2: ah the ExecuteWithDelay+ExecuteInGameThread pattern OP mentioned might be working for me though - glad this was brought up and might get a fix soon)

[SSyl]

Can you try this dll ? https://www.dropbox.com/scl/fi/c6jv1xvq6c34wg64h8e06/UE4SS_maybe_lua_fix.zip?rlkey=w44dur0tqqz6e1y3jzeibcnqb&dl=1

Hey, wanted to report back on testing the fix DLL.

Good news first: The fix seems solid for the original repro. I tested it with both my repro mod (the ExecuteWithDelay + ExecuteInGameThread pattern at 50ms/75ms) and my full mod suite during regular gameplay. 221k+ iterations over about 26 minutes with no crashes. Really nice work.

Then I got curious. One of the things I've been looking forward to in newer UE4SS versions is ExecuteInGameThreadWithDelay, having a single API call is really nice instead of the ExecuteWithDelay + ExecuteInGameThread nesting we've had to do. So I figured I'd do pretty much the same test as my repro mod, except using that for a self-rescheduling timer:

local function PollGameState()
    -- do work
    ExecuteInGameThreadWithDelay(500, PollGameState)
end
ExecuteInGameThreadWithDelay(500, PollGameState)

You can obviously do this with ExecuteWithDelay + ExecuteInGameThread instead, but I wanted to test the single-call API since it's cleaner and seems like the direction things are heading. Unfortunately, this crashes almost immediately (so fast that a few of my dumps came back as 0kb, so probably before the crash reporter could even start), on the very first timer tick, even with a 500ms interval. It also crashes with the fix DLL, so it seems like it might be a different issue from the shared Lua stack problem.

What the dumps look like: The crashes are a bit tricky to analyze. WinDbg's !analyze -v defaults to an NVIDIA driver thread with various access violations (null pointer, invalid read, NX fault, different each time). But checking thread 0 (game thread) with ~0 kp shows a consistent stack:

VCRUNTIME140!memcpy
  std::vector<DelayedGameThreadAction>::erase
    std::erase_if
      RC::process_delayed_actions
        RC::engine_tick_hook

The NVIDIA crashes seem like secondary heap corruption, the GPU driver is probably just the first thing to trip over corrupted memory after whatever happens on the game thread.

I grabbed a full memory dump (using -fullcrashdumpalways) and tried to poke around a bit. The thing that stood out to me was that the vector's data was at one address, but the memcpy that crashed was reading and writing to a completely different address range. And the byte count on the memcpy was something like 26 MB, which seems way too large for a vector that only had 2 elements in it.

It looks to me like the memcpy might be working with pointers to where the vector used to be rather than where it is now. Is it possible that the emplace_back from ExecuteInGameThreadWithDelay inside the callback is causing a reallocation while std::erase_if is still iterating?

My guess for why the ExecuteWithDelay + ExecuteInGameThread pattern doesn't hit this: ExecuteWithDelay runs on a separate thread, so by the time ExecuteInGameThread pushes to the vector, the iteration from the previous tick is already done. With ExecuteInGameThreadWithDelay, the push happens right in the middle of it.

I've attached the new bug reproduction mod DirectDelay (the ExecuteInGameThreadWithDelay version that crashes) even with your fixed DLL:

DirectDelay.zip

You can also find the original repro mod and the new one here with my updated README containing more info/speculation: https://github.com/SSyl/UE4SS-EngineTickReentryBug

I've also attached the non-zero KB minidumps (the dll and pdb will be the same as the one you posted in your previous quoted reply):

DumpsWithFixedDLL.zip

I have my full ~8GB dump as well which you can find here: https://drive.google.com/drive/folders/1q1ptjA997bq--7_Nuk71zIR2gBLFIkNM

[narknon]

@emoose can you try this dll out?

UE4SS.zip

[emoose]

@narknon sadly both ways still crashed with that :(

With ExecuteWithDelay+ExecuteInGameThread I could load in but crashed as soon as I loaded a different area: UEMinidump.zip

ExecuteInGameThreadWithDelay looked like it was working, but forcing game to reload area a few times (using the customization mirror in base & exiting out) ended up having another crash: UEMinidump.zip

LUA for those is pretty much just

local multiplier = 5

local function SafeIsValid(obj)
    if not obj then return false end
    if type(obj) ~= "userdata" then return false end
    local success, result = pcall(function() return obj:IsValid() end)
    return success and result == true
end

local function ExpandPickupItemRadius(ConstructedObject)
	local object = ConstructedObject
	if not SafeIsValid(object) then
		return
	end
	
	if object:HasAnyInternalFlags(EInternalObjectFlags.PendingKill) then
		return
	end
	
	ExecuteInGameThreadWithDelay(50, function()
		print("exec")
		if SafeIsValid(object) and not object:HasAnyInternalFlags(EInternalObjectFlags.PendingKill) and SafeIsValid(object.Trigger) and not object.Trigger:HasAnyInternalFlags(EInternalObjectFlags.PendingKill) then
			print("valid")
			local originalRadius = object.Trigger.SphereRadius or 0.0
			local newRadius = originalRadius * multiplier
			print("newradius" .. newRadius)
			object.Trigger:SetSphereRadius(newRadius, true)
		end
	end)
end

NotifyOnNewObject("/Script/Extensions.PickupItem", ExpandPickupItemRadius)
NotifyOnNewObject("/Script/Extensions.PickupMaterial", ExpandPickupItemRadius)
NotifyOnNewObject("/Script/Extensions.LostExp", ExpandPickupItemRadius)

[narknon]

What game? I can try getting the crash myself

[emoose]

Code Vein 2, just fast travelled on map for the first crash, and opened/closed the chara customize mirror at base 10-20 times for the second, https://gofile.io/d/qXEZNg save should be right in front of the mirror if you need it.

[narknon]

Heh, I get a crash as soon as I press new game in the demo with nothing aside from default mods

[emoose]

Ah that's probably the cheat manager mod, not sure why that causes crash, deleting CheatManagerEnablerMod folder for it should work though

[narknon]

It seems to crash when trying to unregister hooks alongside an engine tick invoke

[narknon]

I'm wondering if there's a member var or vtable issue.

[narknon]

@emoose curious how many results you'd get if you run this on the exe https://github.com/trumank/binfold

[emoose]

Found 377741 functions
Matched 59885 functions with database symbols
Generating PDB file at: C:\Program Files (x86)\Steam\steamapps\common\CodeVein2\CodeVein2\Binaries\Win64\CodeVein2-Win64-Shipping-Protected.pdb
PDB file generated successfully with 59885 functions

PDB from that at https://gofile.io/d/JjPcKF
When I was looking at exe before it did look like a lot of normal UE funcs got denuvo'd though :/