[Bug – NOK] Memory usage steadily increases over time in SceneScape 2025.2 (possible memory leak)

[COZYTRICKSTER]

Bug Description

When running the SceneScape:2025.2 application with default cameras, RAM usage continues to increase over time and does not stabilize. Ideally, memory should be released when not in use, but this does not appear to be happening.

A Grafana dashboard screenshot (attached below) shows continuous memory growth over the last ~18 hours under normal workload.

---

To Reproduce

Steps to reproduce the behavior:

1. Clone the SceneScape repository
2. Build and run the project as per the instructions
3. Monitor memory usage (e.g., via top, htop, or Grafana dashboard)
4. Observe memory usage increasing over time even without additional workload

---

Expected behavior

Memory usage should level off after initial allocations and should not keep growing unbounded with time.

---

Screenshots

• Grafana dashboard screenshot attached below showing steady memory growth over the last ~18 hours under normal workload.

---

Release version

• Version of this project: 2025.2

---

Environment

• OS: Ubuntu 22.04.5 LTS
• CPU: 12th Gen Intel(R) Core(TM) i3-1220PE
• RAM: 16GB
• Deployment: Kubernetes container
• Monitoring/Profiling tools: Grafana

---

Additional context

• Memory usage continues to grow under normal workload without stabilizing.
• The issue is observed with default cameras, with no additional load.

[COZYTRICKSTER]

Additional Analysis / Attempts to Mitigate Memory Leak

Several approaches were tried to reduce or stabilize memory growth, but the issue persists:

---

1. Tracker Pruning in uuid_manager

Tried adding tracker.delete_track(track_id) in the pruneInactiveTracks() function.
This helped in stabilizing memory observed over a longer period, but when logged suspended tracks count and reactivating tracks count, it was observed that suspended tracks were increasing while reactivation of tracks didn't happen; reactivating track count remained 0.

---

2. Changes in predict() function

Added code in the predict() function:

std::vector<std::reference_wrapper<MultiModelKalmanEstimator>> suspendedEstimators;
suspendedEstimators.reserve(mSuspendedKalmanEstimators.size());

for (auto &element : mSuspendedKalmanEstimators)
{
    suspendedEstimators.push_back(std::ref(element.second));
}

#pragma omp parallel for
for (size_t i = 0; i < suspendedEstimators.size(); ++i)
{
    suspendedEstimators[i].get().predict(deltaT);
}

Predict on the suspendedTracks as well.
Suspended Kalman predictors executed, but reactivated tracks size still remains 0.

---

3. Prune logic on C++ side

Tried adding the prune logic in C++ code.
It cleans up suspended tracks which have an active life for > 2 minutes, still see reactivated tracks size = 0.

---

Related logs:

[saratpoluri]

Able to reproduce the issue on our side. Debugging in progress.

[ltalarcz]

Hi @COZYTRICKSTER ,

What was done so far:

1. Python-Level Profiling

Tools used:

• objgraph
• Custom signal-triggered snapshot scripts

Findings:

• Python object counts were stable or slightly decreasing over time.
• No evidence of unbounded Python object accumulation.
• Garbage collection was working as expected.

---

2. C++ Tracker Analysis and some fix proposal

Focus:
Investigation shifted to the C++ tracking code, specifically the TrackManager and its handling of suspended tracks.
I followed your initial observations of suspended tracks (mSuspendedKalmanEstimators) that were never cleaned up if not reactivated.
I decided to give it a try to prune suspended track and verify memory usage.

Solution:

• Added age‑based cleanup for suspended tracks.
• Tracks suspended longer than a configurable timeout (default 60s, settable via environment variable) are automatically deleted.
• Timeout is configurable from Python and exposed via pybind11 bindings.

Code changes:

• New member in TrackManagerConfig for timeout.
• Suspension times tracked in a map.
• Cleanup invoked in every predict().

Memory Observation After Fix:

• Suspended track count remained stable over time.
• Memory usage is stable, I do not observe memory increase

@COZYTRICKSTER could you please try it on your end?

Branch:
ITEP-84109-memory_usage_steadily_increases — fix is available and ready for testing.

[COZYTRICKSTER]

Hi @ltalarcz,

Thanks for the detailed investigation and for sharing the fix.

This is similar to an approach we tried earlier, as mentioned (“Prune logic on C++ side for suspended tracks”). In our case, pruning did help stabilize memory usage; however, our main concern was that suspended tracks were not being reactivated as expected.

Meanwhile, there are a few things worth mentioning regarding the reactivation aspect. One approach I tried was prioritizing suspended tracks over unreliable tracks during matching and assignment with the received detected objects. This did not resolve the reactivation of suspended tracks, but it significantly reduced the rate at which memory usage grew.

That said, I’ll pull the ITEP-84109-memory_usage_steadily_increases branch and run the fix on our side. I’ll share observations and results once I have them.

[COZYTRICKSTER]

Hi @ltalarcz,

I have tested the provided fix (ITEP-84109-memory_usage_steadily_increases) and confirmed that memory usage remains stable over a period of time. The memory leak appears to be addressed by this fix, similar to the approach we had previously tried on our end.

Regarding reactivation of suspended tracks, I have come up with a test case for static objects and tried to trigger the reactivation logic and observed that the reactivation of suspended tracks is working.

Please proceed with merging this fix into the 2025.2 release.

[ltalarcz]

Resolved in: #1005