[Bug - NOK] Object tracking degrades after adding 7th camera in SceneScape 2025.2 (Kubernetes, Helm deployment)

[palwithpen]

🐛 Bug Description

We deployed Intel® SceneScape version 2025.2 on a Kubernetes cluster using the official Helm charts. During scalability testing, we observed a degradation in object tracking accuracy when the number of cameras exceeds six.

With up to six cameras, object detection and tracking are precise and stable. However, after adding and calibrating a seventh camera, the tracker begins to intermittently miss objects across one or more streams. This occurs without any other configuration changes.

🎥 Camera / Stream Configuration

• Stream protocol: RTSP
• Resolution: 1024 × 540
• Frame rate: 5 FPS
• Objects per camera: ~6 moving objects

5PeopleFPS5.mp4

Calibration: Successfully completed for all cameras

Time chunking: Disabled

📋 Steps to Reproduce

• Clone the SceneScape repository and build the images:
  make

• Push the generated Docker images to a container registry.

• Navigate to the Kubernetes deployment directory:
  cd kubernetes

Deploy SceneScape using Helm:

make copy-files && \
make install-cert-manager && \
helm upgrade --install scenescape scenescape-chart/ \
  -n scenescape \
  --create-namespace \
  --timeout=10m \
  --set supass=change_me \
  --set pgserver.password=change_me

• Add six cameras, provide RTSP streams, and complete calibration.

• Observe that object detection and tracking are stable.

• Add a seventh camera and complete calibration.

• Observe that object detection begins to flicker and objects are intermittently missed.

✅ Expected Behavior

After adding and calibrating the seventh camera, object detection and tracking should remain consistent and stable, similar to the behavior observed with six cameras.

❌ Actual Behavior

Once the seventh camera is added:

Object detections intermittently disappear

Tracking becomes unstable across one or more camera streams

📦 Release Information

SceneScape version: 2025.2

🧩 Environment

Operating System: Ubuntu 24.04.3 LTS
Deployment: Kubernetes (Helm)
Hardware: CPU-only
CPU: 13th Gen Intel® Core™ i9-13900E
Python version: 3.10
OpenVINO version: Not applicable / not explicitly used

[saratpoluri]

This issue is being actively debugged.

[tdorauintc]

@palwithpen in 2025.2 release the controller tracker behavior may be sensitive to the camera count and the tracker parameters in tracker-config.json file, especially for (mostly) non-overlapping cameras. This is because with increasing number of non-overlapping cameras the object refresh rate measured by frame count (frames with object detections relative to all frames seen by the tracker) decreases in inverse proportion (e.g. for six non-overlapping cameras each object is refreshed by new detection every 6th frame processed by the tracker). In consequence, the tracker is more prone to lose tracks if the frame-based tracker configuration parameters aren't updated accordingly.

This is only one of possible causes, but it is important to clarify it and make sure configuration is correctly adjusted for your use case (taking into account low FPS and number of objects <50, the performance shouldn't be the bottleneck here).

Let me provide some suggestions how you can deal with it:

• enable time-chunking which can be especially helpful to deal with large number cameras. It has two positive outcomes: the detections are batched in time-chunks of configurable length which results in better performance (tracker processing window does not shrink with increasing number of cameras, processing data in batches enables optimizations) and the object refresh rate seen by the tracker is independent of camera count which simplifies the configuration process.
• keep the time-chunking disabled and use the latest 2026.0 release where we introduced changes that simplify adjusting the tracker configuration to the camera count. In your case (non-overlapping cameras @5 fps) you could set effective_object_update_rate=5*<camera count> and keep the time-based parameters as is.

If none of above options is viable, we can discuss other possibilities.

Please see our tracker configuration guide for 2025.2 and 2026.0 release and reach out to us if you need further help.

[m13v]

this is a classic issue with multi-camera tracking when you scale past 6-7 streams. we've seen similar degradation and it usually comes down to the tracker's global association step becoming the bottleneck, not the detector itself. with 7 cameras at 5 fps each that's 35 frames/sec hitting the association logic, and CPU scheduling starts dropping frames under load even though individual inference looks fine. one thing that helped us was running per-camera detection independently and doing cross-camera association at a lower frequency (1-2 Hz instead of per-frame). also worth checking if your MQTT broker is dropping messages under the increased event load from the additional camera.

[m13v]

we handle up to 25 camera feeds per edge unit and the per-camera independent detection was key for us. some context on the multi-camera monitoring architecture we use: https://github.com/mediar-ai/cyrano-security/blob/main/src/app/t/apartment-security-incident-awareness-monitoring/page.tsx

[saratpoluri]

Thank you for sharing your thoughts Matthew. Time chunking has served us well when the number of cameras increase and is very similar to the approach you stated of doing association at a lower rate. Also helps with leveraging the available compute better. We are also exploring other algorithmic approaches for parallelizing the global association step. MQTT has scaled better than we expected and has not proved to be the bottleneck.

Could you please repost your link. The link you gave gives a 404.

[m13v]

sorry about the broken link! the repo has moved. the relevant pattern is: run detection per camera independently, then do cross-camera association at a lower frequency (every N frames rather than every frame). this keeps the association step from becoming the bottleneck as you add more cameras. time chunking as you described is essentially the same idea. glad to hear MQTT has scaled well for the messaging layer.