Scaler won't scale certain services due to incorrect resource estimates

[ppil]

Describe the bug
I'm running AL4 under proxmox in my homelab, and am currently blocked on upgrading to 4.7. I have over allocation set to 0 for cpu and mem, since those are limited by the VM config (no auto-scaling). I set the minimum instances for most services to 0, since I do not want them spun up while the system's idle. I've had this setup working for the past year+, from 4.5 through the latest 4.6.

Under 4.7.x, the scaler will refuse to scale up various services, seemingly at random. This will result in the submission stalling, the system going idle with pending service queues. If I manually scale these services by setting their minimum instances to 1, then they will scale up and process their queue and the scaler will then scale others. Eventually, this will occur on other services, which can also be manually scaled until processing eventually succeeds. Future submissions will then stall on other services which were not manually scaled to 1.

Enabling debug logging, I can see that the scaler has a wildly inaccurate estimate of used/available resources (likely a known issue, now a problem). When it stalls, it does so under the assessment of there being insufficient resources for the service.

Example

In one case, it claimed there was only 534mb ram available, when htop shows 13gb and the hypervisor shows 8gb. It also claimed there were only 4 cores available when the running containers were idle and cpu utilization was 1% per hypervisor and htop:

"logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Can't make more because not enough resources {'Safelist': (1.0, 512)}"} scaler | {"@timestamp": "2026-02-20 01:28:03,731", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c68c0cefeebe" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available 534.77734375/24026.77734375"} scaler | {"@timestamp": "2026-02-20 01:28:03,787", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c68c0cefeebe" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 4.699999999999999/24"}

In either situation, even if those were the actual resources available, safelist would have had the 1 core and 512mb RAM needed, yet the scaler still refused. I manually scaled safelist up to a minimum of 1 instance, and it launched other services no problem thinking it had negative resources for them... Until it reached avclass and tagcheck, at which point it stalled again. After letting the system sit idle, this was the state in which it was stalled:

"logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Can't make more because not enough resources {'AVClass': (0.25, 512), 'TagCheck': (0.5, 2048)}"} scaler | {"@timestamp": "2026-02-20 01:49:16,370", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c68c0cefeebe" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available -15337.22265625/24026.77734375"} scaler | {"@timestamp": "2026-02-20 01:49:16,437", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c68c0cefeebe" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available -3.8000000000000007/24"}

In reality, CPU utilization was 1% with 14/24GB RAM used, certainly not negative, no OOM reaping occurred via kernel, plenty available to run these services.

To Reproduce
Steps to reproduce the behavior:

1. Create a monolithic docker deployment of 4.7.x
2. 0 overallocation configured
3. Enable various services, 0 minimum instances
4. Submit a sample
5. Watch scaler not scale, processing stalls

Expected behavior
Scaler has a sane estimate of resource utilization
Scaler continues to scale up/down services given available resources

Screenshots

• Docker stats output during the last mentioned stall where scaler saw absurd negative resources available:

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.7.x (4.6 works)
• Docker deployed under a single virtual machine, debian 12 given 24 cores 24gb ram

[cccs-douglass]

The scaler isn't trying to estimate the resources utilization on the system. It is trying to collect the resource totals and the container environment's resource reservations. You will never see reserved resource numbers from scaler that look anything like what is reported for utilization in htop, they aren't trying to measure the same thing.

Overallocation is a multiplier applied to the total resources that the scaler is trying to fit the reservation to. Disabling overallocation would mean setting the field to 1. Zero would mean to always assume there are no free resources and only ever start the minimum number of instances. When a service has items in the queue the minimum instances is raised to 1 if it was zero, so this setup should still work as you have seen in the past.

From my recall, this has been the behaviour since the first releases of Assemblyline 4 and none of these behaviours should have changed in the 4.7 release, but clearly something is up if your system is stalling.

Just to get an idea of where to start with debugging can you confirm:

• You are seeing INFO level logs from scaler of the form " isn't meeting minimum allocation 0 -> 1" when your services are failing to scale up when files are waiting?
• That the containers aren't being created at all? (vs being created but failing to process files or show up on the dashboard)

[ppil]

Yeah info level logs do show the scaler requesting 0->1, but then nothing happens. Those containers never get created. Debug logs then show that nothing happens due to imaginary resource constraints. And again, this doesn't happen for all containers and the scaler will gladly spin some up when it thinks there are negative available resources. IIRC I set over-allocation to 0 to avoid kernel OOM reaping.

[ppil]

Reproducing with Assemblyline 4.7.1.12, more logs, same behavior.

{"@timestamp": "2026-03-27 00:05:20,957", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Safelist isn't meeting minimum allocation 0 -> 1"}  
{"@timestamp": "2026-03-27 00:05:20,957", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Hashlookup isn't meeting minimum allocation 0 -> 1"} 
{"@timestamp": "2026-03-27 00:05:20,967", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "New target for Hashlookup: 0 -> 1"}
{"@timestamp": "2026-03-27 00:05:20,968", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "New target for Safelist: 0 -> 1"}
{"@timestamp": "2026-03-27 00:05:21,084", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 4.199999999999999/24"} 
{"@timestamp": "2026-03-27 00:05:24,516", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 2.1999999999999993/24"}
{"@timestamp": "2026-03-27 00:05:24,579", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available -297.22265625/24026.77734375"}
{"@timestamp": "2026-03-27 00:05:24,579", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Scaling service Safelist: 0 -> 1"} 
{"@timestamp": "2026-03-27 00:05:24,579", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Scaling service Hashlookup: 0 -> 1"}
{"@timestamp": "2026-03-27 00:05:24,588", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "New target for Hashlookup: 1 -> 1"}
{"@timestamp": "2026-03-27 00:05:24,589", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "New target for Safelist: 1 -> 1"}
{"@timestamp": "2026-03-27 00:05:25,934", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Preparing environment for Hashlookup"}
{"@timestamp": "2026-03-27 00:05:25,943", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Launching Hashlookup dependency hashlookup_redis"}
{"@timestamp": "2026-03-27 00:05:25,943", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Killing stale container..."}
{"@timestamp": "2026-03-27 00:05:26,168", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available -297.22265625/24026.77734375"} 
{"@timestamp": "2026-03-27 00:05:26,182", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Preparing environment for Safelist"}
{"@timestamp": "2026-03-27 00:05:26,200", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Updating deployment information for Safelist"}
{"@timestamp": "2026-03-27 00:05:26,243", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 2.1999999999999993/24"} 
{"@timestamp": "2026-03-27 00:05:28,077", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "INFO", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Deployment information for Safelist replaced"}

... the following repeats forever:

{"@timestamp": "2026-03-27 00:07:47,121", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Safelist"}
{"@timestamp": "2026-03-27 00:07:47,121", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Instances \t0 < 1 | 0 < 10"}
{"@timestamp": "2026-03-27 00:07:47,121", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Pressure \t-20.0 < 3.243416490252569 < 60.0"}  
....
{"@timestamp": "2026-03-27 00:07:47,174", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 4.199999999999999/24"}
{"@timestamp": "2026-03-27 00:07:47,301", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available 470.77734375/24026.77734375"}
{"@timestamp": "2026-03-27 00:07:47,301", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Can't make more because not enough resources {'Safelist': (1.0, 512)}"}
{"@timestamp": "2026-03-27 00:07:50,434", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total Memory available 470.77734375/24026.77734375"}
{"@timestamp": "2026-03-27 00:07:50,492", "event": { "module": "assemblyline", "dataset": "assemblyline.scaler" }, "host": { "ip": "x.x.x.x", "hostname": "c6b869472efc" }, "log": { "level": "DEBUG", "logger": "assemblyline.scaler" }, "process": { "pid": "1" }, "message": "Total CPU available 4.199999999999999/24"}   

Running the same sample through 4.6.2.1 (which is my "latest" snapshot that the above was upgraded from), config is the exact same, logs are basically the same, except the scaler will gladly just keep going with negative resources instead of getting hung up. It'll log "Can't make more because not enough resources" then proceed to scale that service anyway.

There seems to be some threshold at which it'll wait for services to scale down before scaling others, but that's well below 100% CPU/RAM actual utilization. OOM reaper doesn't terminate any containers, system stays responsive. I set overallocation to 0 long ago since I was getting OOM'd tasks, based on the (flawed) understanding that this config was meant for use within an auto-scaling cluster - 0 being do not overallocate, this isn't a cluster.

I'll give overallocation of 1 a shot on 4.6.2.1 and 4.7.1.12 and see how that behaves, if the system hangs/tasks get reaped.

[ppil]

Oddly, with overallocation set to 1, scaler@4.6.2.1 won't scale safelist just as 4.7.x does above. It just hangs there... When it's 0 then it runs.

4.7.1.12 with overallocation set to 1 hangs up just as it does with 0. No difference... So basically, 4.6.x misconfigured is the only way to get a full run through without stalling or having processes get OOM'd?

[ppil]

Replaced docker_ctl.py in the scaler image with a copy that uses actual memory usage (container.stats[memory_stats][usage] instead of container.attrs['HostConfig']['Memory'] and 4.6.2.1 processes submissions without issue given overallocation set to 1 now. Tried the same on 4.7.1.12, and that also works without issue.

diff --git a/assemblyline_core/scaler/controllers/docker_ctl.py b/assemblyline_core/scaler/controllers/docker_ctl.py
index 5dcf559..4f0d5df 100644
--- a/assemblyline_core/scaler/controllers/docker_ctl.py
+++ b/assemblyline_core/scaler/controllers/docker_ctl.py
@@ -355,7 +355,14 @@ class DockerController(ControllerInterface):
         mega = 2**20
         total_mem = mem = self._info['MemTotal']/mega - self._reserved_mem
         for container in self.client.containers.list(ignore_removed=True):
-            mem -= container.attrs['HostConfig']['Memory']/mega
+#            mem -= container.attrs['HostConfig']['Memory']/mega
+            memattr = container.attrs['HostConfig']['Memory']
+            try:
+                stats = container.stats(stream=False)
+            except Exception as e:
+                self.log.debug(f'Failed to grab container stats: {e}')
+                stats = {}
+            mem -= stats.get('memory_stats', {}).get('usage', memattr)/mega
         self.log.debug(f'Total Memory available {mem}/{self._info["MemTotal"]/mega}')
         return mem, total_mem

It would probably go faster doing the same for CPU, but would need other logic in there to get a (n_cpu * pct_used)/n_cpu type of metric to keep the return values sane. Might play with that later.

I'm not sure about the reasoning behind calculating "reserved" cpu/memory vs actual utilization, but actual utilization seems like the better choice for my setup. Maybe we can make this a config boolean if reservation is needed for some deployments.

[ppil]

Re-wrote the cpu_info function as well to use container stats, falling back on quota attrs when stats fail. Working great so far, at least on this monolithic deployment. Inclined to start increasing service limits to speed things up now that I can see how resources are being managed, rather than fear OOM reaping and a stalled scaler. Could simplify this by just looking at the overall host usage using psutil/etc, but this keeps the same dependencies and general core reservation logic.

docker_ctl.patch