Split configuration over multiple files

[creideiki]

Our production botnet has grown quite a bit - we're now at 112 bots taking up 3200 lines of runtime.yaml. This is getting unwieldy, and I would like to have a way to break it into manageable chunks.

I wrote this simple patch:

--- a/intelmq/lib/utils.py
+++ b/intelmq/lib/utils.py
@@ -17,6 +17,7 @@
 """
 import base64
 import collections
+import glob
 import grp
 import gzip
 import importlib
@@ -220,6 +221,13 @@ def load_configuration(configuration_filepath: str) -> dict:
         with open(configuration_filepath) as fpconfig:
             try:
                 config = YAML(typ="unsafe", pure=True).load(fpconfig)
+                if isinstance(config, dict):
+                    paths = config.pop('__include', [])
+                    if not isinstance(paths, list):
+                        raise TypeError(f"{configuration_filepath}: value of __include must be a list")
+                    for include_path in paths:
+                        for include_file in glob.glob(include_path):
+                            config.update(load_configuration(include_file))
             except ScannerError as exc:
                 if "found character '\\t' that cannot start any token" in exc.problem:
                     fpconfig.seek(0)

which allows me to have a runtime.yaml containing:

__include:
  - /opt/intelmq/etc/runtime/things.yaml
  - /opt/intelmq/etc/runtime/stuff-*.yaml

and the actual bots configured in the files under /opt/intelmq/etc/runtime/.

This works for us, but I'm wondering if anyone else would be interested in it, so I should take the time to write documentation and tests, bikeshed over the name, etc..

I have not tested this with the manager web application, since we do not use it. From a quick look at the source code, it seems to run everything through intelmqctl, so it should continue to work.

This will definitely not work with intelmqctl writing new configuration, and making it so would probably be a lot of work, having to track which file each bot is configured in. This is also not something we use, since we install the configuration using Puppet, so we have to do any upgrades manually anyway. The automatic upgrade also does not preserve comments or whitespace, so my assumption would be that anyone using automatic upgrades is not worried about editing the file manually, and thus would not care about splitting it.

[sebix]

Hi @creideiki, thanks for the initiative and for sharing the snippet and your experience.

It looks similar to the discussion and thoughts that we had in #1026. It's also close to the idea we had with a runtime.conf.d directory, similar to what many server daemons already have.

I guess it needs a bit of consideration and coordination between users which approach(es) work(s) best for our different use-cases, is most flexible and understandable. One way to evaluate that could be the IEP process, I just fear it takes too much effort and time and in the end the idea just stalls.

ping @aaronkaplan

[creideiki]

It looks similar to the discussion and thoughts that we had in #1026.

Ah, I hadn't noticed that that issue also mentioned file inclusion. We're not particularly interested in that ticket's main thesis of templating the configuration file in IntelMQ itself, since we use Puppet for that.

It's also close to the idea we had with a runtime.conf.d directory, similar to what many server daemons already have.

A runtime.conf.d directory would work for my use case as well.

[aaronkaplan]

I think an IEP would be good, but - we need to act on it quickly (so that it does not stall) -> this is our responsibility ofc.
And also , there is a side-effect of writing it down: the IEP forces one to think things through a bit. This actually makes the code better then :)

I also like the runtime.conf.d approach very much in fact. It feels intuitive, basically what people expect from a Debian/Ubuntu /Linux system these days.

[kamil-certat]

I slso think the size of the runtime.yaml is a problem, I have several hundred bots and maintain the configuration via Ansible, which lets me template the config file.

But besides the configuration size, hundreds of bots are also difficult to manage/observe in IntelMQ Manager. I have an idea to introduce 'workflows' with separate files and separate views in IntelMQ Manager, so you could keep the related bots together but separated from others. It should be easier to maintain the relation bot<->file and potentially implement across the board.

[aaronkaplan]

Could we group them somehow? I assume the workflows idea is exactly that...

…

On 13.07.2026, at 10:04, kamil-certat ***@***.***> wrote: I slso think the size of the runtime.yaml is a problem, I have several hundred bots and maintain the configuration via Ansible, which lets me template the config file. But besides the configuration size, hundreds of bots are also difficult to manage/observe in IntelMQ Manager. I have an idea to introduce 'workflows' with separate files and separate views in IntelMQ Manager, so you could keep the related bots together but separated from others. It should be easier to maintain the relation bot<->file and potentially implement across the board. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[kamil-certat]

I think yes? My idea was that Manager could have additional workflow/group selectbox and this would then be used to ask for the right runtime and position file, so the code change would not be so big. CLI could also take workflow as additional parameter.

[aaronkaplan]

Kamil would the idea above with the runtime.conf.d/ work for you as well?

[kamil-certat]

It could solve one issue (big runtime.yaml), but would not solve the Manager capability limits