ML-assisted severity scoring (learned, context-aware risk_score)

[sercanokur]

Is your feature request related to a problem? Please describe.

Malcolm's current severity/risk scoring (logstash/pipelines/enrichment/23_severity.conf

• logstash/maps/malcolm_severity.yaml) works by attaching string tags to
  event.severity_tags and then summing static integer weights per tag,
  capped at 100, in the ruby_calculate_final_severity_score block.

This additive, static approach has a few inherent limitations:

1. No interaction effects. The score is a flat sum, so high-signal
   combinations (e.g. Outbound traffic + High entropy domain +
   Service on non-standard port + executable file transfer — a classic C2
   pattern) score the same as the sum of their parts, even though together
   they are far more suspicious.
2. Global, context-free thresholds. TOTAL_MEGABYTES_SEVERITY_THRESHOLD,
   FREQ_SEVERITY_THRESHOLD, etc. are the same for every host. A 1 GB transfer
   is normal from a backup server but alarming from an OT PLC. Asset/segment
   context (already available via NetBox enrichment) is not factored in.
3. No analyst feedback loop. False-positive/true-positive dispositions made
   by analysts do not influence future scoring; weights are tuned by hand.
4. Rarity-blind. A tag combination never seen before in the environment
   scores identically to one seen thousands of times per day.

Describe the solution you'd like

Keep the existing tag-generation logic untouched (it is effectively a rich,
interpretable feature-extraction layer) and augment/replace only the final
scoring step with an optional, opt-in ML model. event.severity_tags
(multi-hot) plus a few numeric fields (freq scores, network.bytes,
destination.port, network.direction, asset role) become the feature vector.

Proposed phased rollout:

• Phase 1 — unsupervised, zero-label cold start. A rarity/anomaly model
  (e.g. Isolation Forest) over tag combinations that raises severity for
  unusual combinations, while the existing static sum remains a floor so there
  is never a regression. Works fully offline / air-gapped with no labels.
• Phase 2 — supervised, calibrated model. Once analyst dispositions accrue
  (a simple verdict field convention) a calibrated gradient-boosted model
  (e.g. LightGBM exported to ONNX) outputs a 0–100 risk probability, with
  per-tag contribution output (SHAP) so analysts can see why a score was
  assigned. Weak supervision (YARA/ClamAV signatures, Intelligence hits,
  high-severity Suricata alerts as positives) can bootstrap before labels exist.

Suggested integration, mirroring existing patterns to minimize disruption:

• A new lightweight scoring microservice modeled on freq-server
  (FastAPI + ONNX runtime, model bundled into the image — no external calls).
• A new Logstash ruby filter modeled on logstash/ruby/freq_lookup.rb
  (HTTP call + LRU cache) invoked at the end of 23_severity.conf.
• A single service entry in docker-compose.yml / docker-compose-dev.yml
  and a Dockerfile under Dockerfiles/.
• Gated behind a new env var (e.g. LOGSTASH_SEVERITY_ML, default false),
  so existing behavior is preserved exactly when disabled.

Describe alternatives you've considered

• Manual weight tuning only — does not address interactions, context, or
  feedback; current state.
• A separate standalone anomaly-detection pipeline — higher value for some
  use cases (beaconing, ICS baselining) but a larger surface area and does not
  directly improve the existing risk_score analysts already rely on.
• Heavy deep-learning models — rejected due to Malcolm's frequent
  air-gapped, laptop-to-server deployment footprint; lightweight CPU models
  (Isolation Forest / GBM via ONNX) fit far better.

Additional context

Design goals specific to Malcolm: must run fully offline/air-gapped (model
bundled, no cloud), be lightweight enough for laptop-class deployments, remain
explainable for SOC analysts (preserve the existing severity_tags UX and emit
per-tag contributions), and be fully backward-compatible (opt-in).

Question for the maintainers: Does this direction align with the project's
roadmap and design philosophy for severity scoring? Are there constraints or
preferences (e.g. on adding a new container, model formats, or how analyst
feedback should be captured) we should design around?

If the maintainers think this is a good fit, we are interested in working on it
— starting with a Phase 1 proof-of-concept (the rarity-based scorer microservice

• ruby filter + opt-in integration) for review before going further.

[mmguero]

My team and I read your proposal, and we're open to your idea, given some caveats (a few of which you've mentioned):

• existing resources: If you haven't already, I do recommend you look at the random cut forrest anomaly detection plugin already built into opensearch, just to see if there's anything there either to leverage or that overlaps with either of your phases. Also you could look at the ML commons plugin. Just want to make sure you've aware of existing resources that may help.
• model licensing: would there be any licensing restrictions related to the model? Anything we ship in the project needs to be true open source licensed. I assume the model would be pulled from some public repository during build time? I don't have a ton of practical AI/ML experience yet, so I don't have strong feelings about model formats.
• opt-in integration: you mentioned this, and we agree, whatever this ends up being would need to be off by default for performance reasons
• new containers: I'm not opposed to adding new containers. Unless there are strong reasons not to, I'd recommend either debian:13-slim or alpine:3.23 as the base, if possible, just because that's what most of our existing containers are based on. But that's not a hard-and-fast requirement. If you want to just work on the docker-based deployment first, I'd be able to take it over and do the kubernetes template/chart stuff once we get ready to merge if you didn't want to worry about that at first.
• documentation: we've currently got docs/severity.md, before we merge this in we'd want to make sure we've got good documentation covering your new feature.
• performance: this is my main question, and I understand it's likely one that we'll have to see once you've got some proof-of-concept worked up. For high insertion rates, will even a fast model be fast enough to not seriously gate insertion? Perhaps depending on how selective we are about what triggers these lookups. A command like this can help you profile how long your filter takes compared to the other filters in the pipeline.

I think that's about it. Happy to have other discussions about it here if you'd like to.