Collect logs from MacOS' Unified Logging system

[Japillow]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

MacOS Unified Logging system

This Issue is here to track community request for the implementation of Unified Logging system streaming in Vector for MacOS.

In 2016, Apple moved away from file-based logging to it's own database-based Unified Logging system¹

Use Cases

Unified Logging system logs are historically extensively used in an ad hoc manner by Incident Responders, as shown by:

• Mendiant's parser (https://github.com/mandiant/macos-UnifiedLogs),
• Crowdstrike and SentinelOne articles.

However, querying and streaming the Unified Logging system has also number of non-IR use cases, as outlined in mac4n6's numerous blog posts, in Jamf examples, but also more generally when Mac apps or scripts (Python, shell) use the official Apple Logging APIs.

Attempted Solutions

Based on my research the following systems implement collection of Apple Unified Logs on MacOS

• Commercial systems
  • MDM Vendor:
    • Jamf Protect EDR implements Unified Logs collection and streaming (including to Datadog): https://learn.jamf.com/en-US/bundle/jamf-protect-documentation/page/Unified_Logging.html
    • Other MDM solutions such as Kandji, Addigy, etc., have not implemented Unified Logs collection.
  • Log management platform agents:
    • Splunk: Universal Forwarder with logd and Add-ons ([1] [2])
    • Elastic: Custom macOS Unified Logs (and previously macoslogbeat)
    • Afaik, no other commercial log management platform (including Datadog) support Unified Logs streaming.
• Open-source:
  • Wazuh Agent implements sending to Wazuh server (source), but this requires setting up a Wazuh server.
  • Afaik, no open source project implements macOS Unified Logs streaming to generic outputs (syslog, webhooks, or else). This is where Vector's integration of this feature, as an OSS observability tool that is SIEM-independent, could be very valuable.

For now, apart from Jamf Protect, Splunk Universal Forwarder, Elastic Agent and Wazuh agent, streaming logs from the Unified logging system on MacOS requires:

• Creating custom LaunchDaemons for each predicate filter, to stream to a file (example log exporter)
• then using file-based log collection agent on MacOS to stream to a destination.
  • [Datadog Note] This is a non-exhaustive list of open-source agents that support sending logs to Datadog Log Management:
    • Vector - Datadog Sink,
    • Fluentbit - Datadog output,
    • Fluentd - Datadog plugin)

osquery also allows querying the Unified Logs (which can be setup at regular intervals), but that is nowhere near as efficient as realtime log streaming.

Proposal

Notes on implementation:

• As Apple Unified log is verbose, implementation requires adding support for Predicate Filter

References

• See discussion in Support for MacOS Unified Logging  elastic/integrations#6589 with arguments for and against implementation when looking at it from a security standpoint (especially regarding osquery + MacOS EndpointSecurity API), but this Feature request is more about logs that may be generated by any application who log using MacOS's Logging APIs,
• See https://github.com/jaakkoo/macoslogbeat/blob/master/beater/MacosLogBeat.go for an example implementation in Go of Unified Log streaming.
• See https://github.com/mandiant/macos-UnifiedLogs for an example implementation in Rust of an Unified logs parser used for point-in-time forensics.

Version

v0.50.0

Footnotes

1. The Unified Logging system is sometimes unofficially abbreviated as ULS or AUL. However, contrary to the previous Apple System Logger (ASL), Apple never uses the abreviations ULS, AUL, or the terms Unified Logs or Unified logging. ↩