pub sub framework

Introduction

Most of osquery's virtual tables are generated when an SQL statement requests data. For example, the time gets the current time and returns it as a single column. So whenever a call selects data from time, e.g., SELECT * from time; the current time of the call will return.

From an operating systems perspective, query-time synchronous data retrieval is lossy. Consider the processes table: if a process like ps runs for a fraction of a moment there's no way SELECT * from processes; will ever include the details.

To solve for this osquery exposes a pubsub framework for aggregating operating system information asynchronously at event time, storing related event details in the osquery backing store, and performing a lookup and report at query time. This reporting pipeline is much more complicated than typical query-time virtual table generation. The time of event, storage history, and applicable (final) virtual table data information must be carefully considered. As events occur the rows returned by a query will compound, as such selecting from an event-based virtual table generator should always include a time range.

Architecture

An osquery event publisher is a combination of a threaded run loop and event storage abstraction. The publisher loops on some selected resource or uses operating system APIs to register callbacks. The loop or callback introspects on the event and sends it to every appropriate subscriber. An osquery event subscriber may instruct a publisher, save published data, and must react to a query by returning appropriate data.

The pubsub runflow is exposed as a publisher setUp(), a series of addSubscription(const SubscriptionRef) by subscribers, a publisher configure(), and finally a new thread scheduled with the publisher's run() static method as the entrypoint. For every event the publisher receives it will loop through every Subscription and call fire(const EventContextRef, EventTime) to send the event to the subscriber.

Example: inotify

Filesystem events are the simplest example, let's consider Linux's inotify framework. osquery/events/linux/inotify.cpp is exposed as an osquery publisher.

There's an array of yet-to-be-implemented uses of the inotify publisher, but a simple example includes querying for every change to /etc/passwd. The osquery/tables/events/linux/passwd_changes.cpp table uses a pubsub subscription and implements a subscriber.