docs: expand setup and search guidance

Author: notesofcliff

doc/admin/Configuration.md

@@ -26,6 +26,18 @@ DATABASES = {
 
 **NOTE**: The second argument to `os.getenv` is for the default value if the environment variable is not set.
 
+## Configuration with docker-compose
+When running Delve with Docker, configuration values are typically supplied via
+environment variables defined in an `.env` file.  The project includes a
+`docker-compose.yaml` and a corresponding `.env.example` file.  Copy the example
+file to `.env`, update the values for your database and secret key, and
+`docker-compose` will inject those settings into the containers at startup.
+
+The standard `delve/settings.py` already looks for these environment variables,
+so no additional changes are required to the image.  This approach keeps secrets
+out of the settings file while still allowing easy configuration for different
+environments.
+
 ## Configuring Email Settings
 Django provides a flexible email backend system that allows you to configure various email settings. Here are the steps to configure email settings in `settings.py`:
 

doc/admin/Ingesting_Data.md

@@ -25,7 +25,11 @@ This endpoint is provided to allow the resolution of queries.
 
 The `/api/query` endpoint is what is used to power the Explore UI which is available at `/explore`.
 
-The browsable REST API, which is returned when accessing `/api/query` from a browser can be really useful for debugging if you enable `DEBUG` in your `settings.py` as it allows you to track the SQL statements issued by search commands as well as a ton of other useful information.
+The browsable REST API, which is returned when accessing `/api/query` from a
+browser, can be really useful for debugging when the Django debug toolbar is
+enabled. Setting `DEBUG=True` and installing the toolbar allows you to inspect
+the SQL statements issued by search commands along with a wealth of other
+diagnostic information.
 
 ### Example Usage
 

doc/admin/Installation_and_Setup.md

@@ -106,7 +106,23 @@ python service.py /?
 ```
 
 ### Installing Delve Supervisor Service on Other Platforms
-This feature is coming soon. Until then, please consult your operating system's documentation for information on hosting services (Systemd, etc.) to have Delve automatically started and monitored.
+On non-Windows platforms the recommended approach for running Delve is to use
+Docker and `docker-compose`.  The repository includes a
+`docker-compose.yaml` that defines services for the web server, background
+worker, and a PostgreSQL database.  These containers replace the need for a
+separate supervisor service by handling start-up and restart logic for you.
+
+To get started, copy the provided `.env.example` file to `.env` and adjust the
+settings for your environment. Then launch the stack:
+
+```bash
+cp .env.example .env
+docker compose up -d
+```
+
+`docker-compose` will orchestrate the containers and keep them running, making
+it an easy way to deploy Delve on Linux and macOS without manually configuring
+init systems like Systemd.
 
 
 [Previous: Introduction](/doc/admin/Introduction.md) | [Next: Bootstrap Guide](/doc/admin/Bootstrap_Guide.md)

doc/admin/Performance_Tuning.md

@@ -45,6 +45,22 @@ class MyCustomModel(models.Model):
     datetime_field = models.DateTimeField(db_index=True)
 ```
 
+### BRIN Indexes in PostgreSQL
+For very large tables in PostgreSQL, a [BRIN index](https://www.postgresql.org/docs/current/brin-intro.html)
+can dramatically reduce index size while still allowing efficient scans of
+time-ordered data. Delve stores event timestamps in the `created` field of the
+`events.Event` model, making it a good candidate for a BRIN index:
+
+```sql
+CREATE INDEX CONCURRENTLY events_event_created_brin
+    ON events_event USING BRIN (created);
+```
+
+You can also create this index via a Django migration using
+`django.contrib.postgres.indexes.BrinIndex`. BRIN indexes are especially useful
+when events are inserted roughly in chronological order, which is typical for
+log data.
+
 ## Caching Strategies
 Caching can help reduce the load on your database and improve response times. Django provides several caching backends, including in-memory caching, file-based caching, and more.
 

doc/user/Getting_Started.md

@@ -29,14 +29,23 @@ Delve is a versatile and powerful platform for ingesting, transforming, and sear
 Delve is built around several core concepts that enable its powerful functionality:
 
 ### Events
-Events are the basic unit of data in Delve. They are stored in a database and consist of indexed fields such as `created`, `index`, `source`, and `host`. Additionally, events contain a JSON field called `extracted_fields` for storing information extracted from the event.
+Events are the basic unit of data in Delve. They are stored in a database and
+consist of indexed fields such as `created`, `index`, `source`, and `host`.
+Additionally, events contain a JSON field called `extracted_fields` for storing
+information extracted from the event.
 
-Events can be created through the Delve REST API or via queries using search commands, either interactively or on a scheduled basis.
+Events can be created through the Delve REST API or via queries using search
+commands, either interactively or on a scheduled basis. If you are writing code
+against Delve itself, you may also create `Event` instances directly using the
+Django ORM.
 
 ### Queries
 Queries are used to retrieve, transform, and store data. They can be ephemeral or persisted in the database. A query's `text` field defines the pipeline through which events are processed using search commands.
 
-Queries use a pipeline syntax where commands are chained together with the `|` operator, similar to command-line shells. For example:
+Queries use a pipeline syntax where commands are chained together with the `|`
+operator, similar to command-line shells. All whitespace in a query is collapsed
+so breaking a pipeline across multiple lines is perfectly fine and can aid
+readability. For example:
 
 ```bash
 search --last-15-minutes index=default
@@ -54,16 +63,30 @@ Ingestion is the process of importing data into Delve. This can be done through
 - **Searches**: Use interactive or scheduled queries to ingest data.
 
 ### Field Extraction and Preprocessing
-Field extraction identifies and extracts specific fields from data. Delve supports:
-- **Index-time Extractions**: Applied during data ingestion and persisted in the database.
-- **Search-time Extractions**: Dynamically extract fields during a search (can be persisted to the database or ephemeral).
-- **Preprocessing**: Take action (email, log, etc.) on certain events during ingestion.
+Field extraction identifies and extracts specific fields from data. Delve
+supports:
+- **Index-time Extractions**: Applied during data ingestion and persisted in the
+  database. Update `DELVE_EXTRACTION_MAP` in `settings.py` to map sourcetypes to
+  the appropriate extraction functions.
+- **Search-time Extractions**: Dynamically extract fields during a search (can
+  be persisted to the database or ephemeral). The `qs_update` search command can
+  be used to update events at search time after new fields are extracted.
+- **Preprocessing**: Take action (email, log, etc.) on certain events during
+  ingestion. Register processor functions in `DELVE_PROCESSOR_MAP` in
+  `settings.py` based on the event sourcetype.
 
 ### Search
-Search functionality allows querying and filtering data, as well as performing transformations, visualizations, and more. Delve supports custom search commands written in Python, which can be registered in the configuration.
+Search functionality allows querying and filtering data, as well as performing
+transformations, visualizations, and more. Delve supports custom search
+commands written in Python, which can be registered in the configuration. The
+`DELVE_SEARCH_COMMANDS` setting in `settings.py` can be updated to add or remove
+available search commands.
 
 ### Custom Apps
-Delve enables the creation of custom apps to group related data and code. These apps can include custom search commands, dashboards, REST API endpoints, and more.
+Delve enables the creation of custom apps to group related data and code. These
+apps can include custom search commands, dashboards, REST API endpoints, and
+more. Custom apps are added to the `INSTALLED_APPS` setting in `settings.py` so
+that Django knows to load them.
 
 ### Alerts
 Delve provides search-based and processor-based alerts:

doc/user/Searching_Filtering_and_More.md

@@ -25,7 +25,13 @@ Queryset transformation commands act on Django Querysets (such as the output of
 
 ## Generic Transformations
 
-Generic transformation commands modify data from a source command in some way. These commands are designed to be versatile and work well with diverse sets of data, although they may be slower than their `qs_` counterparts.
+Generic transformation commands modify data from a source command in some way.
+While Queryset Transformations accept and return Django `QuerySet` objects,
+Generic Transformations are designed to accept and return arbitrary Python data
+structures (`int`, `str`, `list`, `dict`, `tuple`, etc.). A list of dictionaries
+representing rows is a common pattern. These commands are versatile and work
+well with diverse sets of data, although they may be slower than their `qs_`
+counterparts.
 
 ### Example Commands
 
@@ -133,10 +139,17 @@ search --index {{ index }} text__icontains="{{ keyword }}"
 
 In this example, `{{ index }}` and `{{ keyword }}` are Jinja2 template variables that will be replaced with their corresponding values from the context.
 
-This can be useful for including different sets of arguments to the same search command to get different results or behavior. It can also be used with the `query_table` and `query_chart` templatetags which accept Django Form instances for use in dashboards and control panels.
+This can be useful for including different sets of arguments to the same search
+command to get different results or behavior. Combined with the `query_table`
+and `query_chart` templatetags, a Django `Form` instance can be rendered in a
+dashboard to collect user input and feed values directly into a query. This
+combination of forms, template tags, and Jinja2 templating makes it easy to
+build powerful interactive dashboards and control panels.
 
 ## send_email Command
-The `send_email` command allows you to send email notifications based on the result set. This command is configured using Django's SMTP settings. For example:
+The `send_email` command allows you to send email notifications based on the
+result set. This command is configured using Django's SMTP settings and only
+sends an email when the result set contains at least one event. For example:
 
 ```bash
 search --index default text__icontains="error" | send_email to="team@example.com" subject="Error Alert"
@@ -175,37 +188,37 @@ make_events --save
 ```
 
 ## Custom Search Commands
-Delve allows you to create custom search commands in Python and register them in `settings.py` under `DELVE_SEARCH_COMMANDS`. This provides powerful and flexible ways to manipulate the result set.
+To create a custom search command, first build an `argparse.ArgumentParser` to
+describe any arguments the command accepts. Then write a function that operates
+on the events and wrap it with the
+`events.search_commands.decorators.search_command` decorator, passing the parser
+instance. The decorator parses the arguments and supplies them to your
+function.
 
 ### Example Custom Command
-Here is an example of a custom search command:
 
 ```python
-# filepath: /delve/search_commands/custom_command.py
-def custom_command(events, **kwargs):
-    # Custom logic to process events
-    for event in events:
-        event['custom_field'] = 'custom_value'
-    return events
-```
+# filepath: /delve/search_commands/shout.py
+from argparse import ArgumentParser
+from events.search_commands.decorators import search_command
 
-Alternatively, you could write it as a generator which could possibly improve memory usage:
+parser = ArgumentParser(prog='shout')
+parser.add_argument('--field', required=True)
 
-```python
-# filepath: /delve/search_commands/custom_command.py
-def custom_command(events, **kwargs):
-    # Custom logic to process events
+@search_command(parser)
+def shout(events, field):
     for event in events:
-        event['custom_field'] = 'custom_value'
+        event[field] = event[field].upper()
         yield event
 ```
-To register the custom command, add it to `settings.py`:
+
+Register the custom command in `settings.py`:
 
 ```python
 # filepath: /delve/settings.py
 DELVE_SEARCH_COMMANDS = {
     ...
-    'custom_command': 'delve.search_commands.custom_command.custom_command',
+    'shout': 'delve.search_commands.shout.shout',
     ...
 }
 ```
@@ -238,6 +251,9 @@ There are some special functions that are particularly useful for searching, tra
   ```python
   qs_annotate transformed_key=KT('json_field__key__1__foobar')
   ```
+- `Func`: The base class for database functions from `django.db.models`. Use
+  this when calling functions like `Lower`, `Upper`, `Length`, `Trim`, `Cast`,
+  `Coalesce`, `Concat`, or `Now`.
 
 #### Available Functions