django-elasticsearch-metrics ("djelme" for short)

Django app for storing time-series metrics in Elasticsearch.

django-elasticsearch-metrics on pypi: https://pypi.org/project/django-elasticsearch-metrics

python importables:

• elasticsearch_metrics
• elasticsearch_metrics.imps.elastic8
• elasticsearch_metrics.imps.elastic6
• ...

Pre-requisites

• Python >=3.10
• Django 4.2, 5.1, or 5.2
• Elasticsearch 8 (or 6, for deprecated back-compat)

Install

pip install django-elasticsearch-metrics

Quickstart

Add "elasticseach_metrics" to INSTALLED_APPS.

# ... in your django project settings.py ...
INSTALLED_APPS += ["elasticsearch_metrics"]

Configure at least one djelme backend:

# ... in your django project settings.py ...
DJELME_BACKENDS = {
    "my-es8-backend": {  # a name from and for you 
        "elasticsearch_metrics.imps.elastic8": {  # importable djelme implementation
            # dictionary of kwargs for the imp's `djelme_backend` constructor
            # (in this case passed thru as kwargs to `elasticsearch8.Elasticsearch`)
            "hosts": "https://my-elastic8.example:9200",
        },
    },
}

In one of your apps, add record types in metrics.py

# myapp/metrics.py

from elasticsearch_metrics.imps.elastic8 import EventRecord


class UsageRecord(EventRecord):
    item_id: int

    class Meta:
        djelme_backend = "my-es8-backend"  # optional if only one backend

Either enable autosetup...

# ... in your django project settings file ...
DJELME_AUTOSETUP = True

...or be sure to run the djelme_backend_setup management command before trying to store anything.

# This will create an index template for usagerecord timeseries indexes
python manage.py djelme_backend_setup

Now add some data:

from myapp.metrics import UsageRecord

# By default we create an index for each day.
# Therefore, this will persist the document
# to an index named for the record type and date
UsageRecord.record(item_id='my.item.id')

Go forth and search!

# search across all timeseries indexes -- get an `elasticsearch8.dsl.Search` object
UsageRecord.search()

Timeseries indexes

By default, behind the scenes, a new elasticsearch index is created for each record type for each month in which a record is saved (using UTC timezone). You can set a default change the per-index timespan by setting Meta.timedepth on the record type.

• index per day, '...YYYY_MM_DD...': timedepth = 3
• index per month, '...YYYY_MM...': timedepth = 2
• index per year, '...YYYY...': timedepth = 1

Index settings

You can configure the index template settings by setting Index.settings on a record type.

class UsageRecord(EventRecord):
    item_id: int

    class Index:
        settings = {"number_of_shards": 2, "refresh_interval": "5s"}

Index templates

Each record type will have its own index template. The index template name and glob pattern are computed from the app label for the containing app and the class's name. For example, a UsageRecord class defined in myapp/metrics.py will have an index template with the name myapp_usagerecord and a template glob pattern of myapp_usagerecord_*.

If you declare a record type outside of an app, you will need to set app_label.

class UsageRecord(EventRecord):
    class Meta:
        app_label = "myapp"

Alternatively, you can set template_name and/or template explicitly.

class UsageRecord(EventRecord):
    item_id: int

    class Meta:
        template_name = "myapp_pviews"
        template = "myapp_pviews_*"

Abstract record types

from elasticsearch_metrics.imps.elastic8 import EventRecord


class MyBaseMetric(EventRecord):
    item_id: int

    class Meta:
        abstract = True


class UsageRecord(MyBaseMetric):
    class Meta:
        app_label = "myapp"

Optional factory_boy integration

import factory
from elasticsearch_metrics.factory import MetricFactory

from ..myapp.metrics import MyMetric


class MyMetricFactory(MetricFactory):
    my_int = factory.Faker("pyint")

    class Meta:
        model = MyMetric


def test_something():
    metric = MyMetricFactory()  # index metric in ES
    assert isinstance(metric.my_int, int)

Configuration

• DJELME_BACKENDS: Named backends for storing or searching records from your django app -- nested mapping from backend name (any string, your choice) to python-importable paths for modules that (like "elasticsearch_metrics.imps.elastic8") to "imp kwargs" config dictionaries given to the imp module's djelme_backend constructor

  # ... in your django project settings.py ...
  DJELME_BACKENDS = {
      "my-es8-backend": {  # a name from and for you 
          "elasticsearch_metrics.imps.elastic8": {  # importable djelme implementation
              # dictionary of kwargs for the imp's `djelme_backend` constructor
              # (in this case passed thru as kwargs to `elasticsearch8.Elasticsearch`)
              "hosts": "https://my-elastic8.example:9200",
          },
      },
  }

• DJELME_AUTOSETUP: Optional feature, default False -- set True to run backend setup automatically when your django app starts (like creating index templates in elasticsearch, if they don't already exist)

• DJELME_DEFAULT_TIMEDEPTH: Set the granularity of timeseries indexes by the number of "time parts" in index names

  DJELME_DEFAULT_TIMEDEPTH = 1  # yearly indexes; YYYY
  DJELME_DEFAULT_TIMEDEPTH = 2  # monthly indexes; YYYY_MM
  DJELME_DEFAULT_TIMEDEPTH = 3  # daily indexes; YYYY_MM_DD (this is the default)
  DJELME_DEFAULT_TIMEDEPTH = 4  # hourly indexes; YYYY_MM_DD_HH
  

  you can also set Meta.timedepth on a specific record type; this will take precedence

Management commands

• djelme_backend_types: Pretty-print a listing of all registered record types.
• djelme_backend_setup: Ensure that index templates have been created for your record types.
• djelme_backend_check: Check if index templates are in sync. Exits with an error code if any templates are out of sync.

Signals

Signals are located in the elasticsearch_metrics.signals module.

• pre_index_template_create(Metric, index_template, using): Sent before PUTting a new index template into Elasticsearch.
• post_index_template_create(Metric, index_template, using): Sent after PUTting a new index template into Elasticsearch.
• pre_save(Metric, instance, using, index): Sent at the beginning of a Metric's save() method.
• post_save(Metric, instance, using, index): Sent at the end of a Metric's save() method.

Caveats

• _source is disabled by default on metric indices in order to save disk space. For most metrics use cases, Users will not need to retrieve the source JSON documents. Be sure to understand the consequences of this: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#_disabling_source . To enable _source, you can override it in class Meta.

class MyMetric(EventRecord):
    class Meta:
        source = metrics.MetaField(enabled=True)

Resources

• Elasticsearch as a Time Series Data Store
• Pythonic Analytics with Elasticsearch
• In Search of Agile Time Series Database

License

MIT Licensed.