bump backend version + offloading documentation

Author: Pascal-Delange

.env.example

@@ -156,3 +156,16 @@ OPENSANCTIONS_API_HOST=http://yente
 # The two pieces of information below will be provided to you after that.
 # FIREBASE_APP_ID=
 # FIREBASE_AUTH_DOMAIN=
+
+# Set to true to offload decision rules content to blob storage.
+# This will enable a background task moving the rule payloads that are older than `OFFLOADING_BEFORE`
+# to blob storage to improve database efficiency, while keeping retention requirements.
+# If enabled, OFFLOADING_BUCKET_URL (see syntax above along with CASE_MANAGER_BUCKET_URL) must be set on the API and worker,
+# while the other variables below need only be set on the worker.
+# OFFLOADING_ENABLED=false
+# OFFLOADING_BUCKET_URL="file://./tempFiles/offloading-bucket?create_dir=true"
+# OFFLOADING_JOB_INTERVAL=30m
+# OFFLOADING_BEFORE=168h
+# OFFLOADING_BATCH_SIZE=10000
+# OFFLOADING_SAVE_POINTS=1000
+# OFFLOADING_WRITES_PER_SEC=200

api

@@ -1,7 +1,7 @@
 name: marble
 
 x-backend-image-version: &backend-image-version
-  image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-backend:v0.52.0
+  image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-backend:v0.52.1
 x-frontend-image-version: &frontend-image-version
   image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-frontend:v0.52.0
 

docker-compose-dev.yaml

@@ -1,7 +1,7 @@
 name: marble
 
 x-backend-image-version: &backend-image-version
-  image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-backend:v0.52.0
+  image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-backend:v0.52.1
 x-frontend-image-version: &frontend-image-version
   image: europe-west1-docker.pkg.dev/marble-infra/marble/marble-frontend:v0.52.0
 
@@ -47,6 +47,14 @@ x-backend-environment: &backend-env
   OPENSANCTIONS_API_HOST: ${OPENSANCTIONS_API_HOST}
   OPENSANCTIONS_AUTH_METHOD: ${OPENSANCTIONS_AUTH_METHOD}
   OPENSANCTIONS_API_KEY: ${OPENSANCTIONS_API_KEY}
+
+  OFFLOADING_ENABLED: ${OFFLOADING_ENABLED}
+  OFFLOADING_BUCKET_URL: ${OFFLOADING_BUCKET_URL}
+  OFFLOADING_JOB_INTERVAL: ${OFFLOADING_JOB_INTERVAL}
+  OFFLOADING_BEFORE: ${OFFLOADING_BEFORE}
+  OFFLOADING_BATCH_SIZE: ${OFFLOADING_BATCH_SIZE}
+  OFFLOADING_SAVE_POINTS: ${OFFLOADING_SAVE_POINTS}
+  OFFLOADING_WRITES_PER_SEC: ${OFFLOADING_WRITES_PER_SEC}
 x-frontend-environment: &frontend-env
   NODE_ENV: production
   PORT: ${APP_PORT:-8080}

docker-compose.yaml

@@ -0,0 +1,21 @@
+## Decision rule execution details offloading
+
+Out of the box, Marble keeps a full audit trace of you decisions by storing not only the result of a rule execution, but also all the intermediate values computed during rule excution from ingested data, aggregates, etc. This allows you to to justify even past decisions in detail, and is also used by the AI agent to review alerts in the case manager in detail.
+
+However, storing those details can quickly create pressure on the database disk as the data is initially stored inline.
+
+### Basic configuration
+
+It is possible to configure Marble to offload this data for a long-term storage outside of the main database, in a bucket storage solution like S3 or GCS. You do this by configuring the environment variables:
+
+- on the background worker: `OFFLOADING_ENABLED` to `true` and `OFFLOADING_BUCKET_URL` to a valid URL of a storage bucket
+- on the API server: `OFFLOADING_BUCKET_URL`
+
+The `OFFLOADING_JOB_INTERVAL`, `OFFLOADING_BEFORE`, `OFFLOADING_BATCH_SIZE`, `OFFLOADING_SAVE_POINTS` and `OFFLOADING_WRITES_PER_SEC` may further be configured on the background worker, but the default values should work in most cases. See the Marble deployment guide on Github for more details on those variables. By default, Marble will offload data on decisions older than a week.
+
+### Storage classes
+
+Old decisions are likely to be rarely consulted. For further cost savings, we recommend you configure your blob storage bucket to automatically move the data to long-term storage classes using lifecycle rules. Objects are stored by decision status prefix (`offloading/decision_rules/{status}` where `status` is one of `error`, `hit`, `no_hit`), so it is possible in most blob storage solutions to configure different lifecycle rules for rules that did or did not result in a hit.
+
+Data offloaded to Google Cloud Storage may further use the custom time metadata attribute instead of the object's creation time for lifecycle rules.
+As an example, Marble's managed environment moves objects to "Nearline" storage class after a month, "Coldline" after 3 months and "Archive" after a year.

installation/data_offloading.md

@@ -52,6 +52,8 @@ Purpose:
 
 - Store CSV files for batch ingestion
 - Store documents for case management
+- Offload detailed data on rule execution from the main database, which would otherwise use up a lot of storage volume for nothing
+- Parquet data storage for analytics queries
 
 Supported Providers:
 
@@ -63,6 +65,8 @@ Supported Providers:
 Configuration details are available in `.env.example`.
 The .env file expects two different buckets for the two purposes, but you may use the same bucket for both.
 
+> ⚠️ **Note**: We strongly suggest you configure data offloading to run in production. See [data offloading](./data_offloading.md)
+
 ### 3. Firebase Authentication
 
 #### Setup Steps

installation/production_run.md

@@ -1,2 +1,2 @@
 MARBLE-FRONT-VERSION=0.52.0
-MARBLE-BACK-VERSION=0.52.0
+MARBLE-BACK-VERSION=0.52.1