ConnectionMaster · pull · Jun 22, 2026 · Jun 22, 2026
diff --git a/sap_hana/README.md b/sap_hana/README.md
@@ -81,6 +81,17 @@ To learn how to set the port number for HANA tenant, single-tenant, and system d
    GRANT SELECT ON SYS_DATABASES.M_VOLUME_IO_TOTAL_STATISTICS TO DD_MONITOR;
    ```
 
+   To collect schema metadata for Data Quality features in Data Observability, grant select on the catalog and monitoring views that store schema, table, and column definitions. These are already covered by the `GRANT CATALOG READ` in step 2, so this is only needed if you skipped that grant:
+
+   ```shell
+   GRANT SELECT ON SYS.SCHEMAS TO DD_MONITOR;
+   GRANT SELECT ON SYS.M_TABLES TO DD_MONITOR;
+   GRANT SELECT ON SYS.TABLE_COLUMNS TO DD_MONITOR;
+   GRANT SELECT ON SYS.VIEWS TO DD_MONITOR;
+   GRANT SELECT ON SYS.VIEW_COLUMNS TO DD_MONITOR;
+   GRANT SELECT ON SYS.M_TABLE_STATISTICS TO DD_MONITOR;
+   ```
+
 4. Finally, run the following command to assign the monitoring role to the desired user:
 
    ```shell
@@ -121,6 +132,21 @@ To learn how to set the port number for HANA tenant, single-tenant, and system d
 
 3. [Restart the Agent][5].
 
+#### Schema collection
+
+The Agent can collect SAP HANA catalog metadata (schemas, tables, views, and columns) for Data Quality features in Data Observability. When the monitoring user has access to `SYS.M_TABLE_STATISTICS`, the Agent also collects row counts and last modification times for tables. Collection is disabled by default. To enable schema collection, ensure that the monitoring user can read the required views (see [Granting privileges](#granting-privileges)) and add the following block to your `sap_hana.d/conf.yaml` file:
+
+```yaml
+   collect_schemas:
+     enabled: true
+     collection_interval: 600
+     max_tables: 2000
+     max_views: 2000
+     max_columns: 500
+```
+
+See the [sample sap_hana.d/conf.yaml][4] for all available schema collection options, including `include_schemas` and `exclude_schemas`.
+
 ### Validation
 
 Run the [Agent's status subcommand][6] and look for `sap_hana` under the Checks section.

diff --git a/sap_hana/assets/configuration/spec.yaml b/sap_hana/assets/configuration/spec.yaml
@@ -94,6 +94,67 @@ files:
           example: false
           type: boolean
       - template: instances/tls
+      - name: collect_schemas
+        hidden: true
+        description: |
+          Configure collection of SAP HANA catalog metadata (schemas, tables,
+          views, and columns) for Data Quality features in Data Observability.
+        options:
+          - name: enabled
+            description: |
+              Enable collection of catalog metadata. When enabled, the Agent
+              queries `SYS.M_TABLES`, `SYS.VIEWS`, `SYS.TABLE_COLUMNS`, and
+              `SYS.VIEW_COLUMNS` and emits schema metadata payloads.
+            value:
+              type: boolean
+              example: false
+          - name: collection_interval
+            description: |
+              Set the schema collection interval (in seconds). Catalog data
+              changes slowly; 600s (10 min) is a reasonable default.
+            value:
+              type: number
+              example: 600
+          - name: max_tables
+            description: |
+              Maximum number of tables to collect per cycle across all schemas.
+            value:
+              type: integer
+              example: 2000
+          - name: max_columns
+            description: |
+              Maximum number of columns to collect per table.
+            value:
+              type: integer
+              example: 500
+          - name: max_views
+            description: |
+              Maximum number of views to collect per cycle across all schemas.
+            value:
+              type: integer
+              example: 2000
+          - name: include_schemas
+            description: |
+              A list of schema names to include. Any schema whose name is in
+              this list is included. If empty, all schemas (other than
+              those excluded) are included.
+            value:
+              type: array
+              items:
+                type: string
+              example:
+                - "MY_SCHEMA"
+          - name: exclude_schemas
+            description: |
+              A list of schema names to exclude. Any schema whose name is in
+              this list is excluded. SAP HANA system schemas are always
+              excluded regardless of this setting.
+            value:
+              type: array
+              items:
+                type: string
+              example:
+                - "TMP_SCHEMA"
       - template: instances/default
   - template: logs
     example:

diff --git a/sap_hana/benchmarks/schema_collection_memory/README.md b/sap_hana/benchmarks/schema_collection_memory/README.md
@@ -0,0 +1,102 @@
+# Schema Collection Memory Benchmark
+
+Demonstrates the memory impact of the `max_tables` and `max_columns` limits added to
+`HanaSchemaCollector`. It stands up a SAP HANA Express container, fills it with a 1000×1000
+schema (1000 tables, 1000 columns each), then runs the real `collect_schemas()` code path
+twice — once with limits enabled and once effectively unlimited — in isolated subprocesses
+and compares peak RSS.
+
+## Prerequisites
+
+- Docker with sufficient memory (≥8 GB recommended for HANA Express).
+- A Python environment with the following installed:
+  ```
+  pip install -e ../../..                  # datadog_checks_base (repo root)
+  pip install -e ../..                     # sap_hana integration
+  pip install hdbcli==2.21.28
+  ```
+  The integration's hatch test environment already satisfies this.
+
+## Run
+
+```bash
+cd sap_hana/benchmarks/schema_collection_memory
+
+# 1. Start SAP HANA Express (startup takes 5–10 minutes).
+PASSWORD=Admin1337 docker compose up -d
+docker logs -f saphanabenchmark   # wait for "Startup finished!"
+
+# 2. Populate the database and run both modes.
+python benchmark.py
+
+# Re-run measurements without recreating the schema.
+python benchmark.py --skip-setup
+
+# 3. Tear down.
+docker compose down --volumes
+```
+
+Results are written to `results/benchmark_results.txt`.
+
+## Tuning
+
+| Constant | File | Default | Notes |
+|---|---|---|---|
+| `SAP_HANA_VERSION` | `docker-compose.yaml` env var | `2.00.076.00.20231004.2` | Pin to a known-good image tag. |
+| `NUM_TABLES` | `setup_database.py` | 1000 | Lower if setup is too slow. |
+| `NUM_COLUMNS` | `setup_database.py` | 1000 | Lower if HANA Express rejects wide tables. |
+
+HANA Express has resource constraints. If `CREATE TABLE` fails with a column-count or
+memory error, set `NUM_COLUMNS` to 500 or lower.
+
+## Notes on memory investigation
+
+### `setfetchsize` (no effect on memory)
+`cursor.setfetchsize(10_000)` was tried on the hdbcli cursor before `execute()` to reduce
+the C-layer result buffer. It had no measurable effect on peak RSS (1042 MiB → 1043 MiB).
+The reason: the dominant memory consumer is the Python-side `_queued_rows` list
+accumulating all table dicts before the single `json.dumps` flush — not the hdbcli C
+layer. The base `SchemaCollector` flushes when `len(_queued_rows) >= payload_chunk_size`
+(default 10,000) or on the last database. Since HANA always reports one database and a
+1000-table schema is below the 10,000 threshold, everything flushes at once. The
+`max_tables` / `max_columns` limits are the effective memory control.
+
+### Column-based flush threshold (1.5x RSS reduction with limits)
+The base class `payload_chunk_size` counts tables, which is a poor proxy for memory when
+tables are wide. `HanaSchemaCollector` overrides `maybe_flush` to flush after every
+`PAYLOAD_COLUMN_CHUNK_SIZE` (50,000) columns instead. For 1000-column tables, this flushes
+every 50 tables, keeping `_queued_rows` from growing unboundedly. Result on the 1000×1000
+schema (RSS before = without column-flush override; RSS after = with override):
+
+| Mode | RSS before | RSS after | Payloads |
+|------|-----------|-----------|---------|
+| unlimited (no limits) | 1,038 MiB | 93.7 MiB | 21 |
+| limited (300 tables × 50 cols) | 56.4 MiB | 61.5 MiB | 1 |
+
+The limited case is unaffected: 300 × 50 = 15,000 columns never reaches the 50,000
+threshold so it still flushes once at the end.
+
+### SQL column limit via `ROW_NUMBER()`
+The original query joined `SYS.TABLE_COLUMNS` without a column count cap, so the server
+returned all 1000 columns per table regardless of `max_columns`; Python discarded the
+excess. Pushing the limit into SQL with a `limited_columns` CTE using `ROW_NUMBER() OVER
+(PARTITION BY schema, table ORDER BY position)` means the server only sends the first
+`max_columns` columns per table. Result on the 1000×1000 schema (duration before = without
+ROW_NUMBER cap; duration after = with cap):
+
+| Mode    | Duration before | Duration after |
+|---------|-----------------|----------------|
+| limited (max_tables=300, max_columns=50)         | 8.1s | 2.2s |
+| unlimited (max_tables=10M, max_columns=10M)      | 48.8s | 50.7s |
+
+Peak RSS was unchanged in both modes — the Python-side column dicts are bounded by
+`max_columns` either way, so peak Python heap stays the same. The gain is query
+efficiency: 285,000 fewer rows sent from the server (300 tables × 950 discarded
+columns) in the limited case.
+
+## Expected outcome
+
+The limited run (max\_tables=300, max\_columns=50) processes 300 tables × 50 columns =
+15,000 column dicts. The unlimited run processes 1000 tables × 1000 columns = 1,000,000
+column dicts, all held in memory at once before the single `json.dumps` flush. The
+unlimited peak RSS is expected to be ~1.5x larger than the limited run.
diff --git a/sap_hana/benchmarks/schema_collection_memory/benchmark.py b/sap_hana/benchmarks/schema_collection_memory/benchmark.py
@@ -0,0 +1,118 @@
+# (C) Datadog, Inc. 2026-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+#!/usr/bin/env python3
+"""Orchestrate limited vs unlimited schema collection runs and compare peak memory."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+
+RESULTS_DIR = os.path.join(os.path.dirname(__file__), 'results')
+RESULTS_FILE = os.path.join(RESULTS_DIR, 'benchmark_results.txt')
+RUN_COLLECTOR = os.path.join(os.path.dirname(__file__), 'run_collector.py')
+SETUP_DATABASE = os.path.join(os.path.dirname(__file__), 'setup_database.py')
+
+HOST = 'localhost'
+PORT = 39019
+
+
+def wait_for_hana(host: str, port: int, timeout: int = 600) -> None:
+    print(f'Waiting for SAP HANA at {host}:{port} (up to {timeout}s)...', flush=True)
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        try:
+            from hdbcli.dbapi import Connection as HanaConnection
+
+            conn = HanaConnection(address=host, port=port, user='system', password='Admin1337')
+            conn.close()
+            print('HANA is ready.', flush=True)
+            return
+        except Exception:
+            time.sleep(5)
+    print('ERROR: HANA did not become ready in time.', file=sys.stderr)
+    sys.exit(1)
+
+
+def run_mode(python: str, mode: str, host: str, port: int) -> dict:
+    print(f'Running mode={mode}...', flush=True)
+    result = subprocess.run(
+        [python, RUN_COLLECTOR, '--mode', mode, '--host', host, '--port', str(port)],
+        capture_output=True,
+        text=True,
+    )
+    if result.returncode != 0:
+        print(f'ERROR running mode={mode}:\n{result.stderr}', file=sys.stderr)
+        sys.exit(1)
+    return json.loads(result.stdout.strip())
+
+
+def mib(value_bytes: int) -> str:
+    return f'{value_bytes / 1024 / 1024:.1f} MiB'
+
+
+def build_report(limited: dict, unlimited: dict) -> str:
+    rss_ratio = unlimited['peak_rss_kb'] / max(limited['peak_rss_kb'], 1)
+    lines = [
+        'SAP HANA Schema Collection Memory Benchmark',
+        '=' * 60,
+        '',
+        f"{'Mode':<12} {'Peak RSS':>12} {'Tracemalloc':>14} {'Payloads':>10} {'Payload bytes':>15} {'Duration':>10}",
+        '-' * 75,
+    ]
+    for r in (limited, unlimited):
+        lines.append(
+            f"{r['mode']:<12}"
+            f" {mib(r['peak_rss_kb'] * 1024):>12}"
+            f" {mib(r['tracemalloc_peak_bytes']):>14}"
+            f" {r['payloads']:>10}"
+            f" {r['total_payload_bytes']:>15,}"
+            f" {r['duration_s']:>9.1f}s"
+        )
+    lines += [
+        '',
+        f"Memory reduction: {rss_ratio:.1f}x less peak RSS with limits enabled",
+        "  limited  : max_tables=300, max_columns=50",
+        "  unlimited: max_tables=10_000_000, max_columns=10_000_000",
+        '',
+    ]
+    return '\n'.join(lines)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description='Run the schema collection memory benchmark.')
+    parser.add_argument('--skip-setup', action='store_true', help='Skip database setup (schema already populated).')
+    parser.add_argument('--python', default=sys.executable, help='Python interpreter for child runs.')
+    parser.add_argument('--host', default=HOST)
+    parser.add_argument('--port', type=int, default=PORT)
+    args = parser.parse_args()
+
+    wait_for_hana(args.host, args.port)
+
+    if not args.skip_setup:
+        print('Running database setup...', flush=True)
+        result = subprocess.run(
+            [args.python, SETUP_DATABASE, '--host', args.host, '--port', str(args.port)],
+            check=True,
+        )
+        _ = result
+
+    limited = run_mode(args.python, 'limited', args.host, args.port)
+    unlimited = run_mode(args.python, 'unlimited', args.host, args.port)
+
+    report = build_report(limited, unlimited)
+    print(report)
+
+    os.makedirs(RESULTS_DIR, exist_ok=True)
+    with open(RESULTS_FILE, 'w') as f:
+        f.write(report)
+    print(f'Results written to {RESULTS_FILE}')
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/sap_hana/benchmarks/schema_collection_memory/docker-compose.yaml b/sap_hana/benchmarks/schema_collection_memory/docker-compose.yaml
@@ -0,0 +1,19 @@
+services:
+  saphanabenchmark:
+    container_name: saphanabenchmark
+    image: saplabs/hanaexpress:${SAP_HANA_VERSION:-latest}
+    ulimits:
+      nofile:
+        soft: 1048576
+        hard: 1048576
+    sysctls:
+      kernel.shmmax: 1073741824
+      net.ipv4.ip_local_port_range: "40000 60999"
+    ports:
+      - "39019:39017"
+    environment:
+      - PASSWORD=${PASSWORD}
+    entrypoint:
+      - sh
+      - -c
+      - echo "{\"master_password\":\"$$PASSWORD\"}" > /tmp/hana_password.json;cat /tmp/hana_password.json;/run_hana --agree-to-sap-license --passwords-url file:///tmp/hana_password.json
diff --git a/sap_hana/benchmarks/schema_collection_memory/results/.gitkeep b/sap_hana/benchmarks/schema_collection_memory/results/.gitkeep
diff --git a/sap_hana/benchmarks/schema_collection_memory/results/benchmark_results.txt b/sap_hana/benchmarks/schema_collection_memory/results/benchmark_results.txt
@@ -0,0 +1,11 @@
+SAP HANA Schema Collection Memory Benchmark
+============================================================
+
+Mode             Peak RSS    Tracemalloc   Payloads   Payload bytes   Duration
+---------------------------------------------------------------------------
+limited          61.5 MiB        8.6 MiB          1       1,772,208       2.2s
+unlimited        93.7 MiB       24.2 MiB         21      95,425,468      50.7s
+
+Memory reduction: 1.5x less peak RSS with limits enabled
+  limited  : max_tables=300, max_columns=50
+  unlimited: max_tables=10_000_000, max_columns=10_000_000
diff --git a/sap_hana/benchmarks/schema_collection_memory/results/benchmark_results_50x50.txt b/sap_hana/benchmarks/schema_collection_memory/results/benchmark_results_50x50.txt
@@ -0,0 +1,11 @@
+SAP HANA Schema Collection Memory Benchmark
+============================================================
+
+Mode             Peak RSS    Tracemalloc   Payloads   Payload bytes   Duration
+---------------------------------------------------------------------------
+limited          45.0 MiB        1.2 MiB          1         243,885       0.3s
+unlimited        45.0 MiB        1.2 MiB          1         243,885       0.3s
+
+Memory reduction: 1.0x less peak RSS with limits enabled
+  limited  : max_tables=300, max_columns=50
+  unlimited: max_tables=10_000_000, max_columns=10_000_000