Get chargeback data results from loki and validate

that the total cost is correct

Used Gemini and Cursor
Validates chargeback total cost
- uses synth data to calculate total cost via script
- run "openstack rating summary get" to get total cost from loki
- compares script_totals and Loki_Totals if same then job passes

Closes: https://issues.redhat.com/browse/OSPRH-26066

Author: ayefimov-1

roles/telemetry_chargeback/.yamllint

@@ -0,0 +1,18 @@
+---
+# Ansible-lint compatible yamllint config for this role only.
+# See: https://ansible.readthedocs.io/projects/lint/rules/yaml/
+extends: default
+
+rules:
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: false
+  braces:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  octal-values:
+    forbid-implicit-octal: true
+    forbid-explicit-octal: true
+  line-length:
+    max: 160
+    level: warning

roles/telemetry_chargeback/README.md

@@ -5,7 +5,7 @@ The **`telemetry_chargeback`** role is designed to test the **RHOSO Cloudkitty**
 The role performs two main functions:
 
 1. **CloudKitty Validation** - Enables and configures the CloudKitty hashmap rating module, then validates its state.
-2. **Synthetic Data Generation** - Generates synthetic Loki log data for testing chargeback scenarios using a Python script and Jinja2 template.
+2. **Synthetic Data Generation & Analysis** - Generates synthetic Loki log data for testing chargeback scenarios and calculates metric totals. The role automatically discovers and processes all scenario files matching `test_*.yml` in the `files/` directory. For each scenario it runs: generate synthetic data, compute syn-totals, ingest to Loki, flush Loki ingester memory, and get cost via CloudKitty rating summary (using begin/end from syn-totals). Retrieve-from-Loki is available but currently commented out in the task flow.
 
 Requirements
 ------------
@@ -15,14 +15,15 @@ It relies on the following being available on the target or control host:
 * The **OpenStack CLI client** must be installed and configured with administrative credentials.
 * Required Python libraries for the `openstack` CLI (e.g., `python3-openstackclient`).
 * Connectivity to the OpenStack API endpoint.
-* **Python 3** with the following libraries for synthetic data generation:
+* **Python 3** with the following libraries for synthetic data generation and analysis:
   * `PyYAML`
   * `Jinja2`
 
 It is expected to be run **after** a successful deployment and configuration of the following components:
 
 * **OpenStack:** A functional OpenStack cloud (RHOSO) environment.
 * **Cloudkitty:** The Cloudkitty service must be installed, configured, and running.
+* **Loki / OpenShift (for ingest and flush):** When using ingest and flush tasks, the control host must have `oc` CLI access, and the Cloudkitty Loki stack (route, certificates, ingester) must be deployed. The role sets Loki push/query URLs and extracts certificates via `setup_loki_env.yml`.
 
 Role Variables
 --------------
@@ -42,22 +43,64 @@ These variables are used internally by the role and typically do not need to be
 |----------|---------------|-------------|
 | `logs_dir_zuul` | `/home/zuul/ci-framework-data/logs` | Remote directory for log files. |
 | `artifacts_dir_zuul` | `/home/zuul/ci-framework-data/artifacts` | Directory for generated artifacts. |
+| `ck_scenario_dir` | `{{ role_path }}/files` | Directory containing scenario files (`test_*.yml`). |
+| `ck_synth_data_suffix` | `.json` | Suffix for generated synthetic data files. |
+| `ck_loki_data_suffix` | `_loki.json` | Suffix for Loki query result JSON files. |
+| `ck_synth_totals_suffix` | `_syn-totals.yml` | Suffix for generated metric totals files (from synthetic data). |
+| `ck_loki_totals_suffix` | `_loki-totals.yml` | Suffix for CloudKitty rating summary output files (from loki_rate task). |
+| `ck_begin_end_suffix` | `_begin_end.yml` | Suffix for begin/end timestamp output files. |
 | `ck_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. |
-| `ck_data_template` | `{{ role_path }}/template/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
-| `ck_data_config` | `{{ role_path }}/files/test_static.yml` | Path to the scenario configuration file. |
-| `ck_output_file_local` | `{{ artifacts_dir_zuul }}/loki_synth_data.json` | Local path for generated synthetic data. |
-| `ck_output_file_remote` | `{{ logs_dir_zuul }}/gen_loki_synth_data.log` | Remote destination for synthetic data. |
+| `ck_data_template` | `{{ role_path }}/templates/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
+| `ck_totals_script` | `{{ role_path }}/files/gen_synth_loki_metrics.totals.py` | Path to the metric totals calculation script. |
+
+### Loki / OpenShift Variables (vars/main.yml)
+
+Used by setup, ingest, flush, and retrieve tasks when running against Loki on OpenShift:
+
+| Variable | Default Value | Description |
+|----------|---------------|-------------|
+| `cert_secret_name` | `cert-cloudkitty-client-internal` | OpenShift secret name for client certificates. |
+| `cert_dir` | `{{ ansible_user_dir }}/ck-certs` | Local directory for extracted ingest/query certs. |
+| `client_secret` | `secret/cloudkitty-lokistack-gateway-client-http` | Secret for flush client certs. |
+| `ca_configmap` | `cm/cloudkitty-lokistack-ca-bundle` | ConfigMap for CA bundle. |
+| `remote_cert_dir` | `osp-certs` | Directory inside the OpenStack pod for certs. |
+| `local_cert_dir` | `{{ ansible_env.HOME }}/flush_certs` | Local directory for flush certs. |
+| `logql_query` | `{service="cloudkitty"}` (overridable via `loki_query`) | LogQL query for Loki. |
+| `ck_namespace` | `openstack` | OpenShift namespace for Cloudkitty/Loki resources. |
+| `openstackpod` | `openstackclient` | OpenStack client pod name for exec/cp. |
+| `lookback` | `6` | Days lookback for Loki query time range. |
+| `limit` | `50` | Limit for Loki query results. |
+
+Loki push/query URLs are set dynamically in `setup_loki_env.yml` from the Cloudkitty Loki route.
+
+### Dynamically Set Variables (gen_synth_loki_data.yml)
+
+These variables are set dynamically for each scenario file during the loop:
+
+| Variable | Description |
+|----------|-------------|
+| `ck_data_file` | Local path for generated JSON data (`{{ artifacts_dir_zuul }}/{{ scenario_name }}.json`) |
+| `ck_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}_syn-totals.yml`) |
+| `ck_begin_end_timestamp` | Local path for begin/end timestamp file (`{{ artifacts_dir_zuul }}/{{ scenario_name }}_begin_end.yml`) |
+| `ck_test_file` | Path to the scenario configuration file (`{{ ck_scenario_dir }}/{{ scenario_name }}.yml`) |
 
 Scenario Configuration
 ----------------------
-The synthetic data generation is controlled by a YAML configuration file (`files/test_static.yml`). This file defines:
+The synthetic data generation is controlled by YAML configuration files in the `files/` directory. Any file matching `test_*.yml` will be automatically discovered and processed.
+
+Each scenario file defines:
 
 * **generation** - Time range configuration (days, step_seconds)
 * **log_types** - List of log type definitions with name, type, unit, qty, price, groupby, and metadata
 * **required_fields** - Fields required for validation
 * **date_fields** - Date fields to add to groupby (week_of_the_year, day_of_the_year, month, year)
 * **loki_stream** - Loki stream configuration (service name)
 
+Example scenario files:
+* `test_static_basic.yml` - Basic static values for qty and price
+* `test_dyn_basic.yml` - Dynamic values distributed across time steps
+* `test_all_qty_zero.yml` - All quantities set to zero for testing
+
 Dependencies
 ------------
 This role has no direct hard dependencies on other Ansible roles.

roles/telemetry_chargeback/files/gen_synth_loki_data.py

@@ -5,10 +5,44 @@
 import yaml
 from datetime import datetime, timezone, timedelta
 from pathlib import Path
-from typing import Dict, Any
+from typing import Dict, Any, List, Union
 from jinja2 import Environment
 
 
+def _get_value_for_step(
+    values: List[Union[int, float]],
+    step_idx: int,
+    num_steps: int
+) -> Union[int, float]:
+    """
+    Get the appropriate value from a list based on the current step index.
+
+    Values are distributed evenly across all steps. For example, if there are
+    12 steps and 4 values, each value covers 3 steps:
+    - Steps 0-2: values[0]
+    - Steps 3-5: values[1]
+    - Steps 6-8: values[2]
+    - Steps 9-11: values[3]
+
+    Args:
+        values: List of values to choose from.
+        step_idx: Current step index (0-based).
+        num_steps: Total number of steps.
+
+    Returns:
+        The value corresponding to the current step.
+    """
+    num_values = len(values)
+    if num_values == 1:
+        return values[0]
+
+    # Calculate how many steps each value covers
+    steps_per_value = num_steps / num_values
+    # Determine which value index to use, clamping to valid range
+    value_idx = min(int(step_idx // steps_per_value), num_values - 1)
+    return values[value_idx]
+
+
 # --- Configure logging with a default level that can be changed ---
 logging.basicConfig(
     level=logging.INFO,
@@ -200,12 +234,18 @@ def generate_loki_data(
                 f"groupby must be a dictionary for {log_type_name}"
             )
 
+        # Ensure qty and price are lists for step-based distribution
+        qty_val = log_type_config["qty"]
+        price_val = log_type_config["price"]
+        qty_list = qty_val if isinstance(qty_val, list) else [qty_val]
+        price_list = price_val if isinstance(price_val, list) else [price_val]
+
         log_types[log_type_name] = {
             "type": log_type_config["type"],
             "unit": log_type_config["unit"],
             "description": log_type_config.get("description"),
-            "qty": log_type_config["qty"],
-            "price": log_type_config["price"],
+            "qty": qty_list,
+            "price": price_list,
             "groupby": groupby.copy(),
             "metadata": log_type_config.get("metadata", {})
         }
@@ -231,15 +271,15 @@ def tojson_preserve_order(obj):
     # --- Render the template in one pass with all the data ---
     logger.info("Rendering final output...")
 
+    # Calculate total number of steps for value distribution
+    num_steps = len(log_data_list)
+    logger.debug(f"Total number of time steps: {num_steps}")
+
     # Pre-calculate log types with date fields for each time step
     log_types_list = []
     for idx, item in enumerate(log_data_list):
-        # For the last entry, use end_time to ensure it shows today's date
-        if idx == len(log_data_list) - 1:
-            dt = end_time
-        else:
-            epoch_seconds = item["nanoseconds"] / 1_000_000_000
-            dt = datetime.fromtimestamp(epoch_seconds, tz=timezone.utc)
+        epoch_seconds = item["nanoseconds"] / 1_000_000_000
+        dt = datetime.fromtimestamp(epoch_seconds, tz=timezone.utc)
 
         iso_year, iso_week, _ = dt.isocalendar()
         day_of_year = dt.timetuple().tm_yday
@@ -267,6 +307,13 @@ def tojson_preserve_order(obj):
             log_type_with_dates = log_type_data.copy()
             log_type_with_dates["groupby"] = log_type_data["groupby"].copy()
             log_type_with_dates["groupby"].update(date_fields)
+            # Select qty and price based on step index distribution
+            log_type_with_dates["qty"] = _get_value_for_step(
+                log_type_data["qty"], idx, num_steps
+            )
+            log_type_with_dates["price"] = _get_value_for_step(
+                log_type_data["price"], idx, num_steps
+            )
             log_types_with_dates[log_type_name] = log_type_with_dates
 
         log_types_list.append(log_types_with_dates)

roles/telemetry_chargeback/files/gen_synth_loki_metrics.totals.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""
+Calculate metric totals and aggregate total from a Loki JSON file.
+
+Output is in YAML format.
+"""
+import json
+import argparse
+import sys
+import yaml
+from pathlib import Path
+
+
+def calculate_totals(json_path: Path, output_path: Path):
+    """
+    Read Loki JSON, calculate step totals (qty * price), and sum them up.
+
+    Args:
+        json_path: Path to the input JSON file.
+        output_path: Path to the output YAML file.
+    """
+    try:
+        with json_path.open('r') as f:
+            data = json.load(f)
+    except Exception as e:
+        print(f"Error reading JSON file {json_path}: {e}")
+        sys.exit(1)
+
+    metric_totals = {}
+    aggregate_total = 0.0
+    time_steps_set = set()
+    # Per-timestamp start/end from log entries (same for all entries at step)
+    time_step_bounds = {}
+
+    # Extract values from the Loki JSON structure
+    for stream in data.get('streams', []):
+        for val_pair in stream.get('values', []):
+            try:
+                # The first element is the timestamp (nanoseconds)
+                timestamp = val_pair[0]
+                time_steps_set.add(timestamp)
+
+                # The second element is a JSON string containing the log entry
+                entry = json.loads(val_pair[1])
+
+                # Start/end for this time step (same for all entries at step)
+                if timestamp not in time_step_bounds:
+                    time_step_bounds[timestamp] = {
+                        "begin": entry.get("start"),
+                        "end": entry.get("end"),
+                    }
+
+                m_type = entry.get('type')
+                if m_type is None:
+                    m_type = 'unknown'
+
+                qty = float(entry.get('qty', 0))
+                price = float(entry.get('price', 0))
+
+                step_total = qty * price
+
+                if m_type not in metric_totals:
+                    metric_totals[m_type] = 0.0
+
+                metric_totals[m_type] += step_total
+                aggregate_total += step_total
+            except (json.JSONDecodeError, ValueError, IndexError) as e:
+                print(f"Warning: Skipping malformed entry: {e}")
+                continue
+
+    # First and last time step timestamps (order by numeric value)
+    sorted_ts = (
+        sorted(time_steps_set, key=lambda t: int(t)) if time_steps_set else []
+    )
+    timestamp_begin = (
+        time_step_bounds[sorted_ts[0]]["begin"] if sorted_ts else None
+    )
+    timestamp_end = (
+        time_step_bounds[sorted_ts[-1]]["end"] if sorted_ts else None
+    )
+
+    # Prepare data for YAML output with time section and rates
+    syth_rate = {
+        m: round(t, 4) for m, t in sorted(metric_totals.items())
+    }
+    syth_rate["total_rate"] = round(aggregate_total, 4)
+
+    output_data = {
+        "time": {
+            "total_time_steps": len(time_steps_set),
+            "begin": timestamp_begin,
+            "end": timestamp_end,
+        },
+        "syth_rate": syth_rate,
+    }
+
+    # Write to output file in YAML format
+    try:
+        with output_path.open('w') as f_out:
+            f_out.write("---\n")
+            yaml.dump(
+                output_data, f_out, default_flow_style=False, sort_keys=False
+            )
+        print(
+            f"Successfully calculated totals and wrote YAML to {output_path}"
+        )
+    except Exception as e:
+        print(f"Error writing to output file {output_path}: {e}")
+        sys.exit(1)
+
+
+def main():
+    """Main entry point for the script."""
+    parser = argparse.ArgumentParser(
+        description="Calculate totals from Loki JSON data"
+    )
+    parser.add_argument(
+        "-j", "--json", required=True, type=Path,
+        help="Path to the input JSON file."
+    )
+    parser.add_argument(
+        "-o", "--output", required=True, type=Path,
+        help="Path to the output YAML file."
+    )
+
+    args = parser.parse_args()
+    calculate_totals(args.json, args.output)
+
+
+if __name__ == "__main__":
+    main()