Get chargeback data results from loki and validate

that the total cost is correct

Used Gemini and Cursor
Validates chargeback total cost
- uses synth data to calculate total cost via script
- run "openstack rating summary get" to get total cost from loki
- compares script_totals and Loki_Totals if same then job passes

Closes: https://issues.redhat.com/browse/OSPRH-26066

Author: ayefimov-1

roles/telemetry_chargeback/README.md

@@ -5,7 +5,7 @@ The **`telemetry_chargeback`** role is designed to test the **RHOSO Cloudkitty**
 The role performs two main functions:
 
 1. **CloudKitty Validation** - Enables and configures the CloudKitty hashmap rating module, then validates its state.
-2. **Synthetic Data Generation** - Generates synthetic Loki log data for testing chargeback scenarios using a Python script and Jinja2 template.
+2. **Synthetic Data Generation & Analysis** - Generates synthetic Loki log data for testing chargeback scenarios and calculates metric totals. The role automatically discovers and processes all scenario files matching `test_*.yml` in the `files/` directory. For each scenario it then runs the load path (ingest to Loki, retrieve from Loki, get cost via CloudKitty rating summary). The ingest and retrieve steps are currently stubs for future implementation.
 
 Requirements
 ------------
@@ -15,7 +15,7 @@ It relies on the following being available on the target or control host:
 * The **OpenStack CLI client** must be installed and configured with administrative credentials.
 * Required Python libraries for the `openstack` CLI (e.g., `python3-openstackclient`).
 * Connectivity to the OpenStack API endpoint.
-* **Python 3** with the following libraries for synthetic data generation:
+* **Python 3** with the following libraries for synthetic data generation and analysis:
   * `PyYAML`
   * `Jinja2`
 
@@ -42,22 +42,41 @@ These variables are used internally by the role and typically do not need to be
 |----------|---------------|-------------|
 | `logs_dir_zuul` | `/home/zuul/ci-framework-data/logs` | Remote directory for log files. |
 | `artifacts_dir_zuul` | `/home/zuul/ci-framework-data/artifacts` | Directory for generated artifacts. |
+| `ck_scenario_dir` | `{{ role_path }}/files` | Directory containing scenario files (`test_*.yml`). |
+| `ck_synth_data_suffix` | `.json` | Suffix for generated synthetic data files. |
+| `ck_synth_totals_suffix` | `_syn-totals.yml` | Suffix for generated metric totals files (from synthetic data). |
+| `ck_loki_totals_suffix` | `_loki-totals.yml` | Suffix for totals retrieved from Loki (reserved for future use). |
 | `ck_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. |
-| `ck_data_template` | `{{ role_path }}/template/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
-| `ck_data_config` | `{{ role_path }}/files/test_static.yml` | Path to the scenario configuration file. |
-| `ck_output_file_local` | `{{ artifacts_dir_zuul }}/loki_synth_data.json` | Local path for generated synthetic data. |
-| `ck_output_file_remote` | `{{ logs_dir_zuul }}/gen_loki_synth_data.log` | Remote destination for synthetic data. |
+| `ck_data_template` | `{{ role_path }}/templates/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
+| `ck_totals_script` | `{{ role_path }}/files/synth_loki_metrics_totals.py` | Path to the metric totals calculation script. |
+
+### Dynamically Set Variables (gen_synth_loki_data.yml)
+
+These variables are set dynamically for each scenario file during the loop:
+
+| Variable | Description |
+|----------|-------------|
+| `ck_data_file` | Local path for generated JSON data (`{{ artifacts_dir_zuul }}/{{ scenario_name }}.json`) |
+| `ck_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}_syn-totals.yml`) |
+| `ck_test_file` | Path to the scenario configuration file (`{{ ck_scenario_dir }}/{{ scenario_name }}.yml`) |
 
 Scenario Configuration
 ----------------------
-The synthetic data generation is controlled by a YAML configuration file (`files/test_static.yml`). This file defines:
+The synthetic data generation is controlled by YAML configuration files in the `files/` directory. Any file matching `test_*.yml` will be automatically discovered and processed.
+
+Each scenario file defines:
 
 * **generation** - Time range configuration (days, step_seconds)
 * **log_types** - List of log type definitions with name, type, unit, qty, price, groupby, and metadata
 * **required_fields** - Fields required for validation
 * **date_fields** - Date fields to add to groupby (week_of_the_year, day_of_the_year, month, year)
 * **loki_stream** - Loki stream configuration (service name)
 
+Example scenario files:
+* `test_static.yml` - Basic static values for qty and price
+* `test_dyn_basic.yml` - Dynamic values distributed across time steps
+* `test_all_qty_zero.yml` - All quantities set to zero for testing
+
 Dependencies
 ------------
 This role has no direct hard dependencies on other Ansible roles.

roles/telemetry_chargeback/files/gen_synth_loki_data.py

@@ -5,10 +5,44 @@
 import yaml
 from datetime import datetime, timezone, timedelta
 from pathlib import Path
-from typing import Dict, Any
+from typing import Dict, Any, List, Union
 from jinja2 import Environment
 
 
+def _get_value_for_step(
+    values: List[Union[int, float]],
+    step_idx: int,
+    num_steps: int
+) -> Union[int, float]:
+    """
+    Get the appropriate value from a list based on the current step index.
+
+    Values are distributed evenly across all steps. For example, if there are
+    12 steps and 4 values, each value covers 3 steps:
+    - Steps 0-2: values[0]
+    - Steps 3-5: values[1]
+    - Steps 6-8: values[2]
+    - Steps 9-11: values[3]
+
+    Args:
+        values: List of values to choose from.
+        step_idx: Current step index (0-based).
+        num_steps: Total number of steps.
+
+    Returns:
+        The value corresponding to the current step.
+    """
+    num_values = len(values)
+    if num_values == 1:
+        return values[0]
+
+    # Calculate how many steps each value covers
+    steps_per_value = num_steps / num_values
+    # Determine which value index to use, clamping to valid range
+    value_idx = min(int(step_idx // steps_per_value), num_values - 1)
+    return values[value_idx]
+
+
 # --- Configure logging with a default level that can be changed ---
 logging.basicConfig(
     level=logging.INFO,
@@ -200,12 +234,18 @@ def generate_loki_data(
                 f"groupby must be a dictionary for {log_type_name}"
             )
 
+        # Ensure qty and price are lists for step-based distribution
+        qty_val = log_type_config["qty"]
+        price_val = log_type_config["price"]
+        qty_list = qty_val if isinstance(qty_val, list) else [qty_val]
+        price_list = price_val if isinstance(price_val, list) else [price_val]
+
         log_types[log_type_name] = {
             "type": log_type_config["type"],
             "unit": log_type_config["unit"],
             "description": log_type_config.get("description"),
-            "qty": log_type_config["qty"],
-            "price": log_type_config["price"],
+            "qty": qty_list,
+            "price": price_list,
             "groupby": groupby.copy(),
             "metadata": log_type_config.get("metadata", {})
         }
@@ -231,15 +271,15 @@ def tojson_preserve_order(obj):
     # --- Render the template in one pass with all the data ---
     logger.info("Rendering final output...")
 
+    # Calculate total number of steps for value distribution
+    num_steps = len(log_data_list)
+    logger.debug(f"Total number of time steps: {num_steps}")
+
     # Pre-calculate log types with date fields for each time step
     log_types_list = []
     for idx, item in enumerate(log_data_list):
-        # For the last entry, use end_time to ensure it shows today's date
-        if idx == len(log_data_list) - 1:
-            dt = end_time
-        else:
-            epoch_seconds = item["nanoseconds"] / 1_000_000_000
-            dt = datetime.fromtimestamp(epoch_seconds, tz=timezone.utc)
+        epoch_seconds = item["nanoseconds"] / 1_000_000_000
+        dt = datetime.fromtimestamp(epoch_seconds, tz=timezone.utc)
 
         iso_year, iso_week, _ = dt.isocalendar()
         day_of_year = dt.timetuple().tm_yday
@@ -267,6 +307,13 @@ def tojson_preserve_order(obj):
             log_type_with_dates = log_type_data.copy()
             log_type_with_dates["groupby"] = log_type_data["groupby"].copy()
             log_type_with_dates["groupby"].update(date_fields)
+            # Select qty and price based on step index distribution
+            log_type_with_dates["qty"] = _get_value_for_step(
+                log_type_data["qty"], idx, num_steps
+            )
+            log_type_with_dates["price"] = _get_value_for_step(
+                log_type_data["price"], idx, num_steps
+            )
             log_types_with_dates[log_type_name] = log_type_with_dates
 
         log_types_list.append(log_types_with_dates)

roles/telemetry_chargeback/files/synth_loki_metrics_totals.py

@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+"""
+Calculate metric totals and aggregate total from a Loki JSON file.
+
+Output is in YAML format.
+"""
+import json
+import argparse
+import sys
+import yaml
+from pathlib import Path
+
+
+def calculate_totals(json_path: Path, output_path: Path):
+    """
+    Read Loki JSON, calculate step totals (qty * price), and sum them up.
+
+    Args:
+        json_path: Path to the input JSON file.
+        output_path: Path to the output YAML file.
+    """
+    try:
+        with json_path.open('r') as f:
+            data = json.load(f)
+    except Exception as e:
+        print(f"Error reading JSON file {json_path}: {e}")
+        sys.exit(1)
+
+    metric_totals = {}
+    aggregate_total = 0.0
+    time_steps_set = set()
+
+    # Extract values from the Loki JSON structure
+    for stream in data.get('streams', []):
+        for val_pair in stream.get('values', []):
+            try:
+                # The first element is the timestamp (nanoseconds)
+                timestamp = val_pair[0]
+                time_steps_set.add(timestamp)
+
+                # The second element is a JSON string containing the log entry
+                entry = json.loads(val_pair[1])
+
+                m_type = entry.get('type')
+                if m_type is None:
+                    m_type = 'unknown'
+
+                qty = float(entry.get('qty', 0))
+                price = float(entry.get('price', 0))
+
+                step_total = qty * price
+
+                if m_type not in metric_totals:
+                    metric_totals[m_type] = 0.0
+
+                metric_totals[m_type] += step_total
+                aggregate_total += step_total
+            except (json.JSONDecodeError, ValueError, IndexError) as e:
+                print(f"Warning: Skipping malformed entry: {e}")
+                continue
+
+    # Prepare data for YAML output following vars/main.yml pattern
+    output_data = {
+        "total_time_steps": len(time_steps_set),
+        "syth_rate": {
+            m: round(t, 4) for m, t in sorted(metric_totals.items())
+        },
+        "total_rate": round(aggregate_total, 4)
+    }
+
+    # Write to output file in YAML format
+    try:
+        with output_path.open('w') as f_out:
+            f_out.write("---\n")
+            yaml.dump(
+                output_data, f_out, default_flow_style=False, sort_keys=False
+            )
+        print(
+            f"Successfully calculated totals and wrote YAML to {output_path}"
+        )
+    except Exception as e:
+        print(f"Error writing to output file {output_path}: {e}")
+        sys.exit(1)
+
+
+def main():
+    """Main entry point for the script."""
+    parser = argparse.ArgumentParser(
+        description="Calculate totals from Loki JSON data"
+    )
+    parser.add_argument(
+        "-j", "--json", required=True, type=Path,
+        help="Path to the input JSON file."
+    )
+    parser.add_argument(
+        "-o", "--output", required=True, type=Path,
+        help="Path to the output YAML file."
+    )
+
+    args = parser.parse_args()
+    calculate_totals(args.json, args.output)
+
+
+if __name__ == "__main__":
+    main()