Adding the rating calculations for uploaded and downloaded

data. This data will then be validated

Author: ayefimov-1

roles/telemetry_chargeback/README.md

@@ -46,9 +46,9 @@ These variables are used internally by the role and typically do not need to be
 | `cloudkitty_scenario_dir` | `{{ role_path }}/files` | Directory containing scenario files (`test_*.yml`). |
 | `cloudkitty_synth_data_suffix` | `-synth_data.json` | Suffix for generated synthetic data files. |
 | `cloudkitty_loki_data_suffix` | `-loki_data.json` | Suffix for Loki query result JSON files. |
-| `cloudkitty_synth_totals_suffix` | `-synth_metrics_totals.yml` | Suffix for generated metric totals files (from synthetic data). |
-| `cloudkitty_loki_totals_suffix` | `-loki_totals.yml` | Suffix for CloudKitty rating summary output files (from loki_rate task). |
-| `cloudkitty_loki_totals_metrics_suffix` | `-loki_metrics_totals.yml` | Suffix for metric totals computed from Loki-retrieved JSON (retrieve_loki_data task). |
+| `cloudkitty_synth_totals_metrics_suffix` | `-synth_metrics_summary.yml` | Suffix for YAML totals from synthetic JSON (`gen_db_summary.py`). |
+| `cloudkitty_loki_totals_suffix` | `-rating.yml` | Suffix for CloudKitty `openstack rating summary` text saved by `loki_rate.yml`. |
+| `cloudkitty_loki_totals_metrics_suffix` | `-loki_metrics_summary.yml` | Suffix for YAML from Loki-retrieved JSON (`gen_db_summary.py` in `retrieve_loki_data.yml`). |
 | `cloudkitty_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. |
 | `cloudkitty_data_template` | `{{ role_path }}/templates/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
 | `cloudkitty_summary_script` | `{{ role_path }}/files/gen_db_summary.py` | Path to the summary script (gen_db_summary.py). |
@@ -87,13 +87,15 @@ Loki push/query URLs are set dynamically in `setup_loki_env.yml` from the Cloudk
 | `-r`, `--reverse` | Reverse timestamp order in JSON output (youngest first, oldest last). |
 | `--debug` | Enable debug logging. |
 
-**gen_synth_loki_metrics_totals.py** — Reads the synthetic (or Loki-retrieved) JSON and writes a YAML with aggregated metrics and time bounds. The output is used by the role for validation and for the Loki query time range.
+**gen_db_summary.py** (`cloudkitty_summary_script`) — This is what the role runs for **both** synthetic totals and Loki-retrieved totals (`gen_synth_loki_data.yml` and `retrieve_loki_data.yml`). It parses Loki-style JSON (streams or `data.result`), sorts entries by timestamp, and writes a single YAML summary used for assertions and for the Loki query window (`time.begin_step.nanosec`).
 
 Output YAML structure:
 
-* **time** — `begin`, `end` (ISO strings), `begin_nano`, `end_nano` (nanosecond timestamps for the first and last time step; used by the Loki query in `retrieve_loki_data.yml`).
-* **data_log** — `total_time_steps`, `metrics_per_step`, `log_count`.
-* **synth_rate** — Per-metric rates and `total_rate`.
+* **time** — `begin_step` / `end_step`, each with `nanosec`, `begin`, `end` (ISO window strings from the log payload).
+* **data_log** — `total_timesteps`, `metrics_per_step`, `log_count`.
+* **rate** — `by_types` (per-type `Rate`) and `total.Rating`.
+
+**gen_synth_loki_metrics_totals.py** — Standalone helper that also reads Loki JSON and emits YAML totals, but with a **different schema** (flat `time` with `begin_nano` / `end_nano`, `data_log.total_time_steps`, and a flat `synth_rate` map including `total_rate`). The role does **not** invoke it; it remains in the repo for manual runs, one-off comparisons, or older workflows. Prefer `gen_db_summary.py` for anything that must match the role’s artifacts and `retrieve_loki_data.yml` expectations.
 
 ### Dynamically Set Variables
 
@@ -109,7 +111,7 @@ Set in **gen_synth_loki_data.yml** for each scenario file during the loop:
 | Variable | Description |
 |----------|-------------|
 | `cloudkitty_data_file` | Local path for generated JSON data (`{{ artifacts_dir_zuul }}/{{ scenario_name }}-synth_data.json`) |
-| `cloudkitty_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}{{ cloudkitty_synth_totals_suffix }}`) |
+| `cloudkitty_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}{{ cloudkitty_synth_totals_metrics_suffix }}`) |
 | `cloudkitty_test_file` | Path to the scenario configuration file (`{{ cloudkitty_scenario_dir }}/{{ scenario_name }}.yml`) |
 
 Scenario Configuration