Add synthetic date generation to telemetry_chargeback role in FVT repo (#331)

Generate synthetic data and write results to a json file

  Push json file to log directory so that during debug file is available for review

  Controller/ci-framework-data/tests/feature-verification-tests

  Do not overwrite syth data json if it already exists

  Using Gemini and Cursor

  Closes https://issues.redhat.com/browse/OSPRH-23746

  Update the synthetic data generation process to ensure that existing JSON files are not overwritten. This change allows for better debugging by preserving previous data outputs.

Closes https://issues.redhat.com/browse/OSPRH-23746

Co-authored-by: Muneesha Yadla <myadla@redhat.com>

Author: ayefimov-1

ci/run_chargeback_tests.yml

@@ -1,12 +1,13 @@
 ---
 - name: "Verify all the applicable projects, endpoints, pods & services for cloudkitty"
   hosts: "{{ cifmw_target_hook_host | default('localhost')  }}"
-  gather_facts: no
+  gather_facts: true
   ignore_errors: true
   environment:
     KUBECONFIG: "{{ cifmw_openshift_kubeconfig }}"
     PATH: "{{ cifmw_path }}"
   vars_files:
+    - vars/common.yml
     - vars/osp18_env.yml
   vars:
     common_pod_status_str: "Running"

roles/telemetry_chargeback/README.md

@@ -2,6 +2,11 @@ telemetry_chargeback
 =========
 The **`telemetry_chargeback`** role is designed to test the **RHOSO Cloudkitty** feature. These tests are specific to the Cloudkitty feature. Tests that are not specific to this feature (e.g., standard OpenStack deployment validation, basic networking) should be added to a common role.
 
+The role performs two main functions:
+
+1. **CloudKitty Validation** - Enables and configures the CloudKitty hashmap rating module, then validates its state.
+2. **Synthetic Data Generation** - Generates synthetic Loki log data for testing chargeback scenarios using a Python script and Jinja2 template.
+
 Requirements
 ------------
 It relies on the following being available on the target or control host:
@@ -10,6 +15,9 @@ It relies on the following being available on the target or control host:
 * The **OpenStack CLI client** must be installed and configured with administrative credentials.
 * Required Python libraries for the `openstack` CLI (e.g., `python3-openstackclient`).
 * Connectivity to the OpenStack API endpoint.
+* **Python 3** with the following libraries for synthetic data generation:
+  * `PyYAML`
+  * `Jinja2`
 
 It is expected to be run **after** a successful deployment and configuration of the following components:
 
@@ -18,12 +26,38 @@ It is expected to be run **after** a successful deployment and configuration of
 
 Role Variables
 --------------
-The role uses a few primary variables to control the testing environment and execution.
+The role uses the following variables to control the testing environment and execution.
+
+### User-Configurable Variables (defaults/main.yml)
 
 | Variable | Default Value | Description |
 |----------|---------------|-------------|
 | `openstack_cmd` | `openstack` | The command used to execute OpenStack CLI calls. This can be customized if the binary is not in the standard PATH. |
 
+### Internal Variables (vars/main.yml)
+
+These variables are used internally by the role and typically do not need to be modified.
+
+| Variable | Default Value | Description |
+|----------|---------------|-------------|
+| `logs_dir_zuul` | `/home/zuul/ci-framework-data/logs` | Remote directory for log files. |
+| `artifacts_dir_zuul` | `/home/zuul/ci-framework-data/artifacts` | Directory for generated artifacts. |
+| `ck_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. |
+| `ck_data_template` | `{{ role_path }}/template/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
+| `ck_data_config` | `{{ role_path }}/files/test_static.yml` | Path to the scenario configuration file. |
+| `ck_output_file_local` | `{{ artifacts_dir_zuul }}/loki_synth_data.json` | Local path for generated synthetic data. |
+| `ck_output_file_remote` | `{{ logs_dir_zuul }}/gen_loki_synth_data.log` | Remote destination for synthetic data. |
+
+Scenario Configuration
+----------------------
+The synthetic data generation is controlled by a YAML configuration file (`files/test_static.yml`). This file defines:
+
+* **generation** - Time range configuration (days, step_seconds)
+* **log_types** - List of log type definitions with name, type, unit, qty, price, groupby, and metadata
+* **required_fields** - Fields required for validation
+* **date_fields** - Date fields to add to groupby (week_of_the_year, day_of_the_year, month, year)
+* **loki_stream** - Loki stream configuration (service name)
+
 Dependencies
 ------------
 This role has no direct hard dependencies on other Ansible roles.
@@ -36,7 +70,7 @@ Example Playbook
   gather_facts: no
 
   tasks:
-    - name: "Run chargeback specific tests" 
+    - name: "Run chargeback specific tests"
       ansible.builtin.import_role:
         name: telemetry_chargeback
 ```