New simple metrics #4038
Description
QA Metrics
Using KIWI as a tool to measure indicators such as execution coverage, success/failure rates, blockages, and approvals will provide quantitative information to support decisions about resource allocation, release schedules, and critical areas of the QA process. The approach I propose is modular, starting with a basic set of metrics, but meaningful enough to reflect the health of the testing process.
The following testing metrics are proposed:
Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts planned to be executed) x 100
Passed Test Cases Coverage = (Number of passed tests / Total number of tests executed) x 100
Failed Test Case Coverage = (Number of failed tests / Total number of test cases executed) x 100
Unexecuted cases = (Unexecuted tests / Total cases of test run) × 100
Test Cases Blocked = (Number of blocked tests / Total number of tests executed) x 100
Number of Test Run Per Time Period= (Number of test run / Total time)
Quality Ratio = (Successful Tests Cases / Total Number of Tests Cases) x 100
Schedule Variance = [(Actual efforts – estimated efforts) / Estimated Efforts)] x 100
These metrics are aligned with good testing practices. Measuring test coverage and results contributes to improving software reliability and makes it easier to identify gaps.
Proposal
Incorporation of a new menu option Telemetry > Testing Metrics:
Page with basic filters (similar to the telemetry/testing/execution/dashboard/ page) => (Product, Version, Build, Test Plan, After, Before, Test Run, Child TPs)
Then, the results table will display the following columns:
Test Run (summary)
Test Execution Coverage %
Passed Test Cases Coverage %
Failed Test Case Coverage %
Unexecuted Cases %
Test Cases Blocked %
Number of Test Runs Per Time Period
Quality Ratio %
Schedule Variance %
Options should include export to CSV, Excel, PDF, print, and hide columns.
Note:
Taking my db, I have crafted this SQL query, which might help.
SELECT
-- Información del test run primero
tr.id AS test_run_id,
tr.summary AS test_run_name,
tr.start_date,
tr.stop_date,
tp.name AS test_plan_name,
-- Métricas a partir de aquí
-- Casos planificados
(SELECT COUNT(*) FROM public.testcases_testcaseplan WHERE plan_id = tp.id) AS total_casos_planificados,
-- Casos ejecutados (únicos)
(SELECT COUNT(DISTINCT te.case_id)
FROM public.testruns_testexecution te
WHERE te.run_id = tr.id) AS pruebas_ejecutadas_total,
-- Aprobados
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 4)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_aprobados,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 4) AS total_aprobados,
-- Fallidos
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 5)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_fallidos,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 5) AS total_fallidos,
-- Bloqueados
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 6)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_bloqueados,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 6) AS total_bloqueados,
-- Ignorados
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 8)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_ignorados,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 8) AS total_ignorados,
-- En espera
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 1)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_en_espera,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 1) AS total_en_espera,
-- En ejecución
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 2)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_en_ejecucion,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 2) AS total_en_ejecucion,
-- En pausa
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 3)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testruns_testexecution te WHERE te.run_id = tr.id), 0)
* 100, 2
) AS cobertura_en_pausa,
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 3) AS total_en_pausa,
-- Cobertura de ejecución (por cantidad distinta de casos ejecutados)
ROUND(
(SELECT COUNT(DISTINCT te.case_id)
FROM public.testruns_testexecution te WHERE te.run_id = tr.id)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testcases_testcaseplan WHERE plan_id = tp.id), 0)
* 100, 2
) AS cobertura_ejecucion,
-- Índice de calidad = aprobados / planificados
ROUND(
(SELECT COUNT(*) FROM public.testruns_testexecution te
WHERE te.run_id = tr.id AND te.status_id = 4)::decimal
/
NULLIF((SELECT COUNT(*) FROM public.testcases_testcaseplan WHERE plan_id = tp.id), 0)
* 100, 2
) AS indice_calidad
FROM public.testruns_testrun tr
JOIN public.testplans_testplan tp ON tr.plan_id = tp.id
ORDER BY tr.start_date DESC;