Evaluation ICSA 2021

This page contains the documentation of the evaluation that we performed for our ICSA 2021 Paper named "Enabling Consistency between Software Artefacts for Software Adaption and Evolution".

Overview and Goals

Our evaluation mainly conducted three characteristics of our approach:

1. Accuracy
   1. Model Accuracy (Comparing the elements within the derived models to reference models)
   2. Prediction Accuracy (Comparing the simulations results of the derived models to monitoring data)
2. Monitoring Overhead
   1. On a per-service level
   2. Cumulated over time (to show that our adaptive monitoring helps to reduce the overhead)
3. Scalability of the transformations within the pipeline

The results and the instructions for reproducing them were divided into several sub-documentations. These are listed in the following. In addition, we annotated the case study/studies which was/were used.

Versions of the casestudies used:

• CoCoME: Cloud variant - 4 Sep 2018
• TeaStore: Customized & instrumented version - based on version 1.3.1

Accuracy

1. At Development-Time
   1. Model Accuracy of the System Model Extraction at Development-Time [TeaStore, CoCoME]
2. At Operation-Time
   1. Model Accuracy when executing Adaption Scenarios at Operation-Time [TeaStore]
   2. Prediction Accuracy when executing Adaption Scenarios at Operation-Time [TeaStore]

Overhead

1. Monitoring Overhead for a single Service Execution [TeaStore]
2. Cumulated Monitoring Overhead over Time at Operation-Time [TeaStore]

Scalability

1. Scalability Characteristics of the Transformations within the Pipeline [Synthetical]