Replication Package: A Systematic Literature Review on Fault Injection Testing of Microservice Systems

This repository contains the replication package for the paper:

A Systematic Literature Review on Fault Injection Testing of Microservice Systems
Senyao Yu, Huayao Wu, Xintao Niu, and Changhai Nie

The replication package includes all raw data and intermediate results from the systematic literature search and selection process, enabling transparency and reproducibility of the study.

---

🔍 Process Overview

The replication package follows the systematic literature review process (see Related Documents/Search and Selection.png):

1. Initial Search: Comprehensive search across four databases
2. Impurity Removal: Filtering of non-research publications
3. Selection by Criteria: Application of inclusion/exclusion criteria with inter-rater reliability assessment
4. Snowballing: Two rounds of backward and forward snowballing to ensure completeness
5. Data Extraction: Systematic extraction of key information from selected papers to address all Research Questions (RQs)

---

📁 Contents

1. Initial Search.xlsx

Contains the raw search results from four digital libraries using the defined search keywords (see Related Documents/Literature Search Process.md for details).

Sheets:

• Initial Search-ACM: Results from ACM Digital Library
• Initial Search-IEEE: Results from IEEE Xplore Digital Library
• Initial Search-Springer: Results from SpringerLink
• Initial Search-ScienceDirect: Results from ScienceDirect
• Initial Search-Merge: Combined results from all databases

2. Impurity Removal.xlsx

Contains the results after removing non-research publications using database-specific filters (see Related Documents/Literature Search Process.md for filtering details).

Sheets:

• Impurity Removal-ACM: Filtered results from ACM
• Impurity Removal-IEEE: Filtered results from IEEE
• Impurity Removal-Springer: Filtered results from Springer
• Impurity Removal-ScienceDirect: Filtered results from ScienceDirect
• Impurity Removal-Merge: Combined filtered results

3. Selection by Criteria.xlsx

Documents the paper selection process based on inclusion and exclusion criteria (see Related Documents/Inclusion & Exclusion Criteria.md for criteria details).

Sheets:

• Selection Result:
  • Decisions from Author 1 and Author 2 for each paper
  • Decisions from Author 3 in case of disagreements
  • Final inclusion/exclusion decisions
• Cohen's Kappa: Calculation process for Cohen's Kappa inter-rater reliability score

4. Round1 Snowballing.xlsx and Round2 Snowballing.xlsx

Document the two rounds of snowballing (both backward and forward).

Sheets (for each file):

• Round1/2 Backward Snowballing: Papers identified from reference lists
• Round1/2 Forward Snowballing: Papers identified through citation tracking
• Round1/2 Snowballing Merge: Combined snowballing results
• Round1/2 Selection by Criteria: Selection decisions for snowballed papers

5. Data Extraction.xlsx

Contains the detailed data extracted from all selected papers to address the RQs (see Related Documents/Research Questions.md for details).

Data Extraction Scheme:

• Metadata: Basic paper information:
  • Title
  • Authors
  • Publication Year
  • Publication Venue
• Specific Data: Customized data extracted to answer RQ1-RQ4, including:
  • Fault Injection Configuration Generation (RQ1)
  • Fault Injection Configuration Execution (RQ2)
  • Test Oracle Identification (RQ3)
  • SUT for Evaluation (RQ4)

---

📄 Citation

If you use this replication package in your research, please cite the original paper:

@ARTICLE{yu2025systematic,
  author={Yu, Senyao and Wu, Huayao and Niu, Xintao and Nie, Changhai},
  journal={IEEE Transactions on Services Computing}, 
  title={A Systematic Literature Review on Fault Injection Testing of Microservice Systems}, 
  year={2025},
  volume={},
  number={},
  pages={1-22},
  keywords={Microservice architectures;Testing;Circuit faults;Resilience;Fault tolerant systems;Fault tolerance;Systematic literature review;Generators;Fault diagnosis;Benchmark testing;Microservice systems;fault injection testing;resilience;fault tolerance},
  doi={10.1109/TSC.2025.3621564}}

---

📧 Contact

For questions regarding this replication package, please contact yusy@smail.nju.edu.cn or hywu@nju.edu.cn.