Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
Analysis
Analysis
 
 
data
data
 
 
prompts
prompts
 
 
scripts
scripts
 
 
README.md
README.md
 
 

Repository files navigation

Replication Package

Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study

Venue: PROMPT-SE 2026 — 1st International Workshop on Empirical Prompt Engineering for Software Engineering
Co-located with: EASE 2026 — 30th International Conference on Evaluation and Assessment in Software Engineering
Date: June 9–12, 2026 | Glasgow, Scotland, United Kingdom
Repository: https://github.com/moaathalshaikh/PROMPT-SE-2026-Replication-Package
DOI: [to be assigned upon publication]

Authors

# Name ORCID Institution Country
1 Moaath Alshaikh 0000-0002-6484-4486 Federal University of Bahia Brazil
2 Tasneem Alshaher 0009-0000-1336-1671 Federal University of Bahia Brazil
3 Ricardo Vieira 0009-0007-1927-7957 Federal University of Bahia Brazil
4 Beatriz Santana 0000-0003-4048-1921 Federal University of Bahia Brazil
5 Clélio Xavier — Federal University of Bahia Brazil
6 José Amâncio Santos 0000-0002-9509-5238 State University of Feira de Santana Brazil
7 Glauco Carneiro 0000-0001-6241-1612 Federal University of Sergipe Brazil
8 Júlio César Leite 0000-0002-0355-0265 Federal University of Bahia Brazil
9 Sávio Freire 0000-0002-3989-9442 Federal Institute of Ceará Brazil
10 Manoel Mendonça (supervisor) 0000-0002-0874-7665 Federal University of Bahia Brazil

Corresponding author: Moaath Alshaikh — moaathalshaikh@ufba.br

Overview

This replication package supports full reproduction of the controlled empirical study presented in the paper. The study evaluates three Large Language Models (LLMs) across two prompt engineering strategies applied to qualitative coding of psychological safety statements from Stack Exchange software engineering communities.

Dimension Detail
Models evaluated Claude Haiku (claude-haiku-4-5-20251001), DeepSeek-Chat (deepseek-chat), Gemini 2.5 Flash (gemini-2.5-flash)
Prompt strategies P01 — Zero-Shot Closed Coding; P02 — Multi-Shot Closed Coding
Independent runs 10 per model–prompt configuration
Total runs 60 (3 models × 2 prompts × 10 runs)
Temperature 0 (all models)
Dataset 116 quotes from Stack Exchange (SEPM + SESE communities)
Coding framework Edmondson (1999) — 7 behavioral categories
Primary metric Cohen's κ (LLM vs. human Gold Standard)
Secondary metrics Intra-model stability (SD of κ), Bias Ratio, per-class F1
Execution environment Google Colab (Python 3.12)

How to Cite

If you use any part of this replication package in your research, please cite:

@inproceedings{alshaikh2026prompt,
  author    = {Alshaikh, Moaath and Alshaher, Tasneem and Vieira, Ricardo
               and Santana, Beatriz and Xavier, Cl{\'e}lio and Santos, Jos{\'e} Am{\^a}ncio
               and Carneiro, Glauco and Leite, J{\'u}lio C{\'e}sar
               and Freire, S{\'a}vio and Mendon{\c{c}}a, Manoel},
  title     = {Prompt Engineering Strategies for {LLM}-based Qualitative Coding
               of Psychological Safety in Software Engineering Communities:
               A Controlled Empirical Study},
  booktitle = {Proceedings of the 1st International Workshop on Empirical
               Prompt Engineering for Software Engineering (PROMPT-SE 2026)},
  series    = {EASE 2026},
  year      = {2026},
  location  = {Glasgow, Scotland, United Kingdom},
  publisher = {ACM},
  doi       = {to be assigned}
}

Repository Structure

PROMPT-SE-2026-Replication-Package/
├── data/
│   ├── quotes.csv
│   └── gold_standard.csv
├── prompts/
│   ├── P01_Zero_Shot_Closed_Coding.txt
│   └── P02_Multi_Shot_Closed_Coding.txt
├── scripts/
│   └── run_Multi_prompt.py
├── Analysis/
│   └── analysis_pipeline.py
└── README.md

Data Description

data/quotes.csv

Quotes collected from two Stack Exchange communities: Project Management (SEPM) and Software Engineering (SESE).

Column Type Description
quote_id integer Unique quote identifier
quote string Raw text of the quote
source string Stack Exchange community of origin

data/gold_standard.csv

Human-coded reference labels produced by Santana et al. (2023) through a multi-researcher consensus process. Inter-researcher disagreement rate: 4.67%. No formal IRR statistic (e.g., Cohen's κ) is reported in the original study.

Column Type Description
quote_id integer Unique quote identifier (matches quotes.csv)
gold_label string Human-assigned psychological safety category

Label Distribution (Gold Standard)

Code Category n %
EC Expressing Concerns 50 43.1
DS Disagreeing with Suggestions or Ideas 35 30.2
RC Recommending Changes 14 12.1
SH Seeking Help 5 4.3
DAE Drawing Attention to Errors 5 4.3
SNF Sharing Negative Feedback 4 3.4
AM Admitting Mistakes 3 2.6
Total 116 100

Reproduction Steps

1. Requirements

pip install anthropic openai google-generativeai \
            pandas numpy scikit-learn scipy seaborn matplotlib

Python 3.12 is recommended (tested on Google Colab).

2. Configure API Keys

Set environment variables before running:

export ANTHROPIC_API_KEY="your-anthropic-key"   # Claude Haiku
export DEEPSEEK_API_KEY="your-deepseek-key"     # DeepSeek-Chat
export GOOGLE_API_KEY="your-google-key"         # Gemini 2.5 Flash

In Google Colab, store keys in the Secrets panel and access via os.environ.get("KEY_NAME").

3. Run LLM Classification

Edit the two configuration lines at the top of scripts/run_Multi_prompt.py:

MODEL_NAME  = "claude-haiku-4-5-20251001"      # or "deepseek-chat" / "gemini-2.5-flash"
PROMPT_NAME = "P01_Zero_Shot_Closed_Coding"    # or "P02_Multi_Shot_Closed_Coding"

Then run:

python scripts/run_Multi_prompt.py

The script executes 10 independent runs and saves each run as a JSON file under results/.
Repeat for all 6 configurations (3 models × 2 prompts).

4. Run Analysis Pipeline

Once all 60 runs are complete:

python Analysis/analysis_pipeline.py

This produces all tables (Cohen's κ, Wilcoxon, stability, bias) and figures used in the paper.

Note: Pre-computed run results (dataset_llm_runs.csv) are not included in this package in order to encourage full reproduction. If you are unable to re-run the experiments due to API access constraints, you may request the pre-computed dataset from the corresponding author at moaathalshaikh@ufba.br.

Prompt Design

Both prompts share the same schema (role assignment, framework definition, step-by-step instructions, category definitions, output format) and differ only in the annotated examples section:

ID Strategy Examples per category Description
P01 Zero-Shot 0 Category names and classification rules only
P02 Multi-Shot 7 (one per category) Includes one annotated example per category

This design isolates the effect of in-context examples while keeping all other prompt elements identical.

License

This replication package is licensed under the
Creative Commons Attribution 4.0 International License (CC BY 4.0).

You are free to share and adapt this material for any purpose, provided that appropriate credit is given, a link to the license is included, and any modifications are indicated.

Related Work

  • Santana, B. et al. (2023). Psychological Safety in Software Engineering Communities on Stack Exchange. UFBA. (Gold Standard source)
  • Edmondson, A. C. (1999). Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly, 44(2), 350–383. (Coding framework)

Contact

Moaath Alshaikh
Ph.D. Candidate — PGCOMP Graduate Program in Computer Science
Federal University of Bahia (UFBA), Salvador, Bahia, Brazil
moaathalshaikh@ufba.br
https://orcid.org/0000-0002-6484-4486

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages