Skip to content

Commit cd7f3a4

Browse files
chrisgeochrisgeo
authored
feat(gitlab): add GitLab fixture data generator (CHAOS-246 part 2/2) (#16)
* feat(gitlab): add Terraform fixture scaffold for CHAOS-246 * feat(gitlab): add seeder story map for CHAOS-246 * feat(gitlab): document fixture generator for CHAOS-246
1 parent 091cbf0 commit cd7f3a4

8 files changed

Lines changed: 1437 additions & 0 deletions

File tree

‎gitlab/README.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# GitLab Demo Data Seeder
2+
3+
This module provisions GitLab structure and invokes a deterministic seeder to generate narrative-shaped GitLab fixture data for Developer Health analytics.
4+
5+
## What this creates
6+
7+
- 1 GitLab group
8+
- 10 GitLab projects mirroring the Atlassian fixture project set
9+
- GitLab issues across a 24-month timeline
10+
- Merge requests with reviewers, optional comments, and fixture commits
11+
- CI pipeline triggers backed by a generated `.gitlab-ci.yml` with build/test/security/deploy stages and jobs
12+
- Tags and releases aligned to story arcs
13+
- `out/manifest.json` with distribution summaries, pipeline success/failure rates, and Investment View theme counts
14+
15+
## Requirements
16+
17+
- GitLab.com or self-managed GitLab with API access
18+
- Personal access token with `api` scope
19+
- Python 3 with `pip3`
20+
- Terraform provider `gitlabhq/gitlab`
21+
22+
## Environment variables
23+
24+
The seeder reads authentication from environment variables:
25+
26+
- `GITLAB_TOKEN` — GitLab personal access token (required outside `--dry-run`)
27+
- `GITLAB_BASE_URL` — GitLab API URL (optional, default `https://gitlab.com/api/v4`)
28+
- `GITLAB_GROUP_PATH` — target group path for direct seeder runs (optional, default `dev-health-demo`)
29+
- `GITLAB_REVIEWERS` — comma-separated usernames eligible for MR reviews (optional)
30+
31+
Terraform equivalents:
32+
33+
- `gitlab_token`
34+
- `gitlab_base_url`
35+
- `group_name`
36+
- `enable_group_creation`
37+
- `enable_project_creation`
38+
- `enable_seed_creation` (default `false`, so Terraform runs the seeder in dry-run mode)
39+
- `reviewer_usernames`
40+
41+
Optional toggles:
42+
43+
- `enable_comments` (default `false`)
44+
- `enable_pipelines` (default `true`)
45+
- `enable_merge_requests` (default `true`)
46+
- `enable_releases` (default `true`)
47+
48+
## Usage
49+
50+
```bash
51+
cd gitlab
52+
terraform init
53+
terraform apply
54+
```
55+
56+
Dry-run is the default in Terraform (`enable_seed_creation = false`). To create GitLab data:
57+
58+
```hcl
59+
gitlab_token = "..."
60+
gitlab_base_url = "https://gitlab.com/api/v4"
61+
group_name = "Dev Health Demo"
62+
enable_seed_creation = true
63+
reviewer_usernames = ["alice", "bob"]
64+
```
65+
66+
Run the seeder directly in dry-run mode:
67+
68+
```bash
69+
python -m venv .venv
70+
source .venv/bin/activate
71+
pip install -r gitlab/seed/requirements.txt
72+
python gitlab/seed/seed_gitlab.py --dry-run
73+
```
74+
75+
Run the seeder directly against GitLab:
76+
77+
```bash
78+
export GITLAB_TOKEN="..."
79+
export GITLAB_BASE_URL="https://gitlab.com/api/v4"
80+
python gitlab/seed/seed_gitlab.py \
81+
--group-path dev-health-demo \
82+
--reviewers alice,bob \
83+
--enable-comments
84+
```
85+
86+
## Story map
87+
88+
`seed/story_map.yaml` defines the 24-month narrative arcs:
89+
90+
- Launch
91+
- Scale
92+
- Reliability Crunch
93+
- Recovery
94+
- Roadmap Reset
95+
96+
Each arc includes Investment View canonical theme mix (`Feature Delivery`, `Operational / Support`, `Maintenance / Tech Debt`, `Quality / Reliability`, `Risk / Security`), issue mix, merge request behavior, and pipeline success/failure rates.
97+
98+
## Idempotency and timestamps
99+
100+
The seeder uses deterministic external IDs (`extid::<hash>`) derived from the story map, project, month, and work type. Re-runs skip existing seeded issues by label. GitLab does not generally allow arbitrary historical pipeline/job timestamps through public APIs, so simulated dates and arc metadata are stored in labels, descriptions, and `out/manifest.json`; live GitLab resources are created at run time.
101+
102+
## Reset / destroy
103+
104+
```bash
105+
terraform destroy
106+
```
107+
108+
Destroy removes Terraform-managed groups/projects. The Python seeder does not independently delete issues, merge requests, pipelines, or releases inside projects.

‎gitlab/main.tf‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
terraform {
2+
required_providers {
3+
gitlab = {
4+
source = "gitlabhq/gitlab"
5+
version = "~> 17.0"
6+
}
7+
null = {
8+
source = "hashicorp/null"
9+
}
10+
}
11+
}
12+
13+
provider "gitlab" {
14+
token = var.gitlab_token
15+
base_url = var.gitlab_base_url
16+
}
17+
18+
locals {
19+
group_path = lower(replace(var.group_name, " ", "-"))
20+
21+
project_map = {
22+
core = "Core"
23+
platform = "Platform"
24+
data = "Data"
25+
infra = "Infra"
26+
security = "Security"
27+
mobile = "Mobile"
28+
web = "Web"
29+
integrations = "Integrations"
30+
payments = "Payments"
31+
reliability = "Reliability"
32+
}
33+
34+
dry_run_flag = var.enable_seed_creation ? "" : "--dry-run"
35+
comments_flag = var.enable_comments ? "--enable-comments" : ""
36+
pipelines_flag = var.enable_pipelines ? "" : "--disable-pipelines"
37+
merge_requests_flag = var.enable_merge_requests ? "" : "--disable-merge-requests"
38+
releases_flag = var.enable_releases ? "" : "--disable-releases"
39+
start_date_flag = var.provision_start_date != "" ? "--start-date ${var.provision_start_date}" : ""
40+
end_date_flag = var.provision_end_date != "" ? "--end-date ${var.provision_end_date}" : ""
41+
monthly_issue_flag = var.monthly_issue_count != 0 ? "--monthly-issue-count ${var.monthly_issue_count}" : ""
42+
reviewers_flag = length(var.reviewer_usernames) > 0 ? "--reviewers ${join(",", var.reviewer_usernames)}" : ""
43+
date_range_valid = var.provision_end_date == "" || var.provision_start_date != ""
44+
}
45+
46+
module "gitlab_structure" {
47+
source = "./modules/gitlab_structure"
48+
49+
group_name = var.group_name
50+
group_path = local.group_path
51+
project_map = local.project_map
52+
enable_group_creation = var.enable_group_creation
53+
enable_project_creation = var.enable_project_creation
54+
project_visibility = var.project_visibility
55+
}
56+
57+
resource "null_resource" "seed" {
58+
depends_on = [module.gitlab_structure]
59+
60+
lifecycle {
61+
precondition {
62+
condition = local.date_range_valid
63+
error_message = "provision_start_date is required when provision_end_date is set."
64+
}
65+
}
66+
67+
triggers = {
68+
story_map_hash = filesha256("${path.module}/seed/story_map.yaml")
69+
script_hash = filesha256("${path.module}/seed/seed_gitlab.py")
70+
seed_hash = sha256(var.seed_string)
71+
batch_size = tostring(var.batch_size)
72+
dry_run = tostring(var.enable_seed_creation)
73+
start_date = var.provision_start_date
74+
end_date = var.provision_end_date
75+
monthly_issues = tostring(var.monthly_issue_count)
76+
}
77+
78+
provisioner "local-exec" {
79+
command = join(" ", compact([
80+
"pip3 install -q -r ${path.module}/seed/requirements.txt &&",
81+
"mkdir -p ${path.module}/out &&",
82+
"python3",
83+
"${path.module}/seed/seed_gitlab.py",
84+
"--base-url", var.gitlab_base_url,
85+
"--group-path", module.gitlab_structure.group_path,
86+
"--story", "${path.module}/seed/story_map.yaml",
87+
"--manifest", "${path.module}/out/manifest.json",
88+
"--seed", var.seed_string,
89+
"--batch-size", tostring(var.batch_size),
90+
local.start_date_flag,
91+
local.end_date_flag,
92+
local.monthly_issue_flag,
93+
local.reviewers_flag,
94+
local.dry_run_flag,
95+
local.comments_flag,
96+
local.pipelines_flag,
97+
local.merge_requests_flag,
98+
local.releases_flag,
99+
]))
100+
101+
environment = {
102+
GITLAB_TOKEN = var.gitlab_token
103+
}
104+
}
105+
}

‎gitlab/modules/gitlab_structure/main.tf‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
terraform {
2+
required_providers {
3+
gitlab = {
4+
source = "gitlabhq/gitlab"
5+
}
6+
}
7+
}
8+
9+
resource "gitlab_group" "demo" {
10+
count = var.enable_group_creation ? 1 : 0
11+
12+
name = var.group_name
13+
path = var.group_path
14+
description = "Seeded group for developer health demo"
15+
visibility_level = "private"
16+
}
17+
18+
data "gitlab_group" "existing" {
19+
count = var.enable_group_creation ? 0 : 1
20+
21+
full_path = var.group_path
22+
}
23+
24+
locals {
25+
group_id = var.enable_group_creation ? gitlab_group.demo[0].id : data.gitlab_group.existing[0].id
26+
group_path = var.enable_group_creation ? gitlab_group.demo[0].full_path : data.gitlab_group.existing[0].full_path
27+
}
28+
29+
resource "gitlab_project" "projects" {
30+
for_each = var.enable_project_creation ? var.project_map : {}
31+
32+
name = each.value
33+
path = each.key
34+
namespace_id = local.group_id
35+
description = "Seeded ${each.value} project for developer health demo"
36+
visibility_level = var.project_visibility
37+
initialize_with_readme = true
38+
}
39+
40+
output "group_id" {
41+
description = "GitLab group id used for seeded projects"
42+
value = local.group_id
43+
}
44+
45+
output "group_path" {
46+
description = "GitLab group full path used by the seeder"
47+
value = local.group_path
48+
}
49+
50+
output "project_paths" {
51+
description = "Full paths for projects created by Terraform"
52+
value = var.enable_project_creation ? {
53+
for key, project in gitlab_project.projects : key => project.path_with_namespace
54+
} : {}
55+
}

‎gitlab/modules/gitlab_structure/variables.tf‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
variable "group_name" {
2+
type = string
3+
description = "GitLab group display name"
4+
}
5+
6+
variable "group_path" {
7+
type = string
8+
description = "GitLab group URL path"
9+
}
10+
11+
variable "project_map" {
12+
type = map(string)
13+
description = "Map of GitLab project path to display name"
14+
}
15+
16+
variable "enable_group_creation" {
17+
type = bool
18+
description = "Enable GitLab group creation"
19+
default = true
20+
}
21+
22+
variable "enable_project_creation" {
23+
type = bool
24+
description = "Enable GitLab project creation"
25+
default = true
26+
}
27+
28+
variable "project_visibility" {
29+
type = string
30+
description = "Visibility level for generated projects"
31+
default = "private"
32+
}

‎gitlab/seed/requirements.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
PyYAML==6.0.1
2+
requests==2.32.4

0 commit comments

Comments
 (0)