[JENKINS-76312] Event-based Dynamic Job Creation

[jenkins-infra-bot]

Proposal Summary

This proposal outlines a native integration in the JobDSL plugin to automate the detection and processing of changes in
the .jobdsl folder (located at the root directory of the target repository) during SCM merge events from providers
like Gerrit, GitHub, BitBucket, and GitLab. By preprocessing scripts with dynamic placeholders and executing them
directly, the plugin would enable seamless, real-time job updates in Jenkins, reducing manual intervention and aligning
with GitOps principles. This eliminates reliance on external tools or scripts, leveraging Jenkins' built-in
configurations for scalability across multi-repo environments.

Key Objectives

• Automation: Respond to merges by scanning for changes in the .jobdsl folder at the repository root (including
  any subpaths recursively), preprocessing .groovy scripts (e.g., substituting placeholders like %%BRANCH%% and
  deriving job names from folder paths), and invoking JobDSL execution.
• Integration: Use Jenkins global and job-level configs for SCM credentials, processing rules, and multi-project
  handling.
• Outcomes: Instant job/folder sync post-merge, with built-in validation, logging, and security.

Detailed Proposal

Problem Statement:
In organizations with high-velocity development, JobDSL scripts organized in the .jobdsl folder at the repository root
require manual triggering after merges, disrupting workflows. Current setups demand custom scripts for placeholder
resolution (e.g., branch-aware paths) and duplicate checks, adding complexity.

Proposed Solution:
Extend the JobDSL plugin to include an SCM event listener that:

• Registers webhooks (or polls) for merge events, parsing payloads to detect changes in the .jobdsl folder at the
  repository root (including any recursive subpaths).
• Preprocesses affected .groovy files:
  • Resolves placeholders (e.g., %%BRANCH%% from merge target; %%JOB_NAME%% derived as "
    parent-project/repo/branch/script").
  • Applies configurable regex replacements and duplicate name prevention.
• Executes via GroovyDSLManager, using internals to auto-generate nested Jenkins folders from paths (e.g., "
  payment/checkout-svc/develop/job-script" → folders "payment > checkout-svc > develop" with job "job-script").
• Supports no-op for unchanged files, with console logging for traceability.

Illustrative Script Example:
A preprocessed snippet might resolve as:

pipelineJob(<span class="code-quote">'payment/checkout-svc/develop/job-script'</span>) {  <span class="code-comment">// Derived from path
</span>    description('')
    disabled(<span class="code-keyword">false</span>)
    parameters {
stringParam {
    trim(<span class="code-keyword">true</span>); name(<span class="code-quote">'GERRIT_BRANCH'</span>); defaultValue(<span class="code-quote">'develop'</span>)  <span class="code-comment">// %%BRANCH%% resolved
</span>    description(<span class="code-quote">'The branch or the target branch of the change.'</span>)
}
stringParam {
    trim(<span class="code-keyword">true</span>); name(<span class="code-quote">'GERRIT_REFSPEC'</span>); defaultValue(<span class="code-quote">'refs/heads/develop'</span>)
    description(<span class="code-quote">'Change refspec'</span>)
}
    }
    <span class="code-comment">// ... (triggers, definition, etc.)
</span>}

Configuration Approach:

• Global (Manage Jenkins > Configure System): Define shared SCM servers (e.g., Gerrit API URLs, credential IDs),
  placeholder mappings, regex rules, and project scopes (included/excluded lists).

Implementation Considerations:

• Leverage existing Jenkins plugins (e.g., Gerrit Trigger) for auth and payload handling.
• Focus on delta changes for performance; async processing for scale.

Acceptance Criteria:

• Successful end-to-end flow: Merge → Detection → Preprocess → Execute → Updated jobs/folders.
• Handles edge cases: No changes, errors, duplicates (fail-fast with logs).
• Secure: Webhook validation, credential isolation.
• Testable: Via mocks and e2e scenarios.

Background/Context:
Our setup uses hierarchical .jobdsl structures (a folder at the root of each repository) for branch-agnostic
automation:

.jobdsl/  # Root-level folder
├── parent-project/  # e.g., <span class="code-quote">"payment"</span>
│   └── repo/# e.g., <span class="code-quote">"checkout-svc"</span>
│       └── %%BRANCH%%/  # e.g., resolves to <span class="code-quote">"develop"</span>
│   └── job-script.groovy  # Job name derived from full path

This enables zero-touch scaling, but manual post-merge runs hinder efficiency. Native support would consolidate this
into the plugin.

Benefits:

• End-to-end automation from merge to job sync.
• Scales for multi-repo/branch velocity.
• Enforces org standards with minimal config.

---

Originally reported by gabrieljuliao, imported from: Event-based Dynamic Job Creation

• status: Open
• priority: Major
• component(s): job-dsl-plugin
• label(s): automation, enhancement, scm
• resolution: Unresolved
• votes: 0
• watchers: 1
• imported: 2025-12-03

Raw content of original issue

#### Proposal Summary

This proposal outlines a native integration in the JobDSL plugin to automate the detection and processing of changes in
the `.jobdsl` folder (located at the root directory of the target repository) during SCM merge events from providers
like Gerrit, GitHub, BitBucket, and GitLab. By preprocessing scripts with dynamic placeholders and executing them
directly, the plugin would enable seamless, real-time job updates in Jenkins, reducing manual intervention and aligning
with GitOps principles. This eliminates reliance on external tools or scripts, leveraging Jenkins' built-in
configurations for scalability across multi-repo environments.

#### Key Objectives

- **Automation:** Respond to merges by scanning for changes in the `.jobdsl` folder at the repository root (including
  any subpaths recursively), preprocessing `.groovy` scripts (e.g., substituting placeholders like `%%BRANCH%%` and
  deriving job names from folder paths), and invoking JobDSL execution.
- **Integration:** Use Jenkins global and job-level configs for SCM credentials, processing rules, and multi-project
  handling.
- **Outcomes:** Instant job/folder sync post-merge, with built-in validation, logging, and security.

#### Detailed Proposal

**Problem Statement:**  
In organizations with high-velocity development, JobDSL scripts organized in the `.jobdsl` folder at the repository root
require manual triggering after merges, disrupting workflows. Current setups demand custom scripts for placeholder
resolution (e.g., branch-aware paths) and duplicate checks, adding complexity.

**Proposed Solution:**  
Extend the JobDSL plugin to include an SCM event listener that:

- Registers webhooks (or polls) for merge events, parsing payloads to detect changes in the `.jobdsl` folder at the
  repository root (including any recursive subpaths).
- Preprocesses affected `.groovy` files:
    - Resolves placeholders (e.g., `%%BRANCH%%` from merge target; `%%JOB_NAME%%` derived as "
      parent-project/repo/branch/script").
    - Applies configurable regex replacements and duplicate name prevention.
- Executes via `GroovyDSLManager`, using internals to auto-generate nested Jenkins folders from paths (e.g., "
  payment/checkout-svc/develop/job-script" → folders "payment > checkout-svc > develop" with job "job-script").
- Supports no-op for unchanged files, with console logging for traceability.

**Illustrative Script Example:**  
A preprocessed snippet might resolve as:

```groovy
pipelineJob('payment/checkout-svc/develop/job-script') {  // Derived from path
    description('')
    disabled(false)
    parameters {
        stringParam {
            trim(true); name('GERRIT_BRANCH'); defaultValue('develop')  // %%BRANCH%% resolved
            description('The branch or the target branch of the change.')
        }
        stringParam {
            trim(true); name('GERRIT_REFSPEC'); defaultValue('refs/heads/develop')
            description('Change refspec')
        }
    }
    // ... (triggers, definition, etc.)
}
```

**Configuration Approach:**

- **Global (Manage Jenkins > Configure System):** Define shared SCM servers (e.g., Gerrit API URLs, credential IDs),
  placeholder mappings, regex rules, and project scopes (included/excluded lists).

**Implementation Considerations:**

- Leverage existing Jenkins plugins (e.g., Gerrit Trigger) for auth and payload handling.
- Focus on delta changes for performance; async processing for scale.

**Acceptance Criteria:**

- Successful end-to-end flow: Merge → Detection → Preprocess → Execute → Updated jobs/folders.
- Handles edge cases: No changes, errors, duplicates (fail-fast with logs).
- Secure: Webhook validation, credential isolation.
- Testable: Via mocks and e2e scenarios.

**Background/Context:**  
Our setup uses hierarchical `.jobdsl` structures (a folder at the root of each repository) for branch-agnostic
automation:

```
.jobdsl/  # Root-level folder
├── parent-project/  # e.g., "payment"
│   └── repo/        # e.g., "checkout-svc"
│       └── %%BRANCH%%/  # e.g., resolves to "develop"
│           └── job-script.groovy  # Job name derived from full path
```

This enables zero-touch scaling, but manual post-merge runs hinder efficiency. Native support would consolidate this
into the plugin.

**Benefits:**

- End-to-end automation from merge to job sync.
- Scales for multi-repo/branch velocity.
- Enforces org standards with minimal config. 

[GabrielJuliao]

Hi, the snippet formatting is wrong, here's the correct one

pipelineJob('%%JOB_NAME%%') {
    description('')
    disabled(false)
    parameters {
        stringParam { trim(true); name('GERRIT_BRANCH'); defaultValue('%%BRANCH%%')
            description('The branch or the target branch of the change.')
        }
        stringParam { trim(true); name('GERRIT_REFSPEC'); defaultValue('refs/heads/%%BRANCH%%')
            description('Change refspec')
        }
    }
    properties {
        pipelineTriggers {
            triggers {
                gerrit {
                    serverName('%%SERVER_NAME%%')
                    gerritProjects {
                        gerritProject {
                            compareType('PLAIN')
                            pattern('api/payment-service')
                            branches { 
                                branch { compareType('PLAIN'); pattern('%%BRANCH%%') }
                            }
                            disableStrictForbiddenFileVerification(true)
                        }
                    }
                    triggerOnEvents {
                        draftPublished()
                        patchsetCreated {                            
                            commitMessageContainsRegEx('')
                            excludeDrafts(true)
                            excludeNoCodeChange(false)
                            excludePrivateState(false)
                            excludeTrivialRebase(false)
                            excludeWipState(false)
                            uploaderNameContainsRegEx('') 
                        }
                    }
                }
            }
        }
    }
    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        url('http://gerrit:8080/a/api/payment-service')
                        credentials('gerrit')
                        refspec('$GERRIT_REFSPEC')
                    }
                    branch('$GERRIT_BRANCH')
                    extensions {
                        choosingStrategy {
                            gerritTrigger()
                        }
                        cloneOptions {
                            honorRefspec(true)
                            noTags(false)
                            depth(1)
                            shallow(false)
                        }
                        submoduleOptions {
                            recursive(true)
                            parentCredentials(true)
                        }
                    }
                }
                scriptPath('Jenkinsfile')
            }
        }
    }
}