Skip to content

AI_AGENTS.md

Latest commit

 

History

History
124 lines (98 loc) · 5.04 KB

AI_AGENTS.md

File metadata and controls

124 lines (98 loc) · 5.04 KB
Raw

Contributing to Tracer Nextflow Test Pipelines

This guide provides clear, actionable instructions for contributing to this repository of bioinformatics test pipelines. The project is organized to ensure reproducibility, maintainability, and ease of use for all contributors.

1. Project Structure: Self-Contained Examples and Environments

  • Self-Contained Examples:
    • Each subfolder under pipelines/ represents a single, fully self-contained example or environment. For example:
      • pipelines/macos-arm64/nextflow-pixi/ (Apple Silicon, Pixi-managed)
      • pipelines/linux-x86-ubuntu/nextflow-pixi/ (Linux x86, Pixi-managed)
      • pipelines/aws-batch/ (AWS Batch cloud execution)
    • Each example/environment must include all files needed to run and test it, including its own environment files (pixi.toml, pixi.lock), scripts, and test data.
    • Do not reference files or scripts outside the current example/environment folder, except for shared utilities in pipelines/scripts/.
    • This isolation ensures that each example can be run, tested, and maintained independently, reducing compatibility issues and debugging complexity.
pipelines/
├── macos-arm64/           # Apple Silicon (M1/M2) pipelines
│   └── nextflow-pixi/     # Pixi-managed Nextflow pipeline (self-contained)
├── linux-x86-ubuntu/      # Linux x86 (Ubuntu) pipelines
│   └── nextflow-pixi/
├── aws-batch/             # AWS Batch cloud execution
│   └── nextflow-pixi/
├── scripts/               # Shared utilities and helper scripts
└── ...
  • Test data for each pipeline is in its own test_data/ subfolder within the example.
  • Output, results, and work directories are auto-generated by Nextflow and should not be versioned.

2. CI/CD Requirement: Every Example Must Be Tested

  • Mandatory CI Example:
    • Every example/environment must have a dedicated GitHub Actions workflow file in .github/workflows/.
    • The workflow should:
      • Set up the environment (e.g., Pixi)
      • Install dependencies
      • Run the pipeline using the provided entry script (e.g., run.sh or start.sh)
      • Verify successful completion and, if possible, check for expected output files
    • Name the workflow file to match the environment (e.g., macos-arm64-nextflow-pixi.yml).
    • This ensures that every example is automatically tested and remains functional as the project evolves.

3. Environment Management: Use Pixi

  • Standardize on Pixi:
    • All new and existing pipeline examples must use Pixi as the environment and dependency manager.
    • Each environment folder must include its own pixi.toml and pixi.lock files.
    • Do not use Conda, Spack, or other managers for new examples.

4. Standardized Entry Point: start.sh

  • Entry Script:
    • Every environment folder must include a start.sh script at its root.
    • start.sh is the default entry point and should:
      • Activate the Pixi environment (if needed)
      • Run the pipeline with minimal user input (ideally zero arguments for a default test run)
      • Print clear usage/help if run with -h or --help
    • Example: 
      #!/usr/bin/env bash
pixi run pipeline

5. Shared Utilities and Scripts

  • Centralize Common Logic:
    • Place all shared setup logic, environment checks, and helper scripts in pipelines/scripts/.
    • Reference these scripts from start.sh or other scripts as needed, but do not duplicate logic across environment folders.
    • Example shared scripts: check_env.sh, setup_java.sh, validate_output.sh

6. Coding Conventions for Nextflow Pipelines

  • Workflow Language: Use Nextflow (preferably DSL2) for all new pipelines.
  • Naming:
    • Use descriptive, lowercase names for files and folders (e.g., main.nf, custom.config, start.sh).
    • Test data files should be clearly named (e.g., sample1.fasta).
  • Documentation:
    • Every environment folder must have a README.md describing:
      • Purpose and expected inputs/outputs
      • How to run the pipeline (with example commands)
      • System/environment requirements
  • Resource Management:
    • Specify CPU, memory, and time in each process block.
    • Use errorStrategy and maxRetries for robust execution.

7. Testing Protocols

  • Test Data: Place minimal test data in each pipeline's test_data/ directory.
  • How to Run Pipelines:
    • Local (Pixi/Conda): 
      # Example for Pixi
cd pipelines/macos-arm64/nextflow-pixi
pixi run pipeline
# Or for Conda
cd pipelines/linux-x86-ubuntu/nextflow-conda
bash run.sh
    • AWS Batch: 
      cd pipelines/aws-batch
make test_rnaseq_aws_batch
  • Validation: Pipelines must complete without error and produce expected output files in results/ or output/.
  • Cleanup: Use provided clean scripts/tasks to remove old results and logs.
  • CI/CD: If automated tests are set up, all pipelines must pass before merging.

Keep this file up to date as the project evolves.