improve dlt dev workflow

[grst]

In this PR I attempt to improve the development experience for someone trying to add new metrics:

• update documentation. consolidate and update existing READMEs. In particular update the section how to confiure secrets and use a local duckdb instance for testing
• added a lightweight CLI to the python scripts using cyclopts. It allows to
  • switch to a local duckdb instance without patching code
  • only run selected resources for testing.
  • When calling without any arguments it behaves as before (default values are set accordingly)

[netlify]

✅ Deploy Preview for nf-core-stats ready!

Name	Link
🔨 Latest commit	02e5ee5
🔍 Latest deploy log	https://app.netlify.com/projects/nf-core-stats/deploys/6901f068ab90570007168337
😎 Deploy Preview	https://deploy-preview-91--nf-core-stats.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[claude]

Claude finished @grst's task —— View job

---

PR Review: Improve DLT Dev Workflow

This PR successfully improves the development experience with well-executed changes. Here's my comprehensive review:

✅ Strengths

Documentation Improvements:

• Consolidated documentation: Great consolidation by removing pipeline/docs/README.md and enhancing the main pipeline/README.md
• Detailed setup instructions: Clear environment variable documentation and local development guidance
• CLI usage examples: Good documentation of the new CLI options with practical examples

CLI Implementation (cyclopts):

• Clean integration: Minimal, focused addition that maintains backward compatibility
• Type safety: Good use of Literal types for resource selection in github_pipeline.py:534-536
• Sensible defaults: All parameters have appropriate defaults, so existing workflows continue unchanged
• Developer-friendly: The --destination duckdb flag addresses a real pain point for local development

Code Quality:

• Consistent patterns: Both pipelines follow the same CLI pattern
• Proper imports: Clean import structure with cyclopts properly added to dependencies
• Minimal surface area: Changes are focused and don't introduce unnecessary complexity

🔍 Minor Issues & Suggestions

Code Quality:

1. Missing parameter documentation in slack_pipeline.py:217: The main() function lacks docstring parameter details compared to the comprehensive documentation in github_pipeline.py:531-552

2. Type hint inconsistency:

   • github_pipeline.py:534-537: Good use of Literal for resource types
   • Consider similar type safety improvements for destination parameter: destination: Literal["motherduck", "duckdb"] = "motherduck"

Security & Configuration:
3. Environment variable handling: The documentation correctly shows DLT's magic environment variables, but consider adding a note about .env file security (already in .gitignore, which is good)

4. Default values alignment: Both pipelines default to motherduck which is appropriate for production, and the CLI allows easy override for development

🚀 Performance & Best Practices

Positive aspects:

• No performance regression: CLI additions are minimal and don't affect runtime performance
• Selective resource execution: The ability to run specific resources with --resources is excellent for development iteration
• Traffic optimization settings: The configurable traffic_only_active_repos and traffic_max_repos parameters provide good control over API usage

Architecture:

• Single responsibility: Each change serves a clear purpose in improving developer experience
• Backward compatibility: Existing workflows and CI/CD pipelines remain unaffected

📝 Test Coverage

While no tests are added in this PR, the changes are low-risk:

• CLI parameter parsing is handled by the well-tested cyclopts library
• The actual pipeline logic remains unchanged
• Default behavior is preserved

Recommendation: Consider adding a simple test to verify CLI argument parsing works correctly, but this is not blocking for this PR.

🏁 Overall Assessment

This is a well-executed PR that achieves its goals effectively:

• ✅ Improves developer experience through local DuckDB support and selective resource execution
• ✅ Maintains production stability with sensible defaults and backward compatibility
• ✅ High code quality with consistent patterns and good documentation
• ✅ Low risk due to minimal, focused changes

Recommendation: APPROVE ✅

The changes are production-ready and provide clear value to developers working with the pipeline. The minor suggestions above are enhancements for future consideration rather than blocking issues.

---