From a41c2a5c328baa32b9b52e89a334cf952ecb87aa Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Sun, 2 Nov 2025 13:05:50 +0100 Subject: [PATCH] feat(skills): add comprehensive Pulumi import skill for click-ops infrastructure Add complete import workflow skill for bringing manually-created infrastructure under Pulumi management: - Core import workflows (single resource, bulk import) - Advanced patterns with automated AWS resource discovery - Real-world examples (S3 buckets, IAM roles, GitHub repos) - discover-aws-resources.py helper script for automation - Safety-first verification and preview steps - 1Password + direnv credential integration --- .claude/skills/pulumi/README.md | 404 +++++++++++ .claude/skills/pulumi/deploy/SKILL.md | 142 ++++ .claude/skills/pulumi/deploy/reference.md | 371 ++++++++++ .claude/skills/pulumi/documentation/SKILL.md | 264 +++++++ .../pulumi/documentation/providers/aws.md | 530 ++++++++++++++ .../pulumi/documentation/providers/github.md | 470 +++++++++++++ .../documentation/providers/onepassword.md | 460 +++++++++++++ .../skills/pulumi/documentation/reference.md | 342 ++++++++++ .claude/skills/pulumi/import/SKILL.md | 290 ++++++++ .../pulumi/import/examples/aws-iam-role.md | 363 ++++++++++ .../pulumi/import/examples/aws-s3-bucket.md | 423 ++++++++++++ .../import/examples/github-repository.md | 388 +++++++++++ .claude/skills/pulumi/import/reference.md | 644 ++++++++++++++++++ .../import/scripts/discover-aws-resources.py | 342 ++++++++++ .claude/skills/pulumi/new-project/SKILL.md | 339 +++++++++ .../skills/pulumi/new-project/reference.md | 470 +++++++++++++ .../pulumi/new-project/templates/Pulumi.yaml | 6 + .../new-project/templates/envrc.template | 19 + .../new-project/templates/main_py.template | 60 ++ .../new-project/templates/pyproject.toml | 33 + .../skills/pulumi/stack-management/SKILL.md | 233 +++++++ .../pulumi/stack-management/reference.md | 412 +++++++++++ .../stack-management/troubleshooting.md | 564 +++++++++++++++ 23 files changed, 7569 insertions(+) create mode 100644 .claude/skills/pulumi/README.md create mode 100644 .claude/skills/pulumi/deploy/SKILL.md create mode 100644 .claude/skills/pulumi/deploy/reference.md create mode 100644 .claude/skills/pulumi/documentation/SKILL.md create mode 100644 .claude/skills/pulumi/documentation/providers/aws.md create mode 100644 .claude/skills/pulumi/documentation/providers/github.md create mode 100644 .claude/skills/pulumi/documentation/providers/onepassword.md create mode 100644 .claude/skills/pulumi/documentation/reference.md create mode 100644 .claude/skills/pulumi/import/SKILL.md create mode 100644 .claude/skills/pulumi/import/examples/aws-iam-role.md create mode 100644 .claude/skills/pulumi/import/examples/aws-s3-bucket.md create mode 100644 .claude/skills/pulumi/import/examples/github-repository.md create mode 100644 .claude/skills/pulumi/import/reference.md create mode 100755 .claude/skills/pulumi/import/scripts/discover-aws-resources.py create mode 100644 .claude/skills/pulumi/new-project/SKILL.md create mode 100644 .claude/skills/pulumi/new-project/reference.md create mode 100644 .claude/skills/pulumi/new-project/templates/Pulumi.yaml create mode 100644 .claude/skills/pulumi/new-project/templates/envrc.template create mode 100644 .claude/skills/pulumi/new-project/templates/main_py.template create mode 100644 .claude/skills/pulumi/new-project/templates/pyproject.toml create mode 100644 .claude/skills/pulumi/stack-management/SKILL.md create mode 100644 .claude/skills/pulumi/stack-management/reference.md create mode 100644 .claude/skills/pulumi/stack-management/troubleshooting.md diff --git a/.claude/skills/pulumi/README.md b/.claude/skills/pulumi/README.md new file mode 100644 index 00000000..9e702371 --- /dev/null +++ b/.claude/skills/pulumi/README.md @@ -0,0 +1,404 @@ +# Pulumi Workflow Skills + +Comprehensive Claude Code skills for working with Pulumi infrastructure-as-code. + +## Overview + +This skill collection provides expert guidance for Pulumi workflows, including deployment, stack management, project creation, and documentation access. Designed for the nf-core/ops workflow with 1Password + direnv credential management. + +## Available Skills + +### 1. Deploying Pulumi Infrastructure (`deploy/`) + +**Triggers**: "deploy", "pulumi up", "preview", "apply changes" + +Preview and deploy infrastructure changes safely: + +- Standard deployment workflow +- Preview before deploy with user confirmation +- 1Password credential loading +- Error handling and troubleshooting +- Advanced deployment patterns (targeted, parallel, CI/CD) + +**Files**: + +- `SKILL.md` - Main deployment workflow +- `reference.md` - Advanced deployment patterns + +### 2. Managing Pulumi Stacks (`stack-management/`) + +**Triggers**: "stack", "switch stack", "stack output", "configuration" + +Manage multiple Pulumi stacks and environments: + +- List and switch stacks +- View stack outputs +- Manage stack configuration +- Multi-environment workflows +- Cross-stack references + +**Files**: + +- `SKILL.md` - Core stack operations +- `reference.md` - Advanced stack patterns +- `troubleshooting.md` - Common issues and solutions + +### 3. Creating New Pulumi Projects (`new-project/`) + +**Triggers**: "new project", "initialize pulumi", "scaffold", "start new infrastructure" + +Initialize new Pulumi projects with proper structure: + +- Interactive project setup +- Template files (.envrc, pyproject.toml, **main**.py) +- 1Password integration by default +- Best practices and conventions +- Component resources and testing + +**Files**: + +- `SKILL.md` - Project initialization workflow +- `reference.md` - Advanced project patterns +- `templates/` - Template files for new projects + - `envrc.template` - 1Password + direnv configuration + - `pyproject.toml` - Python dependencies + - `main_py.template` - Main Pulumi program + - `Pulumi.yaml` - Project configuration + +### 4. Accessing Pulumi Documentation (`documentation/`) + +**Triggers**: "pulumi docs", "how to", "aws pulumi", "github pulumi" + +Quick access to Pulumi documentation and provider guides: + +- WebSearch integration for latest docs +- Cached provider guides (AWS, GitHub, 1Password) +- Common patterns and examples +- API references + +**Files**: + +- `SKILL.md` - Documentation access patterns +- `reference.md` - Advanced topics +- `providers/aws.md` - AWS provider guide +- `providers/github.md` - GitHub provider guide +- `providers/onepassword.md` - 1Password provider guide + +### 5. Importing Existing Infrastructure (`import/`) + +**Triggers**: "import", "click-ops", "existing infrastructure", "bring under management" + +Import manually created "click-ops" resources into Pulumi: + +- Single resource import workflow +- Bulk import with JSON files +- Automated resource discovery +- Verification and safety checks +- Provider-specific import patterns + +**Files**: + +- `SKILL.md` - Core import workflows +- `reference.md` - Advanced import patterns +- `examples/aws-s3-bucket.md` - S3 import walkthrough +- `examples/aws-iam-role.md` - IAM import example +- `examples/github-repository.md` - GitHub import example +- `scripts/discover-aws-resources.py` - AWS discovery helper + +## Key Features + +### Progressive Disclosure + +Skills follow progressive disclosure patterns: + +- **SKILL.md**: Concise overview with common operations +- **reference.md**: Detailed patterns for advanced use cases +- **Supporting files**: Provider guides, templates, troubleshooting + +### 1Password Integration + +All skills include 1Password + direnv patterns: + +- Automatic credential loading +- No secrets in code +- Consistent across all projects +- Easy credential rotation + +### Safety First + +Deployment skills include safety confirmations: + +- Always preview before deploy +- Confirm with user for production changes +- Highlight destructive operations +- Export state before major changes + +### nf-core/ops Conventions + +Skills follow nf-core/ops patterns: + +- S3 backend for state +- Standard project structure +- Tagging conventions +- Naming patterns + +## Usage Examples + +### Deploy Infrastructure + +``` +User: "Deploy the co2_reports infrastructure" + +Claude will: +1. Load credentials from .envrc (1Password) +2. Run pulumi preview +3. Show changes and ask for confirmation +4. Deploy with pulumi up +5. Verify outputs +``` + +### Switch Stacks + +``` +User: "Switch to production stack" + +Claude will: +1. List available stacks +2. Switch to production +3. Verify current stack +4. Show stack outputs +``` + +### Create New Project + +``` +User: "Create a new Pulumi project for monitoring infrastructure" + +Claude will: +1. Ask for project details +2. Create directory structure +3. Generate files from templates +4. Set up dependencies +5. Initialize stack +6. Guide through first deployment +``` + +### Access Documentation + +``` +User: "How do I create an S3 bucket with encryption in Pulumi?" + +Claude will: +1. Load AWS provider guide +2. Show S3 bucket example with encryption +3. Provide code snippet +4. Link to latest docs if needed +``` + +### Import Existing Infrastructure + +``` +User: "Import the nf-core-co2-reports S3 bucket into Pulumi" + +Claude will: +1. Identify resource type and ID +2. Create import JSON or command +3. Execute import +4. Generate Pulumi code +5. Verify no changes in preview +6. Add protection if critical resource +``` + +## Project Structure + +``` +pulumi-workflow/ +├── README.md # This file +├── deploy/ +│ ├── SKILL.md # Deployment workflow +│ └── reference.md # Advanced patterns +├── stack-management/ +│ ├── SKILL.md # Stack operations +│ ├── reference.md # Advanced stack management +│ └── troubleshooting.md # Common issues +├── new-project/ +│ ├── SKILL.md # Project initialization +│ ├── reference.md # Advanced project patterns +│ └── templates/ # Project templates +│ ├── envrc.template +│ ├── pyproject.toml +│ ├── main_py.template +│ └── Pulumi.yaml +├── import/ +│ ├── SKILL.md # Import workflows +│ ├── reference.md # Advanced import patterns +│ ├── examples/ # Import examples +│ │ ├── aws-s3-bucket.md +│ │ ├── aws-iam-role.md +│ │ └── github-repository.md +│ └── scripts/ # Helper scripts +│ └── discover-aws-resources.py +└── documentation/ + ├── SKILL.md # Documentation access + ├── reference.md # Advanced topics + └── providers/ # Provider guides + ├── aws.md + ├── github.md + └── onepassword.md +``` + +## Best Practices + +### When Claude Uses These Skills + +Claude will automatically use these skills when: + +- User mentions Pulumi operations +- Infrastructure deployment is needed +- Stack management is requested +- Documentation is required + +### Skill Selection + +Claude chooses skills based on triggers: + +- **deploy**: Preview, deployment, pulumi up +- **stack-management**: Stacks, outputs, configuration +- **new-project**: Initialize, scaffold, new project +- **import**: Import, click-ops, existing infrastructure, bring under management +- **documentation**: Docs, how-to, provider questions + +### Progressive Disclosure in Action + +1. **Quick Operations**: Use SKILL.md for common tasks +2. **Advanced Needs**: Reference reference.md for complex scenarios +3. **Troubleshooting**: Check troubleshooting.md for issues +4. **Provider-Specific**: Load appropriate provider guide + +## Customization + +### Adding Providers + +To add a new provider guide: + +1. Create `documentation/providers/{provider}.md` +2. Include: + - Authentication + - Common resources + - Code examples + - Best practices + - Documentation links +3. Reference in `documentation/SKILL.md` + +### Updating Templates + +Templates in `new-project/templates/` can be customized: + +- Modify for your organization +- Add additional providers +- Update credential patterns +- Include custom tooling + +### Extending Skills + +Add new skills by: + +1. Creating new directory in `pulumi-workflow/` +2. Adding SKILL.md with proper frontmatter +3. Including reference files as needed +4. Following progressive disclosure patterns + +## Validation + +Skills validated for: + +- ✓ Valid YAML frontmatter +- ✓ Required fields (name, description) +- ✓ Character limits (name ≤ 64, description ≤ 1024) +- ✓ Third-person descriptions +- ✓ Forward-slash paths +- ✓ Progressive disclosure structure + +## Integration with nf-core/ops + +These skills integrate with nf-core/ops patterns: + +### S3 Backend + +```yaml +backend: + url: s3://nf-core-ops-pulumi-state?region=us-east-1&awssdk=v2 +``` + +### Credential Management + +```bash +# .envrc pattern +from_op AWS_ACCESS_KEY_ID="op://Dev/Pulumi-AWS-key/access key id" +from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password" +``` + +### Project Structure + +``` +~/src/nf-core/ops/pulumi/ +├── co2_reports/ +├── megatests_infra/ +└── {your_project}/ +``` + +## Troubleshooting + +### Skills Not Loading + +If skills aren't being used: + +1. Verify files are in `~/.claude/skills/pulumi-workflow/` +2. Check SKILL.md frontmatter is valid +3. Ensure descriptions include relevant triggers + +### Credential Issues + +See `stack-management/troubleshooting.md`: + +- AWS credential errors +- Pulumi passphrase issues +- 1Password connectivity + +### Deployment Failures + +See `deploy/reference.md` for: + +- Error handling patterns +- Rollback strategies +- State management + +## Contributing + +When updating these skills: + +1. **Maintain Structure**: Keep progressive disclosure pattern +2. **Update All Files**: Modify SKILL.md and reference.md as needed +3. **Test Thoroughly**: Verify with real Pulumi operations +4. **Document Changes**: Update this README + +## Version + +**Version**: 1.0.0 +**Created**: 2025-01-02 +**Last Updated**: 2025-01-02 + +## Related Skills + +These skills complement: + +- **jj-workflow**: Version control with Jujutsu +- **nextflow-pipeline-development**: Nextflow infrastructure +- Custom nf-core/ops skills + +## Resources + +- **Pulumi Docs**: https://www.pulumi.com/docs/ +- **nf-core/ops**: GitHub repository +- **1Password**: https://developer.1password.com/ +- **direnv**: https://direnv.net/ diff --git a/.claude/skills/pulumi/deploy/SKILL.md b/.claude/skills/pulumi/deploy/SKILL.md new file mode 100644 index 00000000..c7201548 --- /dev/null +++ b/.claude/skills/pulumi/deploy/SKILL.md @@ -0,0 +1,142 @@ +--- +name: Deploying Pulumi Infrastructure +description: Preview and deploy infrastructure changes using Pulumi. Use when deploying infrastructure, running pulumi up/preview, or applying infrastructure changes. Includes credential management with 1Password and safety confirmations for destructive operations. +--- + +# Deploying Pulumi Infrastructure + +Deploy and manage infrastructure changes safely using Pulumi's preview and deployment workflow. + +## When to Use + +Use this skill when: + +- Deploying infrastructure changes +- Previewing changes before applying +- Running `pulumi up` or `pulumi preview` +- Applying infrastructure as code +- Managing infrastructure updates + +## Standard Deployment Workflow + +Follow this safe deployment pattern: + +### 1. Load Credentials + +Ensure AWS and Pulumi credentials are available: + +```bash +# If using direnv + 1Password (recommended): +# Credentials auto-load from .envrc when entering directory +cd path/to/pulumi/project + +# Verify credentials loaded: +echo $AWS_ACCESS_KEY_ID # Should show key +echo $PULUMI_CONFIG_PASSPHRASE # Should show value +``` + +**If credentials not loaded**: Check `.envrc` exists and direnv is allowed (`direnv allow`). + +### 2. Preview Changes + +**Always preview before deploying:** + +```bash +uv run pulumi preview +``` + +Review the output carefully: + +- **Green `+`**: Resources to be created +- **Yellow `~`**: Resources to be modified +- **Red `-`**: Resources to be deleted + +**⚠️ If resources will be deleted or modified in unexpected ways, STOP and investigate before proceeding.** + +### 3. Confirm with User + +Before running `pulumi up`, **always confirm with the user**: + +- Summarize the changes from the preview +- Highlight any destructive operations (deletes, replacements) +- Ask: "Should I proceed with deployment?" +- Wait for explicit confirmation + +### 4. Deploy Changes + +Only after user confirmation: + +```bash +uv run pulumi up --yes +``` + +The `--yes` flag auto-approves the deployment (safe since user already confirmed based on preview). + +### 5. Verify Deployment + +After successful deployment: + +```bash +# View stack outputs +uv run pulumi stack output + +# Check resource state +uv run pulumi stack --show-urns +``` + +## Quick Reference Commands + +```bash +# Preview changes (dry-run) +uv run pulumi preview + +# Deploy with automatic approval +uv run pulumi up --yes + +# Deploy and save detailed output +uv run pulumi up --yes 2>&1 | tee deployment.log + +# View current stack state +uv run pulumi stack + +# Export stack configuration +uv run pulumi stack export > stack-backup.json +``` + +## Safety Checklist + +Before deploying, verify: + +- [ ] Credentials are loaded (AWS, Pulumi passphrase) +- [ ] Working in correct Pulumi project directory +- [ ] Correct stack selected (`pulumi stack select `) +- [ ] Preview shows expected changes +- [ ] No unexpected deletions or replacements +- [ ] User has confirmed deployment +- [ ] Backup of current state if making major changes + +## Error Handling + +If deployment fails: + +1. **Read the error message carefully** - Pulumi provides detailed errors +2. **Check credentials** - Most failures are authentication issues +3. **Verify permissions** - IAM/RBAC issues are common +4. **Review stack state** - `pulumi stack` shows current state +5. **Consult troubleshooting guide** - See [../stack-management/troubleshooting.md](../stack-management/troubleshooting.md) + +## Advanced Patterns + +For more complex deployment scenarios, see [reference.md](reference.md): + +- Targeted deployments (specific resources) +- Refresh operations +- Import existing infrastructure +- Parallel deployments +- CI/CD integration patterns + +## Related Skills + +- **Stack Management**: Switch stacks, view outputs, manage configuration +- **New Project**: Initialize new Pulumi projects with proper structure +- **Documentation**: Access Pulumi provider documentation diff --git a/.claude/skills/pulumi/deploy/reference.md b/.claude/skills/pulumi/deploy/reference.md new file mode 100644 index 00000000..b09c7c3e --- /dev/null +++ b/.claude/skills/pulumi/deploy/reference.md @@ -0,0 +1,371 @@ +# Pulumi Deployment Reference + +Comprehensive guide to advanced Pulumi deployment patterns and techniques. + +## Table of Contents + +- [Targeted Deployments](#targeted-deployments) +- [Refresh Operations](#refresh-operations) +- [Import Existing Infrastructure](#import-existing-infrastructure) +- [State Management](#state-management) +- [Parallel Deployments](#parallel-deployments) +- [CI/CD Integration](#cicd-integration) +- [Rollback Strategies](#rollback-strategies) +- [Performance Optimization](#performance-optimization) + +## Targeted Deployments + +Deploy specific resources without affecting the entire stack: + +### Deploy Specific Resources + +```bash +# Target a single resource +uv run pulumi up --target urn:pulumi:stack::project::aws:s3/bucket:Bucket::my-bucket + +# Target multiple resources +uv run pulumi up --target bucket1 --target bucket2 + +# Preview targeted deployment +uv run pulumi preview --target resource-name +``` + +### Replace Specific Resources + +Force recreation of a resource: + +```bash +# Replace a specific resource +uv run pulumi up --replace urn:pulumi:stack::project::aws:ec2/instance:Instance::my-instance + +# Preview replacement +uv run pulumi preview --replace resource-name +``` + +**Use cases:** + +- Fix a misconfigured resource without full redeployment +- Test changes to specific components +- Minimize blast radius of changes + +## Refresh Operations + +Synchronize Pulumi state with actual infrastructure: + +### Basic Refresh + +```bash +# Detect and sync drift +uv run pulumi refresh + +# Preview what refresh would do +uv run pulumi preview --refresh +``` + +### Refresh Without Update + +```bash +# Only refresh state, don't apply changes +uv run pulumi refresh --yes + +# Refresh and show differences +uv run pulumi refresh --diff +``` + +**When to refresh:** + +- After manual infrastructure changes +- Before major deployments +- When state seems out of sync +- After infrastructure imports + +## Import Existing Infrastructure + +Bring existing resources under Pulumi management: + +### Import Single Resource + +```bash +# Import an existing S3 bucket +uv run pulumi import aws:s3/bucket:Bucket my-bucket existing-bucket-name + +# Import with specific stack +uv run pulumi import --stack production aws:ec2/instance:Instance my-instance i-0123456789abcdef0 +``` + +### Bulk Import + +For multiple resources, create an import file: + +```json +{ + "resources": [ + { + "type": "aws:s3/bucket:Bucket", + "name": "bucket1", + "id": "my-bucket-1" + }, + { + "type": "aws:s3/bucket:Bucket", + "name": "bucket2", + "id": "my-bucket-2" + } + ] +} +``` + +```bash +# Import from file +uv run pulumi import --file imports.json +``` + +### Import Workflow + +1. **Identify existing resources** to import +2. **Create Pulumi code** matching the existing resources +3. **Run import** command +4. **Verify state** with `pulumi stack --show-urns` +5. **Run preview** to ensure no changes +6. **Adjust code** if preview shows unexpected changes + +## State Management + +### Export State + +```bash +# Export current stack state +uv run pulumi stack export > state-backup.json + +# Export with secrets decrypted +uv run pulumi stack export --show-secrets > state-with-secrets.json +``` + +### Import State + +```bash +# Restore from backup +uv run pulumi stack import --file state-backup.json +``` + +### Cancel Current Update + +If deployment is stuck: + +```bash +# Cancel in-progress update +uv run pulumi cancel +``` + +**⚠️ Warning**: Only use when absolutely necessary. Can leave stack in inconsistent state. + +## Parallel Deployments + +Optimize deployment speed: + +### Configure Parallelism + +```bash +# Increase parallel resource operations (default: 10) +uv run pulumi up --parallel 20 + +# Disable parallelism (sequential deployment) +uv run pulumi up --parallel 1 +``` + +### Trade-offs + +**Higher parallelism:** + +- ✓ Faster deployments +- ✗ Higher API rate limits risk +- ✗ Harder to debug failures + +**Lower parallelism:** + +- ✓ More predictable +- ✓ Easier to debug +- ✗ Slower deployments + +## CI/CD Integration + +### GitHub Actions Example + +```yaml +name: Deploy Infrastructure +on: + push: + branches: [main] + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install uv + run: pip install uv + + - name: Deploy with Pulumi + run: uv run pulumi up --yes --stack production + env: + PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }} +``` + +### Preview on Pull Request + +```yaml +name: Preview Infrastructure Changes +on: pull_request + +jobs: + preview: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + + - name: Preview with Pulumi + run: | + uv run pulumi preview --stack development > preview.txt + cat preview.txt + env: + PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }} + + - name: Comment Preview on PR + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const preview = fs.readFileSync('preview.txt', 'utf8'); + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: `## Pulumi Preview\n\`\`\`\n${preview}\n\`\`\`` + }); +``` + +## Rollback Strategies + +### Using Stack History + +```bash +# List deployment history +uv run pulumi stack history + +# Export previous state +uv run pulumi stack export --version > previous-state.json + +# Restore previous state +uv run pulumi stack import --file previous-state.json +``` + +### Git-Based Rollback + +```bash +# Revert to previous commit +git revert HEAD +git push + +# Deploy previous version +uv run pulumi up --yes +``` + +### Emergency Rollback + +For critical issues: + +1. **Export current state** for investigation +2. **Revert code** to last known good version +3. **Deploy immediately**: `uv run pulumi up --yes` +4. **Verify services** are restored +5. **Investigate** what went wrong + +## Performance Optimization + +### Reduce State File Size + +```bash +# Remove deleted resources from state +uv run pulumi state delete urn:pulumi:stack::project::Type::name --yes +``` + +### Optimize Dependencies + +Explicit dependencies improve parallelism: + +```python +# Bad: Implicit dependencies +bucket = s3.Bucket("bucket") +object = s3.BucketObject("object", bucket=bucket.id) + +# Good: Explicit dependencies enable better parallelism +bucket = s3.Bucket("bucket") +object = s3.BucketObject("object", + bucket=bucket.id, + opts=ResourceOptions(depends_on=[bucket]) +) +``` + +### Stack Splitting + +For large infrastructures, split into multiple stacks: + +``` +infrastructure/ +├── networking/ # VPC, subnets, etc. +├── security/ # IAM, security groups +├── compute/ # EC2, ECS, Lambda +└── data/ # RDS, S3, DynamoDB +``` + +Benefits: + +- Faster deployments (smaller state) +- Better separation of concerns +- Reduced blast radius +- Easier to manage permissions + +### Use Stack References + +Share outputs between stacks: + +```python +# In networking stack +pulumi.export("vpc_id", vpc.id) + +# In compute stack +networking = pulumi.StackReference("organization/networking/production") +vpc_id = networking.get_output("vpc_id") +``` + +## Best Practices Summary + +1. **Always preview** before deploying +2. **Confirm with user** for production deployments +3. **Export state** before major changes +4. **Use targeted updates** when possible +5. **Refresh regularly** to detect drift +6. **Monitor deployments** in CI/CD +7. **Split large stacks** for performance +8. **Document changes** in git commits +9. **Test in development** first +10. **Have rollback plan** ready diff --git a/.claude/skills/pulumi/documentation/SKILL.md b/.claude/skills/pulumi/documentation/SKILL.md new file mode 100644 index 00000000..e6e61bdb --- /dev/null +++ b/.claude/skills/pulumi/documentation/SKILL.md @@ -0,0 +1,264 @@ +--- +name: Accessing Pulumi Documentation +description: Access Pulumi documentation, provider guides, and troubleshooting resources. Use when looking up Pulumi syntax, provider-specific documentation, API references, or best practices for AWS, GitHub, 1Password, or other providers. +--- + +# Accessing Pulumi Documentation + +Quick access to Pulumi documentation and provider-specific guides. + +## When to Use + +Use this skill when: + +- Looking up Pulumi syntax or API +- Finding provider-specific documentation +- Checking resource properties +- Learning best practices +- Troubleshooting provider issues +- Finding code examples + +## Quick Documentation Access + +### Official Pulumi Docs + +For latest documentation, use WebSearch: + +``` +Search: "Pulumi {topic} documentation 2025" +``` + +Examples: + +- "Pulumi AWS S3 bucket documentation 2025" +- "Pulumi stack references documentation 2025" +- "Pulumi component resources guide 2025" + +### Provider-Specific Guides + +This skill includes cached guides for common providers: + +- **AWS**: [providers/aws.md](providers/aws.md) +- **GitHub**: [providers/github.md](providers/github.md) +- **1Password**: [providers/onepassword.md](providers/onepassword.md) + +Load the appropriate guide based on the user's question. + +## Common Documentation Patterns + +### Finding Resource Documentation + +**AWS Resources:** + +``` +Search: "Pulumi AWS {resource} documentation" +# Examples: +- "Pulumi AWS EC2 Instance documentation" +- "Pulumi AWS RDS database documentation" +- "Pulumi AWS IAM policy documentation" +``` + +**GitHub Resources:** + +``` +Search: "Pulumi GitHub {resource} documentation" +# Examples: +- "Pulumi GitHub repository documentation" +- "Pulumi GitHub Actions secret documentation" +``` + +### Finding Examples + +``` +Search: "Pulumi {use case} example" +# Examples: +- "Pulumi Lambda function with API Gateway example" +- "Pulumi VPC with public and private subnets example" +- "Pulumi cross-stack reference example" +``` + +### Finding Best Practices + +``` +Search: "Pulumi {topic} best practices" +# Examples: +- "Pulumi project structure best practices" +- "Pulumi secret management best practices" +- "Pulumi testing best practices" +``` + +## Workflow for Documentation Questions + +1. **Identify the topic**: What is the user asking about? +2. **Check if provider-specific**: Is it AWS, GitHub, 1Password specific? +3. **Load relevant guide**: If available in providers/ directory +4. **Search if needed**: Use WebSearch for latest docs +5. **Provide clear answer**: With code examples when relevant + +## Common Questions + +### How do I...? + +**Create a resource:** + +- Check providers/{provider}.md for examples +- Search: "Pulumi {provider} {resource} example" + +**Configure authentication:** + +- See [providers/aws.md](providers/aws.md) for AWS +- See [providers/github.md](providers/github.md) for GitHub + +**Export outputs:** + +```python +pulumi.export("output_name", resource.property) +``` + +**Reference another stack:** + +```python +other_stack = pulumi.StackReference("org/project/stack") +value = other_stack.get_output("output_name") +``` + +**Use secrets:** + +```python +config = pulumi.Config() +secret_value = config.require_secret("secret_key") +``` + +### What properties are available? + +**Check documentation:** + +``` +Search: "Pulumi {provider} {resource} properties" +``` + +**Common pattern:** +Most resources have: + +- `id`: Resource identifier +- `arn`: AWS Resource Name (AWS resources) +- `name`: Resource name +- `tags`: Resource tags + +### How do I troubleshoot...? + +See: + +- [../stack-management/troubleshooting.md](../stack-management/troubleshooting.md) for common issues +- Provider-specific guides for provider issues + +## Provider Registry + +Access provider registries: + +- **AWS**: https://www.pulumi.com/registry/packages/aws/ +- **GitHub**: https://www.pulumi.com/registry/packages/github/ +- **1Password**: https://www.pulumi.com/registry/packages/onepassword/ +- **All providers**: https://www.pulumi.com/registry/ + +## Code Examples + +### Basic Resource Creation + +```python +import pulumi +import pulumi_aws as aws + +# Create resource with explicit properties +bucket = aws.s3.Bucket( + "my-bucket", + bucket="my-unique-bucket-name", + acl="private", + tags={ + "Environment": pulumi.get_stack(), + }, +) + +# Export output +pulumi.export("bucket_name", bucket.id) +``` + +### Using Configuration + +```python +import pulumi + +config = pulumi.Config() + +# Get required value +region = config.require("region") + +# Get optional value with default +instance_type = config.get("instance_type") or "t3.micro" + +# Get boolean +enable_monitoring = config.get_bool("enable_monitoring") or False + +# Get secret +database_password = config.require_secret("database_password") +``` + +### Resource Options + +```python +from pulumi import ResourceOptions + +# Explicit dependency +resource_b = aws.Resource("b", + opts=ResourceOptions(depends_on=[resource_a]) +) + +# Use different provider +resource_eu = aws.Resource("eu", + opts=ResourceOptions(provider=provider_eu) +) + +# Protect from deletion +critical_resource = aws.Resource("critical", + opts=ResourceOptions(protect=True) +) + +# Custom timeout +long_running = aws.Resource("long", + opts=ResourceOptions( + custom_timeouts=CustomTimeouts( + create="30m", + update="20m", + delete="10m", + ) + ) +) +``` + +## Quick Reference + +| Task | Resource | +| -------------- | ---------------------------------------------------- | +| AWS docs | [providers/aws.md](providers/aws.md) | +| GitHub docs | [providers/github.md](providers/github.md) | +| 1Password docs | [providers/onepassword.md](providers/onepassword.md) | +| Latest docs | WebSearch: "Pulumi {topic} 2025" | +| Examples | WebSearch: "Pulumi {use case} example" | +| API reference | https://www.pulumi.com/docs/reference/ | +| Registry | https://www.pulumi.com/registry/ | + +## Advanced Topics + +For complex scenarios, see [reference.md](reference.md): + +- Dynamic providers +- Automation API +- Policy as code +- Testing frameworks +- Provider customization + +## Related Skills + +- **Deploy**: Preview and deploy infrastructure changes +- **Stack Management**: Manage stacks and configuration +- **New Project**: Initialize new Pulumi projects diff --git a/.claude/skills/pulumi/documentation/providers/aws.md b/.claude/skills/pulumi/documentation/providers/aws.md new file mode 100644 index 00000000..09047a73 --- /dev/null +++ b/.claude/skills/pulumi/documentation/providers/aws.md @@ -0,0 +1,530 @@ +# Pulumi AWS Provider Guide + +Quick reference for common AWS resources and patterns in Pulumi. + +## Table of Contents + +- [Authentication](#authentication) +- [Common Resources](#common-resources) +- [IAM and Security](#iam-and-security) +- [Networking](#networking) +- [Storage](#storage) +- [Compute](#compute) +- [Best Practices](#best-practices) + +## Authentication + +### Using Environment Variables + +```bash +# Set via .envrc (recommended with 1Password) +export AWS_ACCESS_KEY_ID="your-key" +export AWS_SECRET_ACCESS_KEY="your-secret" +export AWS_REGION="us-east-1" +``` + +### Configure Provider + +```python +import pulumi_aws as aws + +# Use default credentials +# (from environment or AWS CLI config) + +# Or explicit configuration +provider = aws.Provider("custom", + region="us-east-1", + access_key="key", + secret_key="secret", +) +``` + +### Multiple Regions + +```python +# Primary region +bucket_east = aws.s3.Bucket("east") + +# Secondary region provider +provider_west = aws.Provider("west", region="us-west-2") +bucket_west = aws.s3.Bucket("west", + opts=pulumi.ResourceOptions(provider=provider_west) +) +``` + +## Common Resources + +### S3 Bucket + +```python +import pulumi_aws as aws + +bucket = aws.s3.Bucket( + "my-bucket", + bucket="unique-bucket-name", + acl="private", + versioning=aws.s3.BucketVersioningArgs( + enabled=True, + ), + server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs( + rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs( + apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs( + sse_algorithm="AES256", + ), + ), + ), + tags={ + "Environment": pulumi.get_stack(), + }, +) + +# Block public access +aws.s3.BucketPublicAccessBlock( + "block-public", + bucket=bucket.id, + block_public_acls=True, + block_public_policy=True, + ignore_public_acls=True, + restrict_public_buckets=True, +) + +pulumi.export("bucket_name", bucket.id) +pulumi.export("bucket_arn", bucket.arn) +``` + +### Lambda Function + +```python +import pulumi_aws as aws + +# IAM role for Lambda +role = aws.iam.Role( + "lambda-role", + assume_role_policy="""{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Service": "lambda.amazonaws.com"}, + "Action": "sts:AssumeRole" + }] + }""", +) + +# Attach basic execution policy +aws.iam.RolePolicyAttachment( + "lambda-execution", + role=role.name, + policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", +) + +# Lambda function +function = aws.lambda_.Function( + "my-function", + runtime="python3.11", + handler="index.handler", + role=role.arn, + code=pulumi.AssetArchive({ + ".": pulumi.FileArchive("./lambda"), + }), + environment=aws.lambda_.FunctionEnvironmentArgs( + variables={ + "KEY": "value", + }, + ), +) + +pulumi.export("function_name", function.name) +pulumi.export("function_arn", function.arn) +``` + +### EC2 Instance + +```python +import pulumi_aws as aws + +# Get latest Amazon Linux AMI +ami = aws.ec2.get_ami( + most_recent=True, + owners=["amazon"], + filters=[ + aws.ec2.GetAmiFilterArgs( + name="name", + values=["amzn2-ami-hvm-*-x86_64-gp2"], + ), + ], +) + +# Security group +sg = aws.ec2.SecurityGroup( + "web-sg", + description="Allow HTTP", + ingress=[ + aws.ec2.SecurityGroupIngressArgs( + protocol="tcp", + from_port=80, + to_port=80, + cidr_blocks=["0.0.0.0/0"], + ), + ], +) + +# Instance +instance = aws.ec2.Instance( + "web-server", + instance_type="t3.micro", + ami=ami.id, + vpc_security_group_ids=[sg.id], + tags={ + "Name": "WebServer", + }, +) + +pulumi.export("instance_id", instance.id) +pulumi.export("public_ip", instance.public_ip) +``` + +## IAM and Security + +### IAM User with Policy + +```python +import pulumi_aws as aws +import json + +# IAM user +user = aws.iam.User( + "ci-user", + name="my-ci-user", + tags={ + "Purpose": "CI/CD", + }, +) + +# Access key +access_key = aws.iam.AccessKey( + "ci-key", + user=user.name, +) + +# IAM policy +policy_document = { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + ], + "Resource": "arn:aws:s3:::my-bucket/*", + }], +} + +policy = aws.iam.Policy( + "ci-policy", + policy=json.dumps(policy_document), + description="CI/CD bucket access", +) + +# Attach policy to user +aws.iam.UserPolicyAttachment( + "ci-attach", + user=user.name, + policy_arn=policy.arn, +) + +pulumi.export("user_name", user.name) +pulumi.export("access_key_id", access_key.id) +pulumi.export("access_key_secret", access_key.secret) +``` + +### Security Group with Rules + +```python +import pulumi_aws as aws + +sg = aws.ec2.SecurityGroup( + "app-sg", + description="Application security group", + vpc_id=vpc.id, +) + +# SSH access +aws.ec2.SecurityGroupRule( + "ssh", + type="ingress", + security_group_id=sg.id, + protocol="tcp", + from_port=22, + to_port=22, + cidr_blocks=["10.0.0.0/8"], +) + +# HTTP access +aws.ec2.SecurityGroupRule( + "http", + type="ingress", + security_group_id=sg.id, + protocol="tcp", + from_port=80, + to_port=80, + cidr_blocks=["0.0.0.0/0"], +) + +# Egress (all traffic) +aws.ec2.SecurityGroupRule( + "egress", + type="egress", + security_group_id=sg.id, + protocol="-1", + from_port=0, + to_port=0, + cidr_blocks=["0.0.0.0/0"], +) +``` + +## Networking + +### VPC with Subnets + +```python +import pulumi_aws as aws + +# VPC +vpc = aws.ec2.Vpc( + "main", + cidr_block="10.0.0.0/16", + enable_dns_hostnames=True, + enable_dns_support=True, + tags={"Name": "main-vpc"}, +) + +# Internet Gateway +igw = aws.ec2.InternetGateway( + "igw", + vpc_id=vpc.id, + tags={"Name": "main-igw"}, +) + +# Public subnet +public_subnet = aws.ec2.Subnet( + "public", + vpc_id=vpc.id, + cidr_block="10.0.1.0/24", + availability_zone="us-east-1a", + map_public_ip_on_launch=True, + tags={"Name": "public-subnet"}, +) + +# Route table for public subnet +public_rt = aws.ec2.RouteTable( + "public-rt", + vpc_id=vpc.id, + routes=[ + aws.ec2.RouteTableRouteArgs( + cidr_block="0.0.0.0/0", + gateway_id=igw.id, + ), + ], + tags={"Name": "public-rt"}, +) + +# Associate route table with subnet +aws.ec2.RouteTableAssociation( + "public-rta", + subnet_id=public_subnet.id, + route_table_id=public_rt.id, +) + +pulumi.export("vpc_id", vpc.id) +pulumi.export("public_subnet_id", public_subnet.id) +``` + +## Storage + +### RDS Database + +```python +import pulumi_aws as aws + +# DB subnet group +db_subnet_group = aws.rds.SubnetGroup( + "db-subnets", + subnet_ids=[subnet1.id, subnet2.id], + tags={"Name": "DB subnets"}, +) + +# DB instance +db = aws.rds.Instance( + "postgres", + engine="postgres", + engine_version="14.7", + instance_class="db.t3.micro", + allocated_storage=20, + db_name="mydb", + username="admin", + password=config.require_secret("db_password"), + db_subnet_group_name=db_subnet_group.name, + vpc_security_group_ids=[db_sg.id], + skip_final_snapshot=True, + tags={"Name": "postgres-db"}, +) + +pulumi.export("db_endpoint", db.endpoint) +pulumi.export("db_name", db.db_name) +``` + +## Compute + +### ECS Service + +```python +import pulumi_aws as aws + +# ECS Cluster +cluster = aws.ecs.Cluster( + "app-cluster", + name="app-cluster", +) + +# Task definition +task_definition = aws.ecs.TaskDefinition( + "app-task", + family="app", + network_mode="awsvpc", + requires_compatibilities=["FARGATE"], + cpu="256", + memory="512", + container_definitions=pulumi.Output.json_dumps([{ + "name": "app", + "image": "nginx:latest", + "portMappings": [{ + "containerPort": 80, + "protocol": "tcp", + }], + }]), +) + +# ECS Service +service = aws.ecs.Service( + "app-service", + cluster=cluster.arn, + task_definition=task_definition.arn, + desired_count=2, + launch_type="FARGATE", + network_configuration=aws.ecs.ServiceNetworkConfigurationArgs( + subnets=[subnet.id], + security_groups=[sg.id], + assign_public_ip=True, + ), +) + +pulumi.export("cluster_name", cluster.name) +``` + +## Best Practices + +### Tagging Strategy + +```python +import pulumi + +# Define common tags +common_tags = { + "Project": pulumi.get_project(), + "Environment": pulumi.get_stack(), + "ManagedBy": "Pulumi", +} + +# Apply to resources +resource = aws.Resource( + "name", + tags={**common_tags, "Component": "specific"}, +) +``` + +### Resource Naming + +```python +import pulumi + +stack = pulumi.get_stack() +project = pulumi.get_project() + +# Include stack in names +bucket = aws.s3.Bucket( + "data-bucket", + bucket=f"{project}-data-{stack}", +) + +# Use logical names for resources +instance = aws.ec2.Instance( + "web-server", # Logical name + tags={"Name": f"{project}-web-{stack}"}, # Display name +) +``` + +### Security Defaults + +```python +# Always block public access for S3 +aws.s3.BucketPublicAccessBlock( + "block-public", + bucket=bucket.id, + block_public_acls=True, + block_public_policy=True, + ignore_public_acls=True, + restrict_public_buckets=True, +) + +# Enable encryption by default +server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs( + rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs( + apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs( + sse_algorithm="AES256", + ), + ), +) + +# Use private ACLs +acl="private" +``` + +## Common Patterns + +### Using Data Sources + +```python +# Get current AWS account +current = aws.get_caller_identity() +account_id = current.account_id + +# Get current region +region = aws.get_region() + +# Get existing VPC +vpc = aws.ec2.get_vpc(default=True) + +# Use in resources +bucket = aws.s3.Bucket( + "account-bucket", + bucket=f"my-bucket-{account_id}", +) +``` + +### Conditional Resources + +```python +import pulumi + +stack = pulumi.get_stack() + +# Only create in production +if stack == "production": + monitoring = aws.cloudwatch.MetricAlarm(...) +``` + +## Documentation Links + +- **AWS Provider**: https://www.pulumi.com/registry/packages/aws/ +- **API Reference**: https://www.pulumi.com/registry/packages/aws/api-docs/ +- **Examples**: https://github.com/pulumi/examples/tree/master/aws-py-* diff --git a/.claude/skills/pulumi/documentation/providers/github.md b/.claude/skills/pulumi/documentation/providers/github.md new file mode 100644 index 00000000..741d5fd8 --- /dev/null +++ b/.claude/skills/pulumi/documentation/providers/github.md @@ -0,0 +1,470 @@ +# Pulumi GitHub Provider Guide + +Quick reference for managing GitHub resources with Pulumi. + +## Table of Contents + +- [Authentication](#authentication) +- [Common Resources](#common-resources) +- [Repository Management](#repository-management) +- [Secrets and Variables](#secrets-and-variables) +- [Teams and Permissions](#teams-and-permissions) +- [Best Practices](#best-practices) + +## Authentication + +### Using Personal Access Token + +```bash +# Set via environment variable +export GITHUB_TOKEN="ghp_xxxxxxxxxxxxx" + +# Or via .envrc with 1Password +from_op GITHUB_TOKEN="op://Dev/GitHub-Token/credential" +``` + +### Configure Provider + +```python +import pulumi_github as github + +# Provider will use GITHUB_TOKEN from environment +# Or configure explicitly: +provider = github.Provider("github", + token="ghp_xxxxxxxxxxxxx", + owner="nf-core", +) +``` + +### Required Token Permissions + +For managing repositories and secrets, token needs: + +- `repo` - Full repository access +- `admin:org` - Organization management +- `workflow` - GitHub Actions workflow management + +## Common Resources + +### Create Repository + +```python +import pulumi_github as github + +repo = github.Repository( + "my-repo", + name="my-awesome-repo", + description="Repository managed by Pulumi", + visibility="public", # or "private" + has_issues=True, + has_projects=True, + has_wiki=True, + auto_init=True, + gitignore_template="Python", + license_template="apache-2.0", +) + +pulumi.export("repo_url", repo.html_url) +pulumi.export("clone_url", repo.ssh_clone_url) +``` + +### Repository with Protection Rules + +```python +import pulumi_github as github + +repo = github.Repository( + "protected-repo", + name="production-app", + visibility="private", +) + +# Branch protection for main +github.BranchProtection( + "main-protection", + repository_id=repo.node_id, + pattern="main", + required_pull_request_reviews=github.BranchProtectionRequiredPullRequestReviewsArgs( + dismiss_stale_reviews=True, + require_code_owner_reviews=True, + required_approving_review_count=2, + ), + enforce_admins=True, + require_signed_commits=True, + required_status_checks=github.BranchProtectionRequiredStatusChecksArgs( + strict=True, + contexts=["ci/test", "ci/lint"], + ), +) +``` + +## Repository Management + +### Repository Settings + +```python +import pulumi_github as github + +repo = github.Repository( + "configured-repo", + name="my-repo", + description="Fully configured repository", + + # Visibility + visibility="public", + + # Features + has_issues=True, + has_projects=False, + has_wiki=False, + has_downloads=True, + has_discussions=True, + + # Settings + allow_merge_commit=True, + allow_squash_merge=True, + allow_rebase_merge=False, + delete_branch_on_merge=True, + + # Security + vulnerability_alerts=True, + + # Templates + gitignore_template="Python", + license_template="apache-2.0", + + # Topics + topics=["python", "automation", "pulumi"], + + # Homepage + homepage_url="https://example.com", +) +``` + +### Repository Collaborators + +```python +import pulumi_github as github + +# Add collaborator with specific permission +github.RepositoryCollaborator( + "contributor", + repository=repo.name, + username="contributor-username", + permission="push", # pull, push, maintain, triage, admin +) + +# Add team with access +github.TeamRepository( + "team-access", + team_id=team.id, + repository=repo.name, + permission="push", +) +``` + +### Repository Files + +```python +import pulumi_github as github + +# Create file in repository +github.RepositoryFile( + "readme", + repository=repo.name, + file=".github/PULL_REQUEST_TEMPLATE.md", + content="""## Description +Please include a summary of the changes. + +## Testing +How has this been tested? +""", + branch="main", + commit_message="Add PR template", + commit_author="Pulumi", + commit_email="noreply@pulumi.com", +) +``` + +## Secrets and Variables + +### Actions Secrets + +```python +import pulumi_github as github +import pulumi + +# Repository secret +github.ActionsSecret( + "aws-key", + repository=repo.name, + secret_name="AWS_ACCESS_KEY_ID", + plaintext_value=pulumi.Output.secret("AKIA..."), +) + +# Organization secret (available to multiple repos) +github.ActionsOrganizationSecret( + "org-secret", + secret_name="ORG_WIDE_TOKEN", + visibility="selected", # all, private, selected + selected_repository_ids=[repo1.repo_id, repo2.repo_id], + plaintext_value=pulumi.Output.secret("secret-value"), +) + +# Environment secret +github.ActionsEnvironmentSecret( + "prod-secret", + repository=repo.name, + environment="production", + secret_name="DATABASE_URL", + plaintext_value=pulumi.Output.secret("postgres://..."), +) +``` + +### Actions Variables + +```python +import pulumi_github as github + +# Repository variable (non-sensitive) +github.ActionsVariable( + "region", + repository=repo.name, + variable_name="AWS_REGION", + value="us-east-1", +) + +# Organization variable +github.ActionsOrganizationVariable( + "org-var", + variable_name="DEPLOY_ENV", + value="production", + visibility="selected", + selected_repository_ids=[repo.repo_id], +) +``` + +### Example: Complete Secret Setup + +```python +import pulumi_github as github +import pulumi + +# AWS credentials for CI/CD +github.ActionsSecret( + "aws-access-key", + repository="modules", + secret_name="CO2_REPORTS_AWS_ACCESS_KEY_ID", + plaintext_value=access_key.id, +) + +github.ActionsSecret( + "aws-secret-key", + repository="modules", + secret_name="CO2_REPORTS_AWS_SECRET_ACCESS_KEY", + plaintext_value=pulumi.Output.secret(access_key.secret), +) + +github.ActionsVariable( + "aws-region", + repository="modules", + variable_name="CO2_REPORTS_AWS_REGION", + value="eu-north-1", +) +``` + +## Teams and Permissions + +### Create Team + +```python +import pulumi_github as github + +team = github.Team( + "platform-team", + name="Platform Team", + description="Infrastructure and DevOps team", + privacy="closed", # secret, closed +) + +# Add team members +github.TeamMembership( + "member1", + team_id=team.id, + username="user1", + role="maintainer", # member, maintainer +) +``` + +### Team Repository Access + +```python +import pulumi_github as github + +# Grant team access to repository +github.TeamRepository( + "team-repo-access", + team_id=team.id, + repository=repo.name, + permission="push", # pull, triage, push, maintain, admin +) +``` + +## Best Practices + +### Repository Naming + +```python +# Use consistent naming +repo = github.Repository( + "nf-co2-reports", # Pulumi resource name + name="nf-co2-reports", # GitHub repo name + description="CO2 footprint reporting infrastructure", +) +``` + +### Secret Management + +```python +# Never hardcode secrets +# BAD: +plaintext_value="hardcoded-secret" + +# GOOD: Use Pulumi secrets +plaintext_value=pulumi.Output.secret(secret_value) + +# BETTER: Reference from 1Password via config +config = pulumi.Config() +secret = config.require_secret("github_token") +``` + +### Default Branch Protection + +Always protect default branch: + +```python +github.BranchProtection( + "main-protection", + repository_id=repo.node_id, + pattern=repo.default_branch, + required_pull_request_reviews=github.BranchProtectionRequiredPullRequestReviewsArgs( + required_approving_review_count=1, + ), +) +``` + +### Repository Templates + +Use repository templates for consistency: + +```python +# Template repository +template = github.Repository( + "repo-template", + name="repository-template", + is_template=True, + visibility="public", +) + +# Create repo from template +new_repo = github.Repository( + "from-template", + name="new-project", + template=github.RepositoryTemplateArgs( + owner="nf-core", + repository=template.name, + ), +) +``` + +## Common Patterns + +### Bulk Secret Creation + +```python +import pulumi_github as github + +secrets = { + "AWS_ACCESS_KEY_ID": access_key_id, + "AWS_SECRET_ACCESS_KEY": secret_access_key, + "AWS_REGION": region, +} + +for secret_name, secret_value in secrets.items(): + github.ActionsSecret( + f"secret-{secret_name.lower()}", + repository=repo.name, + secret_name=secret_name, + plaintext_value=pulumi.Output.secret(secret_value), + ) +``` + +### Environment-Based Configuration + +```python +import pulumi + +stack = pulumi.get_stack() + +repo = github.Repository( + f"app-{stack}", + name=f"application-{stack}", + visibility="private" if stack == "production" else "public", +) +``` + +### Webhook Configuration + +```python +import pulumi_github as github + +webhook = github.RepositoryWebhook( + "ci-webhook", + repository=repo.name, + configuration=github.RepositoryWebhookConfigurationArgs( + url="https://ci.example.com/webhook", + content_type="json", + insecure_ssl=False, + secret=pulumi.Output.secret("webhook-secret"), + ), + events=["push", "pull_request"], +) +``` + +## Troubleshooting + +### Token Permissions + +If getting permission errors: + +1. Verify token has required scopes +2. Check organization SSO requirements +3. Verify repository access + +### Secret Not Updating + +GitHub Actions secrets are write-only: + +```python +# Force update with replace +github.ActionsSecret( + "secret", + repository=repo.name, + secret_name="NAME", + plaintext_value=new_value, + opts=pulumi.ResourceOptions(replace_on_changes=["plaintext_value"]), +) +``` + +### Rate Limiting + +GitHub API has rate limits: + +- Use `--parallel 5` to reduce concurrent requests +- Cache data sources when possible + +## Documentation Links + +- **GitHub Provider**: https://www.pulumi.com/registry/packages/github/ +- **API Reference**: https://www.pulumi.com/registry/packages/github/api-docs/ +- **Examples**: https://github.com/pulumi/examples/tree/master/github-* diff --git a/.claude/skills/pulumi/documentation/providers/onepassword.md b/.claude/skills/pulumi/documentation/providers/onepassword.md new file mode 100644 index 00000000..3a857bd8 --- /dev/null +++ b/.claude/skills/pulumi/documentation/providers/onepassword.md @@ -0,0 +1,460 @@ +# Pulumi 1Password Provider Guide + +Quick reference for accessing secrets from 1Password in Pulumi programs. + +## Table of Contents + +- [Authentication](#authentication) +- [Reading Secrets](#reading-secrets) +- [Common Patterns](#common-patterns) +- [Best Practices](#best-practices) + +## Authentication + +### Service Account Token + +```bash +# Set via environment variable +export OP_SERVICE_ACCOUNT_TOKEN="ops_xxxxxxxxxxxxx" + +# Or via Pulumi config +uv run pulumi config set pulumi-onepassword:service_account_token --secret +``` + +### Using direnv (Recommended) + +```bash +# In .envrc +from_op OP_SERVICE_ACCOUNT_TOKEN="op://Employee/Service-Account/credential" +``` + +## Reading Secrets + +### Basic Secret Access + +```python +import pulumi_onepassword as onepassword + +# Read a secret from 1Password +secret = onepassword.get_item_secret_output( + vault="Dev", + item="AWS-Key", + field="password", +) + +# Use in Pulumi resources +resource = SomeResource( + "resource", + secret_value=secret, +) +``` + +### Reading Specific Fields + +```python +import pulumi_onepassword as onepassword + +# Read access key ID +access_key_id = onepassword.get_item_secret_output( + vault="Dev", + item="AWS-Key", + field="access key id", +) + +# Read secret access key +secret_access_key = onepassword.get_item_secret_output( + vault="Dev", + item="AWS-Key", + field="secret access key", +) + +# Use in AWS provider +import pulumi_aws as aws + +provider = aws.Provider( + "aws-from-1password", + access_key=access_key_id, + secret_key=secret_access_key, + region="us-east-1", +) +``` + +### Reading Multiple Secrets + +```python +import pulumi_onepassword as onepassword + +secrets = { + "github_token": onepassword.get_item_secret_output( + vault="Dev", + item="GitHub-Token", + field="credential", + ), + "database_password": onepassword.get_item_secret_output( + vault="Dev", + item="Database-Password", + field="password", + ), + "api_key": onepassword.get_item_secret_output( + vault="Dev", + item="API-Key", + field="credential", + ), +} +``` + +## Common Patterns + +### AWS Credentials from 1Password + +```python +import pulumi_onepassword as onepassword +import pulumi_aws as aws + +# Read AWS credentials +aws_access_key = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="access key id", +) + +aws_secret_key = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="secret access key", +) + +# Configure AWS provider +provider = aws.Provider( + "aws", + access_key=aws_access_key, + secret_key=aws_secret_key, + region="us-east-1", +) + +# Create resource with provider +bucket = aws.s3.Bucket( + "bucket", + opts=pulumi.ResourceOptions(provider=provider), +) +``` + +### GitHub Token from 1Password + +```python +import pulumi_onepassword as onepassword +import pulumi_github as github + +# Read GitHub token +github_token = onepassword.get_item_secret_output( + vault="Dev", + item="GitHub-Token", + field="credential", +) + +# Configure GitHub provider +provider = github.Provider( + "github", + token=github_token, + owner="nf-core", +) + +# Create repository +repo = github.Repository( + "repo", + name="my-repo", + opts=pulumi.ResourceOptions(provider=provider), +) +``` + +### Database Passwords + +```python +import pulumi_onepassword as onepassword +import pulumi_aws as aws + +# Read database password +db_password = onepassword.get_item_secret_output( + vault="Dev", + item="RDS-Master-Password", + field="password", +) + +# Create RDS instance +db = aws.rds.Instance( + "postgres", + engine="postgres", + instance_class="db.t3.micro", + allocated_storage=20, + username="admin", + password=db_password, +) +``` + +## Best Practices + +### Vault Organization + +Organize secrets by environment: + +``` +Vaults: +├── Dev # Development secrets +├── Staging # Staging environment secrets +├── Production # Production secrets +└── Employee # Personal/shared secrets +``` + +### Secret Naming + +Use descriptive names: + +```python +# Good +onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-Key", + field="access key id", +) + +# Bad +onepassword.get_item_secret_output( + vault="Dev", + item="key1", + field="value", +) +``` + +### Environment-Specific Secrets + +```python +import pulumi +import pulumi_onepassword as onepassword + +stack = pulumi.get_stack() + +# Map stack to vault +vault_map = { + "development": "Dev", + "staging": "Staging", + "production": "Production", +} + +vault = vault_map.get(stack, "Dev") + +# Read from appropriate vault +secret = onepassword.get_item_secret_output( + vault=vault, + item="API-Key", + field="credential", +) +``` + +### Caching Secrets + +For frequently accessed secrets: + +```python +import pulumi_onepassword as onepassword + +# Read once +class Secrets: + aws_access_key = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="access key id", + ) + aws_secret_key = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="secret access key", + ) + +# Reuse +provider1 = aws.Provider("provider1", + access_key=Secrets.aws_access_key, + secret_key=Secrets.aws_secret_key, +) + +provider2 = aws.Provider("provider2", + access_key=Secrets.aws_access_key, + secret_key=Secrets.aws_secret_key, +) +``` + +## Alternative: Using direnv + +For local development, prefer direnv over 1Password provider: + +### .envrc Pattern + +```bash +# .envrc +export OP_ACCOUNT=nf-core + +source_url "https://github.com/tmatilai/direnv-1password/raw/v1.0.1/1password.sh" \ + "sha256-4dmKkmlPBNXimznxeehplDfiV+CvJiIzg7H1Pik4oqY=" + +# Load secrets from 1Password +from_op AWS_ACCESS_KEY_ID="op://Dev/Pulumi-AWS-key/access key id" +from_op AWS_SECRET_ACCESS_KEY="op://Dev/Pulumi-AWS-key/secret access key" +from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password" +``` + +Then in Python code: + +```python +import os + +# Credentials available from environment +# (no 1Password provider needed) +aws_key = os.getenv("AWS_ACCESS_KEY_ID") +aws_secret = os.getenv("AWS_SECRET_ACCESS_KEY") +``` + +### When to Use Each Approach + +**Use 1Password Provider when:** + +- Running in CI/CD without direnv +- Need to access secrets programmatically +- Sharing infrastructure code that needs secrets + +**Use direnv + 1Password when:** + +- Local development +- Interactive workflows +- Want credentials in environment for all tools + +## Troubleshooting + +### Service Account Not Found + +**Error:** + +``` +error: [ERROR] 2025/01/02 service account not found +``` + +**Solution:** +Verify OP_SERVICE_ACCOUNT_TOKEN is set: + +```bash +echo $OP_SERVICE_ACCOUNT_TOKEN +``` + +### Item Not Found + +**Error:** + +``` +error: item "AWS-Key" not found in vault "Dev" +``` + +**Solutions:** + +1. Check item name spelling +2. Verify vault name +3. Ensure service account has access to vault + +### Field Not Found + +**Error:** + +``` +error: field "password" not found in item +``` + +**Solution:** +Check field name in 1Password (exact match required): + +```python +# Common field names: +- "password" +- "credential" +- "access key id" +- "secret access key" +- "username" +``` + +### Permission Denied + +**Error:** + +``` +error: service account does not have permission to access item +``` + +**Solution:** +Grant service account access to vault in 1Password admin console. + +## Example: Complete Infrastructure Setup + +```python +"""Infrastructure using 1Password for all secrets.""" +import pulumi +import pulumi_onepassword as onepassword +import pulumi_aws as aws +import pulumi_github as github + +# Read AWS credentials +aws_key = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="access key id", +) + +aws_secret = onepassword.get_item_secret_output( + vault="Dev", + item="Pulumi-AWS-key", + field="secret access key", +) + +# Read GitHub token +github_token = onepassword.get_item_secret_output( + vault="Dev", + item="GitHub-Token", + field="credential", +) + +# Configure providers +aws_provider = aws.Provider( + "aws", + access_key=aws_key, + secret_key=aws_secret, + region="us-east-1", +) + +github_provider = github.Provider( + "github", + token=github_token, + owner="nf-core", +) + +# Create resources +bucket = aws.s3.Bucket( + "bucket", + opts=pulumi.ResourceOptions(provider=aws_provider), +) + +repo = github.Repository( + "repo", + name="my-repo", + opts=pulumi.ResourceOptions(provider=github_provider), +) + +# Store bucket info as GitHub secret +github.ActionsSecret( + "bucket-name", + repository=repo.name, + secret_name="S3_BUCKET", + plaintext_value=bucket.id, + opts=pulumi.ResourceOptions(provider=github_provider), +) +``` + +## Documentation Links + +- **1Password Provider**: https://www.pulumi.com/registry/packages/onepassword/ +- **API Reference**: https://www.pulumi.com/registry/packages/onepassword/api-docs/ +- **direnv-1password**: https://github.com/tmatilai/direnv-1password diff --git a/.claude/skills/pulumi/documentation/reference.md b/.claude/skills/pulumi/documentation/reference.md new file mode 100644 index 00000000..5895bea0 --- /dev/null +++ b/.claude/skills/pulumi/documentation/reference.md @@ -0,0 +1,342 @@ +# Pulumi Documentation Reference + +Advanced topics and comprehensive documentation resources. + +## Table of Contents + +- [Core Concepts](#core-concepts) +- [Advanced Features](#advanced-features) +- [Testing and Quality](#testing-and-quality) +- [Performance](#performance) +- [Official Resources](#official-resources) + +## Core Concepts + +### Resource Model + +Pulumi resources follow a declarative model: + +1. **Inputs**: Configuration provided to resource +2. **Outputs**: Computed values from resource +3. **Dependencies**: Automatic or explicit resource relationships +4. **State**: Current infrastructure state tracked by Pulumi + +### Output Values + +Outputs are Promise-like values resolved during deployment: + +```python +import pulumi + +# Outputs are not immediately available +bucket = aws.s3.Bucket("bucket") + +# Use apply() to work with output values +bucket_name = bucket.id.apply(lambda name: f"full-name-{name}") + +# Or use in other resources directly +object = aws.s3.BucketObject("object", + bucket=bucket.id, # Pulumi handles dependency automatically +) +``` + +### Resource Options + +Control resource behavior: + +```python +from pulumi import ResourceOptions, CustomTimeouts + +resource = Resource("name", + opts=ResourceOptions( + # Parent-child relationship + parent=parent_resource, + + # Explicit dependencies + depends_on=[other_resource], + + # Provider instance + provider=custom_provider, + + # Protect from deletion + protect=True, + + # Ignore changes to fields + ignore_changes=["tags"], + + # Custom timeouts + custom_timeouts=CustomTimeouts( + create="30m", + update="20m", + delete="10m", + ), + + # Replace on changes + replace_on_changes=["*"], + ), +) +``` + +## Advanced Features + +### Dynamic Providers + +Create custom resource providers: + +```python +from pulumi.dynamic import Resource, ResourceProvider, CreateResult + +class MyProvider(ResourceProvider): + def create(self, inputs): + # Custom create logic + return CreateResult(id_="unique-id", outs=inputs) + + def update(self, id, old_inputs, new_inputs): + # Custom update logic + return UpdateResult(outs=new_inputs) + + def delete(self, id, props): + # Custom delete logic + pass + +class MyResource(Resource): + def __init__(self, name, props, opts=None): + super().__init__(MyProvider(), name, props, opts) +``` + +### Automation API + +Programmatically manage Pulumi operations: + +```python +import pulumi.automation as auto + +# Create or select stack +stack = auto.create_or_select_stack( + stack_name="dev", + project_name="my-project", + program=lambda: None, # Your Pulumi program +) + +# Set configuration +stack.set_config("aws:region", auto.ConfigValue(value="us-east-1")) + +# Run operations +up_result = stack.up() +print(f"Update summary: {up_result.summary}") + +# Destroy +stack.destroy() +``` + +### Policy as Code + +Enforce compliance with policies: + +```python +# policy.py +from pulumi_policy import ( + EnforcementLevel, + PolicyPack, + ResourceValidationArgs, + ResourceValidationPolicy, +) + +def s3_bucket_encryption(args: ResourceValidationArgs, report_violation): + if args.resource_type == "aws:s3/bucket:Bucket": + encryption = args.props.get("serverSideEncryptionConfiguration") + if not encryption: + report_violation("S3 buckets must have encryption enabled") + +PolicyPack( + name="aws-compliance", + enforcement_level=EnforcementLevel.MANDATORY, + policies=[ + ResourceValidationPolicy( + name="s3-bucket-encryption", + description="Require S3 bucket encryption", + validate=s3_bucket_encryption, + ), + ], +) +``` + +### Transformations + +Modify resources automatically: + +```python +import pulumi + +def add_tags(args): + if args.type_.startswith("aws:"): + args.props["tags"] = args.props.get("tags", {}) + args.props["tags"]["ManagedBy"] = "Pulumi" + args.props["tags"]["Stack"] = pulumi.get_stack() + return pulumi.ResourceTransformationResult( + props=args.props, + opts=args.opts, + ) + +# Apply transformation to all resources +pulumi.runtime.register_stack_transformation(add_tags) +``` + +## Testing and Quality + +### Unit Testing + +Test component resources: + +```python +import unittest +from unittest.mock import patch, Mock +import pulumi + + +class TestInfrastructure(unittest.TestCase): + @pulumi.runtime.test + def test_bucket_creation(self): + import __main__ as program + + # Mock resources + def check_bucket(args): + assert args.bucket.startswith("my-project-") + assert args.acl == "private" + + pulumi.runtime.set_mocks(BucketMocks()) + + # Import program triggers resource creation + # Verify expectations in check_bucket +``` + +### Integration Testing + +Test actual deployments: + +```bash +#!/bin/bash +# integration_test.sh + +# Create test stack +pulumi stack init test-$RANDOM + +# Deploy +pulumi up --yes + +# Run tests against deployed infrastructure +python tests/integration_tests.py + +# Cleanup +pulumi destroy --yes +pulumi stack rm --yes +``` + +### Property Testing + +Test resource properties: + +```python +@pulumi.runtime.test +def test_all_buckets_private(): + import __main__ as program + + def check_bucket(args): + if args.type_ == "aws:s3/bucket:Bucket": + assert args.props["acl"] == "private", \ + f"Bucket {args.name} is not private" + + # Verify all buckets are private +``` + +## Performance + +### Parallel Operations + +```python +import pulumi + +# Create multiple resources in parallel +buckets = [] +for i in range(10): + bucket = aws.s3.Bucket(f"bucket-{i}") + buckets.append(bucket) + +# Pulumi handles parallelization automatically +``` + +### Resource Providers + +Use resource providers for large-scale operations: + +```python +# Don't create providers repeatedly +# BAD: +for region in regions: + provider = aws.Provider(region, region=region) + bucket = aws.s3.Bucket("bucket", opts=ResourceOptions(provider=provider)) + +# GOOD: Reuse providers +providers = {region: aws.Provider(region, region=region) for region in regions} +for region in regions: + bucket = aws.s3.Bucket(f"bucket-{region}", + opts=ResourceOptions(provider=providers[region]) + ) +``` + +### Reduce State Operations + +```bash +# Skip refresh for faster previews +pulumi preview --skip-refresh + +# Reduce parallelism if hitting rate limits +pulumi up --parallel 5 +``` + +## Official Resources + +### Documentation + +- **Main Docs**: https://www.pulumi.com/docs/ +- **Get Started**: https://www.pulumi.com/docs/get-started/ +- **Concepts**: https://www.pulumi.com/docs/intro/concepts/ +- **API Reference**: https://www.pulumi.com/docs/reference/ + +### Providers + +- **Registry**: https://www.pulumi.com/registry/ +- **AWS**: https://www.pulumi.com/registry/packages/aws/ +- **GitHub**: https://www.pulumi.com/registry/packages/github/ +- **1Password**: https://www.pulumi.com/registry/packages/onepassword/ + +### Examples + +- **Examples Repo**: https://github.com/pulumi/examples +- **AWS Examples**: https://github.com/pulumi/examples/tree/master/aws-py-* +- **Tutorials**: https://www.pulumi.com/docs/tutorials/ + +### Community + +- **Slack**: https://slack.pulumi.com +- **GitHub Discussions**: https://github.com/pulumi/pulumi/discussions +- **Stack Overflow**: https://stackoverflow.com/questions/tagged/pulumi + +### Learning + +- **Pulumi Learn**: https://www.pulumi.com/learn/ +- **Blog**: https://www.pulumi.com/blog/ +- **Videos**: https://www.pulumi.com/resources/ + +## Quick Tips + +1. **Use WebSearch** for latest documentation +2. **Check provider docs** for resource properties +3. **Read examples** for common patterns +4. **Test in development** before production +5. **Use TypeScript/Python** for better IDE support +6. **Enable logging** for debugging (--logtostderr -v=9) +7. **Join Slack** for community help +8. **Read changelogs** for breaking changes +9. **Use stack references** for modular infrastructure +10. **Document your infrastructure** in code comments diff --git a/.claude/skills/pulumi/import/SKILL.md b/.claude/skills/pulumi/import/SKILL.md new file mode 100644 index 00000000..1b72edea --- /dev/null +++ b/.claude/skills/pulumi/import/SKILL.md @@ -0,0 +1,290 @@ +--- +name: Importing Existing Infrastructure +description: Import manually created "click-ops" resources into Pulumi management. Use when bringing existing AWS, GitHub, or other cloud resources under infrastructure-as-code control. Supports single resource and bulk import workflows with automatic code generation. +--- + +# Importing Existing Infrastructure + +Bring manually created "click-ops" infrastructure under Pulumi management safely and systematically. + +## When to Use + +Use this skill when: + +- Migrating manually created cloud resources to infrastructure-as-code +- Bringing "click-ops" resources under version control +- Consolidating infrastructure from console/CLI creation to Pulumi +- Adopting Pulumi for existing infrastructure +- Taking over infrastructure created by others + +## Quick Single Resource Import + +### Step 1: Identify Resource + +Find the resource type token and ID: + +1. **Go to Pulumi Registry**: https://www.pulumi.com/registry/ +2. **Search for resource** (e.g., "AWS S3 Bucket") +3. **Find Import section** in documentation +4. **Copy type token** and **identify ID format** + +**Example:** + +- Resource: S3 bucket named `nf-core-co2-reports` +- Type token: `aws:s3/bucket:Bucket` +- ID (lookup property): `nf-core-co2-reports` + +### Step 2: Import Resource + +```bash +# Navigate to Pulumi project +cd ~/src/nf-core/ops/pulumi/co2_reports + +# Import resource +uv run pulumi import + +# Example: +uv run pulumi import aws:s3/bucket:Bucket co2-reports-bucket nf-core-co2-reports +``` + +### Step 3: Add Generated Code + +Pulumi automatically generates code for the resource: + +```python +# Copy generated code snippet into __main__.py +bucket = aws.s3.Bucket("co2-reports-bucket", + bucket="nf-core-co2-reports", + # ... generated properties ... +) +``` + +### Step 4: Verify Import + +```bash +# Run preview - should show NO changes +uv run pulumi preview + +# If changes shown, adjust code to match actual configuration +``` + +## Bulk Import Workflow + +For importing multiple resources efficiently: + +### Step 1: Create Import JSON File + +```json +{ + "resources": [ + { + "type": "aws:s3/bucket:Bucket", + "name": "co2-reports-bucket", + "id": "nf-core-co2-reports" + }, + { + "type": "aws:s3/bucketVersioning:BucketVersioning", + "name": "co2-reports-versioning", + "id": "nf-core-co2-reports" + }, + { + "type": "aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock", + "name": "co2-reports-public-block", + "id": "nf-core-co2-reports" + } + ] +} +``` + +Save as `resources-to-import.json`. + +### Step 2: Execute Bulk Import + +```bash +# Import all resources and generate code +uv run pulumi import -f resources-to-import.json -o imported-resources.py -y +``` + +Options: + +- `-f`: Input JSON file +- `-o`: Output file for generated code +- `-y`: Auto-approve without prompts + +### Step 3: Integrate Generated Code + +```bash +# Review generated code +cat imported-resources.py + +# Copy relevant sections to your __main__.py +# Adapt as needed for your project structure +``` + +### Step 4: Verify All Imports + +```bash +# Preview should show NO changes +uv run pulumi preview + +# View imported resources in state +uv run pulumi stack --show-urns +``` + +## Import Verification Checklist + +After importing, verify: + +- [ ] All resources appear in `pulumi stack --show-urns` +- [ ] `pulumi preview` shows **no changes** +- [ ] Generated code added to `__main__.py` +- [ ] Resource names match actual cloud resource names +- [ ] Dependencies between resources are correct +- [ ] Critical resources are protected (`protect=True`) +- [ ] Documentation added (what was imported and why) + +## Safety Guidelines + +### Protection + +Always protect critical imported resources: + +```python +bucket = aws.s3.Bucket( + "production-bucket", + bucket="nf-core-production-data", + opts=pulumi.ResourceOptions(protect=True) # Prevent accidental deletion +) +``` + +### Verification + +**Never skip verification:** + +```bash +# ALWAYS run preview after import +uv run pulumi preview + +# If changes shown, code doesn't match cloud reality +# Adjust code until preview shows no changes +``` + +### Documentation + +Document imports in code: + +```python +""" +Imported Resources: +- nf-core-co2-reports bucket: Imported 2025-01-02 + Originally created 2024-11-15 for CO2 footprint tracking + Import command: uv run pulumi import aws:s3/bucket:Bucket co2-reports-bucket nf-core-co2-reports +""" +``` + +## Common Resource Imports + +### AWS S3 Bucket + +```bash +# Type token: aws:s3/bucket:Bucket +# ID format: bucket name +uv run pulumi import aws:s3/bucket:Bucket my-bucket existing-bucket-name +``` + +### AWS IAM Role + +```bash +# Type token: aws:iam/role:Role +# ID format: role name +uv run pulumi import aws:iam/role:Role my-role existing-role-name +``` + +### AWS EC2 Instance + +```bash +# Type token: aws:ec2/instance:Instance +# ID format: instance ID (i-xxxxxxxxxxxx) +uv run pulumi import aws:ec2/instance:Instance my-instance i-0123456789abcdef0 +``` + +### GitHub Repository + +```bash +# Type token: github:index/repository:Repository +# ID format: repository name (owner/repo) +uv run pulumi import github:index/repository:Repository my-repo nf-core/ops +``` + +## Troubleshooting + +### Import Fails: Resource Not Found + +**Problem:** `error: resource not found` + +**Solutions:** + +1. Verify resource exists in cloud +2. Check correct region/account (provider configuration) +3. Verify ID format is correct (check Pulumi Registry) + +### Preview Shows Changes After Import + +**Problem:** `pulumi preview` shows changes despite import + +**Solutions:** + +1. Generated code missing properties - add all properties from generated code +2. Property values don't match - adjust code to match actual values +3. Auto-naming mismatch - ensure explicit names specified + +### Dependency Errors + +**Problem:** `error: resource depends on another resource` + +**Solutions:** + +1. Import parent resources first +2. Specify parent in import JSON +3. Import resources in dependency order + +See [../stack-management/troubleshooting.md](../stack-management/troubleshooting.md) for more issues. + +## Advanced Patterns + +For complex import scenarios, see [reference.md](reference.md): + +- Automated resource discovery +- Large-scale bulk imports +- Handling complex dependencies +- Import with parent resources +- Cross-stack imports +- State management after import + +## Examples + +See detailed provider-specific examples: + +- [AWS S3 Bucket Import](examples/aws-s3-bucket.md) - Complete S3 infrastructure +- [AWS IAM Role Import](examples/aws-iam-role.md) - Roles and policies +- [GitHub Repository Import](examples/github-repository.md) - Repository and settings + +## Helper Scripts + +Discover resources for import: + +```bash +# AWS resource discovery +python scripts/discover-aws-resources.py --region eu-west-1 --resource-type s3 + +# Generates import JSON file automatically +``` + +See [scripts/discover-aws-resources.py](scripts/discover-aws-resources.py) for details. + +## Related Skills + +- **Deploy**: Deploy infrastructure after import +- **Stack Management**: Organize imported resources across stacks +- **Documentation**: Find type tokens and import IDs +- **New Project**: Create projects for imported infrastructure diff --git a/.claude/skills/pulumi/import/examples/aws-iam-role.md b/.claude/skills/pulumi/import/examples/aws-iam-role.md new file mode 100644 index 00000000..6fa055cd --- /dev/null +++ b/.claude/skills/pulumi/import/examples/aws-iam-role.md @@ -0,0 +1,363 @@ +# Importing AWS IAM Roles and Policies + +Import existing IAM roles, policies, and their attachments into Pulumi. + +## Scenario: Import CI User and Policy + +Import the `nf-core-co2-reports-ci` IAM user and its associated policy. + +### Resources to Import + +1. **IAM User**: `nf-core-co2-reports-ci` +2. **IAM Policy**: `nf-core-co2-reports-bucket-access` +3. **Policy Attachment**: User to Policy attachment + +## Step 1: Discover Resources + +### Find IAM User + +```bash +# List users +aws iam list-users | jq '.Users[] | select(.UserName | contains("nf-core"))' + +# Get user details +aws iam get-user --user-name nf-core-co2-reports-ci +``` + +### Find Attached Policies + +```bash +# List attached policies +aws iam list-attached-user-policies --user-name nf-core-co2-reports-ci + +# Get policy ARN and details +aws iam get-policy --policy-arn arn:aws:iam::728131696474:policy/nf-core-co2-reports-bucket-access +``` + +## Step 2: Create Import JSON + +Create `iam-import.json`: + +```json +{ + "resources": [ + { + "type": "aws:iam/user:User", + "name": "ci_user", + "id": "nf-core-co2-reports-ci" + }, + { + "type": "aws:iam/policy:Policy", + "name": "bucket_access_policy", + "id": "arn:aws:iam::728131696474:policy/nf-core-co2-reports-bucket-access" + }, + { + "type": "aws:iam/userPolicyAttachment:UserPolicyAttachment", + "name": "ci_policy_attachment", + "id": "nf-core-co2-reports-ci/arn:aws:iam::728131696474:policy/nf-core-co2-reports-bucket-access" + } + ] +} +``` + +**Note:** UserPolicyAttachment ID format is `{user-name}/{policy-arn}` + +## Step 3: Execute Import + +```bash +cd ~/src/nf-core/ops/pulumi/co2_reports + +# Import resources +uv run pulumi import -f iam-import.json -o imported-iam.py -y +``` + +## Step 4: Review Generated Code + +Example generated code in `imported-iam.py`: + +```python +import pulumi +import pulumi_aws as aws +import json + +# IAM User +ci_user = aws.iam.User( + "ci_user", + name="nf-core-co2-reports-ci", + tags={ + "Purpose": "CI/CD", + }, +) + +# IAM Policy +bucket_access_policy = aws.iam.Policy( + "bucket_access_policy", + name="nf-core-co2-reports-bucket-access", + description="Write access to modules/ prefix in nf-core-co2-reports S3 bucket for CI/CD", + policy=json.dumps({ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::nf-core-co2-reports", + "Condition": { + "StringLike": { + "s3:prefix": ["modules/*"] + } + } + }, + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject" + ], + "Resource": "arn:aws:s3:::nf-core-co2-reports/modules/*" + } + ] + }), +) + +# Policy Attachment +ci_policy_attachment = aws.iam.UserPolicyAttachment( + "ci_policy_attachment", + user=ci_user.name, + policy_arn=bucket_access_policy.arn, +) +``` + +## Step 5: Add to Program with Protection + +Add to `__main__.py` with protection enabled: + +```python +# === IMPORTED IAM RESOURCES === + +# CI User (imported - protect from deletion) +ci_user = aws.iam.User( + "ci_user", + name="nf-core-co2-reports-ci", + tags={ + "Purpose": "CI/CD access to CO2 reports bucket", + "Imported": "2025-01-02", + }, + opts=pulumi.ResourceOptions(protect=True) # Critical resource +) + +# Bucket access policy (imported) +bucket_access_policy = aws.iam.Policy( + "bucket_access_policy", + name="nf-core-co2-reports-bucket-access", + description="Write access to modules/ prefix in nf-core-co2-reports S3 bucket for CI/CD", + policy=json.dumps({ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::nf-core-co2-reports", + "Condition": { + "StringLike": { + "s3:prefix": ["modules/*"] + } + } + }, + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject" + ], + "Resource": "arn:aws:s3:::nf-core-co2-reports/modules/*" + } + ] + }), + opts=pulumi.ResourceOptions(protect=True) +) + +# Attach policy to user (imported) +ci_policy_attachment = aws.iam.UserPolicyAttachment( + "ci_policy_attachment", + user=ci_user.name, + policy_arn=bucket_access_policy.arn, +) +``` + +## Step 6: Import Access Keys (Optional) + +**⚠️ Warning:** Access keys cannot be imported because the secret key is not retrievable after creation. + +**Options:** + +1. **Leave existing keys** in place (don't manage with Pulumi) +2. **Create new keys** with Pulumi and rotate in GitHub secrets +3. **Delete old keys** after creating new ones + +### Creating New Access Keys + +```python +# Create new access key (Pulumi-managed) +access_key = aws.iam.AccessKey( + "ci_access_key", + user=ci_user.name, +) + +# Update GitHub secrets with new keys +github.ActionsSecret( + "aws_key_id", + repository="modules", + secret_name="CO2_REPORTS_AWS_ACCESS_KEY_ID", + plaintext_value=access_key.id, +) + +github.ActionsSecret( + "aws_secret_key", + repository="modules", + secret_name="CO2_REPORTS_AWS_SECRET_ACCESS_KEY", + plaintext_value=pulumi.Output.secret(access_key.secret), +) + +# Export for verification +pulumi.export("access_key_id", access_key.id) +``` + +## Step 7: Verify Import + +```bash +# Preview - should show no changes to imported resources +uv run pulumi preview + +# Check imported resources +uv run pulumi stack --show-urns | grep iam +``` + +## Advanced: Import IAM Role with Trust Policy + +For IAM roles (like Lambda execution roles): + +### Discover Role + +```bash +# Get role details +aws iam get-role --role-name my-lambda-execution-role + +# Get attached policies +aws iam list-attached-role-policies --role-name my-lambda-execution-role + +# Get inline policies +aws iam list-role-policies --role-name my-lambda-execution-role +``` + +### Import JSON + +```json +{ + "resources": [ + { + "type": "aws:iam/role:Role", + "name": "lambda_execution_role", + "id": "my-lambda-execution-role" + }, + { + "type": "aws:iam/rolePolicyAttachment:RolePolicyAttachment", + "name": "lambda_basic_execution", + "id": "my-lambda-execution-role/arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" + } + ] +} +``` + +### Generated Code + +```python +# IAM Role +lambda_execution_role = aws.iam.Role( + "lambda_execution_role", + name="my-lambda-execution-role", + assume_role_policy=json.dumps({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { + "Service": "lambda.amazonaws.com" + }, + "Action": "sts:AssumeRole" + }] + }), +) + +# Attach AWS managed policy +lambda_basic_execution = aws.iam.RolePolicyAttachment( + "lambda_basic_execution", + role=lambda_execution_role.name, + policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", +) +``` + +## Common Issues + +### Issue: Policy ARN Not Found + +**Problem:** + +``` +error: policy 'arn:aws:iam::123:policy/my-policy' not found +``` + +**Solution:** Verify policy exists and ARN is correct: + +```bash +aws iam list-policies --scope Local | jq '.Policies[] | select(.PolicyName=="my-policy")' +``` + +### Issue: Attachment Import Fails + +**Problem:** + +``` +error: attachment not found with id 'user/policy-arn' +``` + +**Solution:** Import user and policy first, then attachment: + +```bash +# Import in order +uv run pulumi import aws:iam/user:User ci_user nf-core-co2-reports-ci +uv run pulumi import aws:iam/policy:Policy policy arn:aws:iam::123:policy/my-policy +uv run pulumi import aws:iam/userPolicyAttachment:UserPolicyAttachment attachment "nf-core-co2-reports-ci/arn:aws:iam::123:policy/my-policy" +``` + +### Issue: Policy Document Changes + +**Problem:** Preview shows policy document changes + +**Solution:** Format policy JSON consistently: + +```python +# Use json.dumps for consistent formatting +policy=json.dumps({ + "Version": "2012-10-17", + "Statement": [...] +}, indent=2, sort_keys=True) +``` + +## Best Practices + +1. **Always protect IAM resources**: `protect=True` +2. **Document trust relationships**: Comment assume role policies +3. **Group related resources**: Import role with all its attachments +4. **Rotate access keys**: Create new keys with Pulumi, delete old ones +5. **Audit permissions**: Review imported policies for least privilege +6. **Test access**: Verify CI/CD still works after import + +## Related Examples + +- [AWS S3 Bucket Import](aws-s3-bucket.md) - Import S3 resources +- [GitHub Repository Import](github-repository.md) - Import GitHub settings diff --git a/.claude/skills/pulumi/import/examples/aws-s3-bucket.md b/.claude/skills/pulumi/import/examples/aws-s3-bucket.md new file mode 100644 index 00000000..8deee4af --- /dev/null +++ b/.claude/skills/pulumi/import/examples/aws-s3-bucket.md @@ -0,0 +1,423 @@ +# Importing AWS S3 Buckets + +Complete walkthrough for importing existing S3 buckets and related configurations into Pulumi. + +## Scenario: Import nf-core-co2-reports Bucket + +### Background + +The `nf-core-co2-reports` bucket was created manually via AWS Console for CO2 footprint tracking. We want to bring it under Pulumi management. + +**Bucket details:** + +- Name: `nf-core-co2-reports` +- Region: `eu-north-1` +- Features: Versioning enabled, encryption enabled, public access blocked + +## Step 1: Identify Resources + +### Primary Resource + +**S3 Bucket:** + +- Type token: `aws:s3/bucket:Bucket` +- ID format: bucket name +- ID value: `nf-core-co2-reports` + +### Related Resources + +**Bucket Versioning:** + +- Type token: `aws:s3/bucketVersioningV2:BucketVersioningV2` +- ID format: bucket name +- ID value: `nf-core-co2-reports` + +**Encryption Configuration:** + +- Type token: `aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2` +- ID format: bucket name +- ID value: `nf-core-co2-reports` + +**Public Access Block:** + +- Type token: `aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock` +- ID format: bucket name +- ID value: `nf-core-co2-reports` + +## Step 2: Create Import JSON + +Create `s3-import.json`: + +```json +{ + "resources": [ + { + "type": "aws:s3/bucket:Bucket", + "name": "co2_reports_bucket", + "id": "nf-core-co2-reports" + }, + { + "type": "aws:s3/bucketVersioningV2:BucketVersioningV2", + "name": "co2_reports_versioning", + "id": "nf-core-co2-reports" + }, + { + "type": "aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2", + "name": "co2_reports_encryption", + "id": "nf-core-co2-reports" + }, + { + "type": "aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock", + "name": "co2_reports_public_block", + "id": "nf-core-co2-reports" + } + ] +} +``` + +## Step 3: Execute Import + +```bash +# Navigate to project +cd ~/src/nf-core/ops/pulumi/co2_reports + +# Ensure credentials loaded +echo $AWS_ACCESS_KEY_ID # Should show value +echo $PULUMI_CONFIG_PASSPHRASE # Should show value + +# Execute import +uv run pulumi import -f s3-import.json -o imported-s3.py -y +``` + +**Expected output:** + +``` +Importing (AWSMegatests) + +View Live: https://app.pulumi.com/... + + Type Name Status + + pulumi:pulumi:Stack co2_reports-AWSMegatests created + = ├─ aws:s3/bucket:Bucket co2_reports_bucket imported + = ├─ aws:s3/bucketVersioningV2:BucketVersioningV2 co2_reports_versioning imported + = ├─ aws:s3/bucketServerSideEncryptionConfigurationV2 co2_reports_encryption imported + = └─ aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock co2_reports_public_block imported + +Resources: + + 1 created + = 4 imported + 5 changes + +Duration: 12s +``` + +## Step 4: Review Generated Code + +Check `imported-s3.py`: + +```python +import pulumi +import pulumi_aws as aws + +# Main bucket +co2_reports_bucket = aws.s3.Bucket( + "co2_reports_bucket", + bucket="nf-core-co2-reports", + acl="private", + tags={ + "Environment": pulumi.get_stack(), + "Project": "co2-reports", + } +) + +# Versioning configuration +co2_reports_versioning = aws.s3.BucketVersioningV2( + "co2_reports_versioning", + bucket=co2_reports_bucket.id, + versioning_configuration=aws.s3.BucketVersioningV2VersioningConfigurationArgs( + status="Enabled", + ), +) + +# Encryption configuration +co2_reports_encryption = aws.s3.BucketServerSideEncryptionConfigurationV2( + "co2_reports_encryption", + bucket=co2_reports_bucket.id, + rules=[aws.s3.BucketServerSideEncryptionConfigurationV2RuleArgs( + apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs( + sse_algorithm="AES256", + ), + )], +) + +# Public access block +co2_reports_public_block = aws.s3.BucketPublicAccessBlock( + "co2_reports_public_block", + bucket=co2_reports_bucket.id, + block_public_acls=True, + block_public_policy=True, + ignore_public_acls=True, + restrict_public_buckets=True, +) + +# Export outputs +pulumi.export("bucket_name", co2_reports_bucket.id) +pulumi.export("bucket_arn", co2_reports_bucket.arn) +``` + +## Step 5: Add to Pulumi Program + +### Option A: Replace Existing Code + +If `__main__.py` already has the bucket creation code, replace it: + +```python +# __main__.py + +"""Pulumi program for co2_reports infrastructure. + +Imported Resources: +- nf-core-co2-reports S3 bucket + Imported: 2025-01-02 + Originally created: 2024-11-15 for CO2 footprint tracking + Import command: uv run pulumi import -f s3-import.json -y +""" + +import pulumi +import pulumi_aws as aws +import pulumi_github as github +import pulumi_onepassword as onepassword + +# Configuration +config = pulumi.Config() +aws_config = pulumi.Config("aws") +region = aws_config.require("region") + +# === IMPORTED RESOURCES === + +# Main S3 bucket (imported from existing infrastructure) +bucket = aws.s3.Bucket( + "co2_reports_bucket", + bucket="nf-core-co2-reports", + acl="private", + tags={ + "Environment": pulumi.get_stack(), + "Project": "co2-reports", + "ManagedBy": "Pulumi", + "Imported": "2025-01-02", + }, + opts=pulumi.ResourceOptions(protect=True) # Protect from accidental deletion +) + +# Versioning (imported) +bucket_versioning = aws.s3.BucketVersioningV2( + "co2_reports_versioning", + bucket=bucket.id, + versioning_configuration=aws.s3.BucketVersioningV2VersioningConfigurationArgs( + status="Enabled", + ), +) + +# Encryption (imported) +bucket_encryption = aws.s3.BucketServerSideEncryptionConfigurationV2( + "co2_reports_encryption", + bucket=bucket.id, + rules=[aws.s3.BucketServerSideEncryptionConfigurationV2RuleArgs( + apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs( + sse_algorithm="AES256", + ), + )], +) + +# Public access block (imported) +bucket_public_block = aws.s3.BucketPublicAccessBlock( + "co2_reports_public_block", + bucket=bucket.id, + block_public_acls=True, + block_public_policy=True, + ignore_public_acls=True, + restrict_public_buckets=True, +) + +# === NEW RESOURCES (if any) === + +# IAM user for CI/CD access +ci_user = aws.iam.User( + "ci_user", + name="nf-core-co2-reports-ci", + tags={ + "Purpose": "CI/CD access to CO2 reports bucket", + }, +) + +# Access key for CI user +access_key = aws.iam.AccessKey( + "ci_access_key", + user=ci_user.name, +) + +# IAM policy for bucket access +policy = aws.iam.Policy( + "bucket_access_policy", + policy=bucket.arn.apply(lambda arn: json.dumps({ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": arn, + "Condition": { + "StringLike": { + "s3:prefix": ["modules/*"] + } + } + }, + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject" + ], + "Resource": f"{arn}/modules/*" + } + ] + })), +) + +# Attach policy to user +policy_attachment = aws.iam.UserPolicyAttachment( + "ci_policy_attachment", + user=ci_user.name, + policy_arn=policy.arn, +) + +# GitHub secrets for CI/CD +github.ActionsSecret( + "aws_key_id", + repository="modules", + secret_name="CO2_REPORTS_AWS_ACCESS_KEY_ID", + plaintext_value=access_key.id, +) + +github.ActionsSecret( + "aws_secret_key", + repository="modules", + secret_name="CO2_REPORTS_AWS_SECRET_ACCESS_KEY", + plaintext_value=pulumi.Output.secret(access_key.secret), +) + +github.ActionsVariable( + "aws_region", + repository="modules", + variable_name="CO2_REPORTS_AWS_REGION", + value=region, +) + +# Export outputs +pulumi.export("bucket_name", bucket.id) +pulumi.export("bucket_arn", bucket.arn) +pulumi.export("ci_user_name", ci_user.name) +pulumi.export("access_key_id", access_key.id) +``` + +## Step 6: Verify Import + +```bash +# Run preview - should show NO changes to imported resources +uv run pulumi preview +``` + +**Expected output:** + +``` +Previewing update (AWSMegatests) + +View Live: https://app.pulumi.com/... + + Type Name Plan + pulumi:pulumi:Stack co2_reports-AWSMegatests + ~ ├─ aws:iam/user:User ci_user update + ~ └─ ... ... ... + +Resources: + ~ 2 to update + 4 unchanged + +Duration: 3s +``` + +✅ **Success**: Imported resources show as "unchanged" + +## Step 7: Test Deployment + +```bash +# Deploy any new changes +uv run pulumi up --yes + +# Verify bucket still works +aws s3 ls s3://nf-core-co2-reports/ --region eu-north-1 +``` + +## Common Issues + +### Issue: Preview Shows Changes to Bucket + +**Problem:** + +``` +~ aws:s3/bucket:Bucket: (update) + [urn=urn:pulumi:AWSMegatests::co2_reports::aws:s3/bucket:Bucket::co2_reports_bucket] + ~ acl: "private" => "public-read" +``` + +**Solution:** Generated code doesn't match actual configuration. Update code: + +```python +bucket = aws.s3.Bucket( + "co2_reports_bucket", + bucket="nf-core-co2-reports", + acl="private", # Match actual ACL + # Add any other properties shown in diff +) +``` + +### Issue: Import Fails - Bucket Not Found + +**Problem:** + +``` +error: resource 'nf-core-co2-reports' not found +``` + +**Solutions:** + +1. Verify bucket exists: `aws s3 ls s3://nf-core-co2-reports/` +2. Check region is correct in provider configuration +3. Verify AWS credentials have permission to access bucket + +### Issue: Related Resources Not Imported + +**Problem:** Versioning/encryption not included in import + +**Solution:** Import each configuration separately: + +```bash +uv run pulumi import aws:s3/bucketVersioningV2:BucketVersioningV2 co2_reports_versioning nf-core-co2-reports +uv run pulumi import aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2 co2_reports_encryption nf-core-co2-reports +``` + +## Next Steps + +1. **Import IAM resources** if any exist for the bucket +2. **Import bucket policies** if configured +3. **Import lifecycle rules** if configured +4. **Test CI/CD integration** with imported bucket +5. **Document in README** what was imported and when + +## Related Examples + +- [AWS IAM Role Import](aws-iam-role.md) - Import IAM resources +- [GitHub Repository Import](github-repository.md) - Import GitHub settings diff --git a/.claude/skills/pulumi/import/examples/github-repository.md b/.claude/skills/pulumi/import/examples/github-repository.md new file mode 100644 index 00000000..a7c992d7 --- /dev/null +++ b/.claude/skills/pulumi/import/examples/github-repository.md @@ -0,0 +1,388 @@ +# Importing GitHub Repositories + +Import existing GitHub repositories and their configurations into Pulumi. + +## Scenario: Import nf-core/modules Repository Settings + +Import the nf-core/modules repository and related GitHub resources created via the web interface. + +### Resources to Import + +1. **Repository**: `nf-core/modules` +2. **Branch Protection**: `main` branch protection rules +3. **Team Access**: Team permissions for the repository +4. **Actions Secrets**: (Note: Cannot import existing secrets - must recreate) + +## Step 1: Identify Resources + +### Repository Details + +```bash +# Get repository info via GitHub CLI +gh repo view nf-core/modules --json name,owner,isPrivate,description + +# Or via API +curl -H "Authorization: token $GITHUB_TOKEN" \ + https://api.github.com/repos/nf-core/modules +``` + +**Type token**: `github:index/repository:Repository` +**ID format**: `owner/repo` +**ID value**: `nf-core/modules` + +### Branch Protection + +```bash +# List protected branches +gh api repos/nf-core/modules/branches/main/protection + +# Get protection rules +gh api repos/nf-core/modules/branches/main/protection \ + --jq '.required_pull_request_reviews' +``` + +**Type token**: `github:index/branchProtection:BranchProtection` +**ID format**: `repository:pattern` +**ID value**: `nf-core/modules:main` + +### Team Access + +```bash +# List teams with access +gh api repos/nf-core/modules/teams + +# Get team ID +gh api orgs/nf-core/teams/platform-team --jq '.id' +``` + +**Type token**: `github:index/teamRepository:TeamRepository` +**ID format**: `team-id:repository` +**ID value**: `12345:nf-core/modules` + +## Step 2: Create Import JSON + +Create `github-import.json`: + +```json +{ + "resources": [ + { + "type": "github:index/repository:Repository", + "name": "modules_repo", + "id": "nf-core/modules" + }, + { + "type": "github:index/branchProtection:BranchProtection", + "name": "main_protection", + "id": "nf-core/modules:main" + }, + { + "type": "github:index/teamRepository:TeamRepository", + "name": "platform_team_access", + "id": "12345:nf-core/modules" + } + ] +} +``` + +## Step 3: Configure GitHub Provider + +Ensure GitHub provider is configured: + +```python +# In __main__.py or as environment variable +import pulumi_github as github + +# Provider uses GITHUB_TOKEN from environment +# Or configure explicitly: +provider = github.Provider( + "github", + token=github_token, + owner="nf-core", +) +``` + +## Step 4: Execute Import + +```bash +cd ~/src/nf-core/ops/pulumi/github-infrastructure + +# Ensure GitHub token is set +echo $GITHUB_TOKEN # Should show token + +# Import resources +uv run pulumi import -f github-import.json -o imported-github.py -y +``` + +## Step 5: Review Generated Code + +Example in `imported-github.py`: + +```python +import pulumi +import pulumi_github as github + +# Repository +modules_repo = github.Repository( + "modules_repo", + name="modules", + description="Repository of modules for use with the nf-core pipeline framework", + visibility="public", + has_issues=True, + has_projects=True, + has_wiki=False, + has_discussions=True, + allow_merge_commit=True, + allow_squash_merge=True, + allow_rebase_merge=False, + delete_branch_on_merge=True, + vulnerability_alerts=True, + topics=["nextflow", "nf-core", "bioinformatics"], +) + +# Branch Protection +main_protection = github.BranchProtection( + "main_protection", + repository_id=modules_repo.node_id, + pattern="main", + required_pull_request_reviews=github.BranchProtectionRequiredPullRequestReviewsArgs( + dismiss_stale_reviews=True, + require_code_owner_reviews=True, + required_approving_review_count=2, + ), + enforce_admins=True, + require_signed_commits=False, + required_status_checks=github.BranchProtectionRequiredStatusChecksArgs( + strict=True, + contexts=["ci/test", "ci/lint"], + ), +) + +# Team Access +platform_team_access = github.TeamRepository( + "platform_team_access", + team_id="12345", + repository=modules_repo.name, + permission="push", # pull, triage, push, maintain, admin +) +``` + +## Step 6: Add to Program + +Add to your Pulumi program with documentation: + +```python +"""GitHub Infrastructure for nf-core/modules + +Imported Resources: +- nf-core/modules repository + Imported: 2025-01-02 + Originally created via GitHub web interface +""" + +import pulumi +import pulumi_github as github + +# === IMPORTED GITHUB RESOURCES === + +# Main repository (imported) +modules_repo = github.Repository( + "modules_repo", + name="modules", + description="Repository of modules for use with the nf-core pipeline framework", + visibility="public", + has_issues=True, + has_projects=True, + has_wiki=False, + has_discussions=True, + allow_merge_commit=True, + allow_squash_merge=True, + allow_rebase_merge=False, + delete_branch_on_merge=True, + vulnerability_alerts=True, + topics=["nextflow", "nf-core", "bioinformatics"], + homepage_url="https://nf-co.re/modules", + opts=pulumi.ResourceOptions(protect=True) # Critical repository +) + +# Branch protection (imported) +main_protection = github.BranchProtection( + "main_protection", + repository_id=modules_repo.node_id, + pattern="main", + required_pull_request_reviews=github.BranchProtectionRequiredPullRequestReviewsArgs( + dismiss_stale_reviews=True, + require_code_owner_reviews=True, + required_approving_review_count=2, + ), + enforce_admins=True, + required_status_checks=github.BranchProtectionRequiredStatusChecksArgs( + strict=True, + contexts=["ci/test", "ci/lint"], + ), +) + +# === NEW GITHUB RESOURCES === + +# New Actions secrets (existing secrets cannot be imported) +github.ActionsSecret( + "co2_aws_key", + repository=modules_repo.name, + secret_name="CO2_REPORTS_AWS_ACCESS_KEY_ID", + plaintext_value=pulumi.Output.secret(aws_access_key_id), +) + +github.ActionsSecret( + "co2_aws_secret", + repository=modules_repo.name, + secret_name="CO2_REPORTS_AWS_SECRET_ACCESS_KEY", + plaintext_value=pulumi.Output.secret(aws_secret_access_key), +) + +# Export outputs +pulumi.export("repository_url", modules_repo.html_url) +pulumi.export("repository_clone_url", modules_repo.ssh_clone_url) +``` + +## Step 7: Verify Import + +```bash +# Preview - should show no changes to imported resources +uv run pulumi preview + +# Check repository still accessible +gh repo view nf-core/modules +``` + +## Important Notes + +### Cannot Import Actions Secrets + +**GitHub Actions secrets cannot be imported** because they are write-only in GitHub's API. + +**Options:** + +1. **Recreate secrets** with Pulumi +2. **Leave existing secrets** unmanaged (manual management) +3. **Use placeholder values** and update manually + +### Recreating Secrets + +```python +# Create new secrets (will overwrite existing) +github.ActionsSecret( + "secret_name", + repository=repo.name, + secret_name="EXISTING_SECRET", + plaintext_value=pulumi.Output.secret(new_value), +) +``` + +### Repository Node ID + +Branch protection requires repository's `node_id`, not `name`: + +```python +# Correct +branch_protection = github.BranchProtection( + "protection", + repository_id=repo.node_id, # Use node_id + pattern="main", +) + +# Incorrect +branch_protection = github.BranchProtection( + "protection", + repository_id=repo.name, # Wrong - will fail + pattern="main", +) +``` + +## Advanced: Import Multiple Repositories + +### Discover All Organization Repositories + +```bash +# List all org repos +gh repo list nf-core --limit 100 --json name,owner + +# Generate import JSON +gh repo list nf-core --json name,owner --jq \ + '[.[] | {type: "github:index/repository:Repository", name: (.name | gsub("-"; "_")), id: (.owner.login + "/" + .name)}]' \ + > repos-import.json +``` + +### Bulk Import + +```bash +# Import all repositories +uv run pulumi import -f repos-import.json -o imported-repos.py -y +``` + +## Common Issues + +### Issue: Repository Not Found + +**Problem:** + +``` +error: repository 'nf-core/modules' not found +``` + +**Solutions:** + +1. Verify token has access to repository +2. Check organization name is correct +3. Verify repository exists: `gh repo view nf-core/modules` + +### Issue: Branch Protection Import Fails + +**Problem:** + +``` +error: branch protection not found for 'nf-core/modules:main' +``` + +**Solutions:** + +1. Verify branch protection exists on that branch +2. Check branch name is correct (case-sensitive) +3. Ensure token has admin access to repository + +### Issue: Team ID Not Found + +**Problem:** + +``` +error: team repository access not found with id '12345:nf-core/modules' +``` + +**Solutions:** + +1. Verify team has access to repository: + +```bash +gh api repos/nf-core/modules/teams +``` + +2. Get correct team ID: + +```bash +gh api orgs/nf-core/teams/platform-team --jq '.id' +``` + +3. Use correct format: `team-id:owner/repo` + +## Best Practices + +1. **Protect repositories**: `protect=True` on critical repos +2. **Import branch protection**: Ensure protection rules are managed +3. **Recreate secrets**: Import workflow, recreate secrets +4. **Document permissions**: Comment why teams have access +5. **Test webhooks**: Verify integrations still work +6. **Audit access**: Review team permissions after import + +## Related Examples + +- [AWS S3 Bucket Import](aws-s3-bucket.md) - Import S3 resources +- [AWS IAM Role Import](aws-iam-role.md) - Import IAM resources diff --git a/.claude/skills/pulumi/import/reference.md b/.claude/skills/pulumi/import/reference.md new file mode 100644 index 00000000..dcbf05b1 --- /dev/null +++ b/.claude/skills/pulumi/import/reference.md @@ -0,0 +1,644 @@ +# Pulumi Import Reference + +Advanced patterns and techniques for importing existing infrastructure into Pulumi. + +## Table of Contents + +- [Import Methods](#import-methods) +- [Automated Discovery](#automated-discovery) +- [Handling Dependencies](#handling-dependencies) +- [Large-Scale Import Strategies](#large-scale-import-strategies) +- [Provider-Specific Patterns](#provider-specific-patterns) +- [State Management](#state-management) +- [Troubleshooting](#troubleshooting) + +## Import Methods + +### CLI-Based Import (Recommended) + +**Single resource:** + +```bash +uv run pulumi import +``` + +**Advantages:** + +- Automatic code generation +- Immediate feedback +- Simple workflow +- Protection enabled by default + +**Use when:** + +- Importing few resources +- Learning import workflow +- Quick one-off imports + +### Bulk Import + +**Multiple resources from JSON:** + +```bash +uv run pulumi import -f resources.json -o generated.py -y +``` + +**Advantages:** + +- Efficient for many resources +- Declarative approach +- Can be scripted +- Generates organized code + +**Use when:** + +- Importing many resources +- Systematic migration +- Automated workflows + +### Code-Based Import + +**Using ResourceOptions:** + +```python +bucket = aws.s3.Bucket( + "existing-bucket", + bucket="my-existing-bucket", + opts=pulumi.ResourceOptions(import_="my-existing-bucket") +) +``` + +Then run `pulumi up` to import. + +**Advantages:** + +- Code-first approach +- Good for multi-stack scenarios +- Integrates with existing code + +**Use when:** + +- Managing multiple stacks +- Conditional imports +- Programmatic control needed + +## Automated Discovery + +### AWS Resource Discovery + +Automate finding resources to import: + +```python +#!/usr/bin/env python3 +"""Discover AWS resources for Pulumi import.""" + +import boto3 +import json +from typing import List, Dict + +def discover_s3_buckets(region: str, prefix: str = "nf-core-") -> List[Dict]: + """Discover S3 buckets matching prefix.""" + s3 = boto3.client('s3', region_name=region) + buckets = s3.list_buckets()['Buckets'] + + resources = [] + for bucket in buckets: + if bucket['Name'].startswith(prefix): + resources.append({ + "type": "aws:s3/bucket:Bucket", + "name": bucket['Name'].replace(prefix, '').replace('-', '_'), + "id": bucket['Name'] + }) + + # Discover bucket configurations + try: + # Versioning + versioning = s3.get_bucket_versioning(Bucket=bucket['Name']) + if versioning.get('Status') == 'Enabled': + resources.append({ + "type": "aws:s3/bucketVersioning:BucketVersioning", + "name": f"{bucket['Name'].replace(prefix, '').replace('-', '_')}_versioning", + "id": bucket['Name'] + }) + except Exception as e: + print(f"Warning: Could not check versioning for {bucket['Name']}: {e}") + + return resources + +def discover_iam_roles(region: str, prefix: str = "nf-core-") -> List[Dict]: + """Discover IAM roles matching prefix.""" + iam = boto3.client('iam', region_name=region) + roles = iam.list_roles()['Roles'] + + resources = [] + for role in roles: + if role['RoleName'].startswith(prefix): + resources.append({ + "type": "aws:iam/role:Role", + "name": role['RoleName'].replace(prefix, '').replace('-', '_'), + "id": role['RoleName'] + }) + + # Discover attached policies + try: + policies = iam.list_attached_role_policies(RoleName=role['RoleName']) + for policy in policies['AttachedPolicies']: + # Only import custom policies (not AWS managed) + if not policy['PolicyArn'].startswith('arn:aws:iam::aws:'): + resources.append({ + "type": "aws:iam/policy:Policy", + "name": policy['PolicyName'].replace('-', '_'), + "id": policy['PolicyArn'] + }) + except Exception as e: + print(f"Warning: Could not check policies for {role['RoleName']}: {e}") + + return resources + +def generate_import_json(resources: List[Dict], output_file: str): + """Generate import JSON file.""" + import_data = {"resources": resources} + + with open(output_file, 'w') as f: + json.dump(import_data, f, indent=2) + + print(f"Generated import file: {output_file}") + print(f"Found {len(resources)} resources to import") + +# Usage +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description='Discover AWS resources for Pulumi import') + parser.add_argument('--region', default='us-east-1', help='AWS region') + parser.add_argument('--prefix', default='nf-core-', help='Resource name prefix') + parser.add_argument('--resource-type', choices=['s3', 'iam', 'all'], default='all') + parser.add_argument('--output', default='discovered-resources.json', help='Output JSON file') + + args = parser.parse_args() + + resources = [] + + if args.resource_type in ['s3', 'all']: + print(f"Discovering S3 buckets in {args.region}...") + resources.extend(discover_s3_buckets(args.region, args.prefix)) + + if args.resource_type in ['iam', 'all']: + print(f"Discovering IAM roles...") + resources.extend(discover_iam_roles(args.region, args.prefix)) + + generate_import_json(resources, args.output) + + print(f"\nNext steps:") + print(f"1. Review {args.output}") + print(f"2. Run: uv run pulumi import -f {args.output} -o imported.py -y") + print(f"3. Add generated code to your Pulumi program") + print(f"4. Verify: uv run pulumi preview") +``` + +**Usage:** + +```bash +# Discover all nf-core resources +python scripts/discover-aws-resources.py --region eu-west-1 --prefix nf-core- + +# Import discovered resources +uv run pulumi import -f discovered-resources.json -o imported.py -y +``` + +### GitHub Resource Discovery + +```python +#!/usr/bin/env python3 +"""Discover GitHub resources for Pulumi import.""" + +import requests +import json +from typing import List, Dict + +def discover_github_repos(org: str, token: str) -> List[Dict]: + """Discover GitHub repositories in organization.""" + headers = { + 'Authorization': f'token {token}', + 'Accept': 'application/vnd.github.v3+json' + } + + url = f'https://api.github.com/orgs/{org}/repos' + response = requests.get(url, headers=headers) + repos = response.json() + + resources = [] + for repo in repos: + resources.append({ + "type": "github:index/repository:Repository", + "name": repo['name'].replace('-', '_'), + "id": repo['full_name'] + }) + + return resources + +# Usage +if __name__ == "__main__": + import os + + org = "nf-core" + token = os.getenv("GITHUB_TOKEN") + + if not token: + print("Error: GITHUB_TOKEN environment variable not set") + exit(1) + + print(f"Discovering repositories in {org}...") + resources = discover_github_repos(org, token) + + with open('github-repos.json', 'w') as f: + json.dump({"resources": resources}, f, indent=2) + + print(f"Found {len(resources)} repositories") + print("Run: uv run pulumi import -f github-repos.json -o imported-github.py -y") +``` + +## Handling Dependencies + +### Parent-Child Relationships + +When resources have parent-child relationships, specify parents in import JSON: + +```json +{ + "resources": [ + { + "type": "aws:s3/bucket:Bucket", + "name": "my_bucket", + "id": "my-bucket-name" + }, + { + "type": "aws:s3/bucketVersioning:BucketVersioning", + "name": "my_bucket_versioning", + "id": "my-bucket-name", + "parent": "urn:pulumi:stack::project::aws:s3/bucket:Bucket::my_bucket" + }, + { + "type": "aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock", + "name": "my_bucket_public_block", + "id": "my-bucket-name", + "parent": "urn:pulumi:stack::project::aws:s3/bucket:Bucket::my_bucket" + } + ] +} +``` + +**Finding parent URN:** + +```bash +# Import parent first +uv run pulumi import aws:s3/bucket:Bucket my_bucket my-bucket-name + +# Get parent URN +uv run pulumi stack --show-urns | grep my_bucket +``` + +### Import Order + +Import resources in dependency order: + +**Phase 1: Foundational Resources** + +- VPCs +- S3 buckets +- IAM roles (without policies) + +**Phase 2: Network Resources** + +- Subnets +- Route tables +- Security groups + +**Phase 3: Application Resources** + +- EC2 instances +- ECS services +- Lambda functions + +**Phase 4: Data Resources** + +- RDS databases +- DynamoDB tables +- ElastiCache clusters + +### Cross-Resource Dependencies + +Use explicit dependencies in generated code: + +```python +# Import bucket +bucket = aws.s3.Bucket("bucket", bucket="my-bucket") + +# Import policy that depends on bucket +policy = aws.iam.Policy( + "bucket_policy", + policy=bucket.arn.apply(lambda arn: json.dumps({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": "s3:GetObject", + "Resource": f"{arn}/*" + }] + })), + opts=pulumi.ResourceOptions(depends_on=[bucket]) +) +``` + +## Large-Scale Import Strategies + +### Phased Approach + +**Phase 1: Discovery (Week 1)** + +- Audit existing infrastructure +- Document resources by project/application +- Identify dependencies +- Generate import JSON files + +**Phase 2: Foundational Import (Week 2)** + +- Import VPCs, S3 buckets, base IAM roles +- Verify imports +- Set up protection +- Document in code + +**Phase 3: Application Import (Week 3-4)** + +- Import compute resources +- Import application-specific resources +- Verify all integrations +- Test deployments + +**Phase 4: Cleanup (Week 5)** + +- Remove unused resources +- Consolidate duplicate resources +- Optimize resource organization +- Update documentation + +### Organization Patterns + +**Option A: Separate Import Project** + +``` +pulumi/ +├── existing-infrastructure/ # Imported resources +│ ├── __main__.py +│ └── Pulumi.yaml +└── new-infrastructure/ # New resources + ├── __main__.py + └── Pulumi.yaml +``` + +**Option B: Mixed in Existing Project** + +```python +# __main__.py + +# === Imported Resources === +imported_bucket = aws.s3.Bucket( + "imported_megatests", + bucket="nf-core-awsmegatests", + # ... imported properties ... +) + +# === New Resources === +new_bucket = aws.s3.Bucket( + "new_co2_reports", + bucket=f"nf-core-co2-reports-{pulumi.get_stack()}", + # ... new configuration ... +) +``` + +**Option C: By Domain** + +``` +pulumi/ +├── networking/ # VPCs, subnets (imported + new) +├── storage/ # S3, EFS (imported + new) +├── compute/ # EC2, ECS (imported + new) +└── data/ # RDS, DynamoDB (imported + new) +``` + +### Batch Processing + +For very large imports: + +```bash +# Split into batches of 50 resources +split -l 50 all-resources.json batch- + +# Import each batch +for batch in batch-*; do + echo "Importing $batch..." + uv run pulumi import -f "$batch" -y + sleep 5 # Rate limiting +done +``` + +## Provider-Specific Patterns + +### AWS S3 Complete Infrastructure + +Import bucket with all configurations: + +```json +{ + "resources": [ + { + "type": "aws:s3/bucket:Bucket", + "name": "my_bucket", + "id": "my-bucket-name" + }, + { + "type": "aws:s3/bucketVersioningV2:BucketVersioningV2", + "name": "my_bucket_versioning", + "id": "my-bucket-name" + }, + { + "type": "aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2", + "name": "my_bucket_encryption", + "id": "my-bucket-name" + }, + { + "type": "aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock", + "name": "my_bucket_public_block", + "id": "my-bucket-name" + }, + { + "type": "aws:s3/bucketLifecycleConfigurationV2:BucketLifecycleConfigurationV2", + "name": "my_bucket_lifecycle", + "id": "my-bucket-name" + } + ] +} +``` + +### AWS IAM Role with Policies + +```json +{ + "resources": [ + { + "type": "aws:iam/role:Role", + "name": "lambda_role", + "id": "my-lambda-execution-role" + }, + { + "type": "aws:iam/policy:Policy", + "name": "lambda_policy", + "id": "arn:aws:iam::123456789:policy/my-lambda-policy" + }, + { + "type": "aws:iam/rolePolicyAttachment:RolePolicyAttachment", + "name": "lambda_attach", + "id": "my-lambda-execution-role/arn:aws:iam::123456789:policy/my-lambda-policy" + } + ] +} +``` + +### GitHub Repository with Settings + +```bash +# Import repository +uv run pulumi import github:index/repository:Repository ops_repo nf-core/ops + +# Import branch protection +uv run pulumi import github:index/branchProtection:BranchProtection ops_main_protection "nf-core/ops:main" + +# Import team access +uv run pulumi import github:index/teamRepository:TeamRepository ops_team_access "12345:nf-core/ops" +``` + +## State Management + +### Backup Before Import + +Always backup state before large imports: + +```bash +# Export current state +uv run pulumi stack export > pre-import-backup.json + +# Perform import +uv run pulumi import -f resources.json -y + +# If issues occur, restore +uv run pulumi stack import --file pre-import-backup.json +``` + +### State Verification + +After import, verify state integrity: + +```bash +# List all resources +uv run pulumi stack --show-urns + +# Check for orphaned resources +uv run pulumi refresh + +# Verify no pending changes +uv run pulumi preview +``` + +### Removing Imported Resources + +If you need to remove imported resources from Pulumi management without deleting them: + +```bash +# Remove from state but keep in cloud +uv run pulumi state delete --yes + +# Example +uv run pulumi state delete urn:pulumi:production::project::aws:s3/bucket:Bucket::old_bucket --yes +``` + +## Troubleshooting + +### Import Fails with "Resource Already Exists" + +**Problem:** Resource already managed by another Pulumi stack or Terraform + +**Solutions:** + +1. Check if resource is in another stack +2. If migrating from Terraform, remove from Terraform state first +3. Verify not already imported + +### Generated Code Doesn't Match + +**Problem:** Preview shows changes after adding generated code + +**Solutions:** + +1. **Check for auto-naming**: Explicitly set resource names + +```python +# Bad +bucket = aws.s3.Bucket("bucket") # Pulumi adds random suffix + +# Good +bucket = aws.s3.Bucket("bucket", bucket="actual-bucket-name") +``` + +2. **Add missing properties**: Include ALL properties from generated code + +3. **Check provider configuration**: Ensure region/account match + +### Dependency Import Errors + +**Problem:** Cannot import child resource without parent + +**Solutions:** + +1. Import parent first +2. Use parent URN in import JSON +3. Import resources in correct order + +### Protection Errors + +**Problem:** Cannot delete/replace protected imported resource + +**Solutions:** + +1. **Disable protection temporarily:** + +```python +opts=pulumi.ResourceOptions(protect=False) +``` + +2. **Or use state commands:** + +```bash +uv run pulumi state unprotect +``` + +### Large Import Timeouts + +**Problem:** Import times out for many resources + +**Solutions:** + +1. Break into smaller batches +2. Increase timeout: `--timeout 30m` +3. Import incrementally + +## Best Practices Summary + +1. **Audit first**: Know what you're importing before starting +2. **Start small**: Import incrementally, test as you go +3. **Backup state**: Always export state before large imports +4. **Verify always**: Run preview after every import +5. **Protect resources**: Enable protection on critical resources +6. **Document imports**: Record what, when, why in code comments +7. **Use discovery**: Automate finding resources with scripts +8. **Test thoroughly**: Verify integrations after import +9. **Clean up**: Remove unused resources after import +10. **Train team**: Ensure team understands import workflow diff --git a/.claude/skills/pulumi/import/scripts/discover-aws-resources.py b/.claude/skills/pulumi/import/scripts/discover-aws-resources.py new file mode 100755 index 00000000..5e7ef075 --- /dev/null +++ b/.claude/skills/pulumi/import/scripts/discover-aws-resources.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Discover AWS resources for Pulumi import. + +This script discovers AWS resources matching specified criteria and generates +a Pulumi import JSON file for bulk import operations. + +Usage: + python discover-aws-resources.py --region eu-west-1 --resource-type s3 + python discover-aws-resources.py --prefix nf-core- --output resources.json +""" + +import boto3 +import json +import argparse +import sys +from typing import List, Dict + + +def discover_s3_buckets(region: str, prefix: str = "nf-core-") -> List[Dict]: + """Discover S3 buckets matching prefix. + + Args: + region: AWS region + prefix: Bucket name prefix filter + + Returns: + List of resource dictionaries for import + """ + print(f"Discovering S3 buckets in {region} with prefix '{prefix}'...") + + s3 = boto3.client("s3", region_name=region) + resources = [] + + try: + buckets = s3.list_buckets()["Buckets"] + + for bucket in buckets: + if not bucket["Name"].startswith(prefix): + continue + + resource_name = bucket["Name"].replace(prefix, "").replace("-", "_") + print(f" Found bucket: {bucket['Name']}") + + # Main bucket + resources.append( + { + "type": "aws:s3/bucket:Bucket", + "name": f"{resource_name}_bucket", + "id": bucket["Name"], + } + ) + + # Try to discover bucket configurations + try: + # Versioning + versioning = s3.get_bucket_versioning(Bucket=bucket["Name"]) + if versioning.get("Status") == "Enabled": + print(" - Versioning enabled") + resources.append( + { + "type": "aws:s3/bucketVersioningV2:BucketVersioningV2", + "name": f"{resource_name}_versioning", + "id": bucket["Name"], + } + ) + + # Encryption + encryption = s3.get_bucket_encryption(Bucket=bucket["Name"]) + if encryption.get("ServerSideEncryptionConfiguration"): + print(" - Encryption configured") + resources.append( + { + "type": "aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2", + "name": f"{resource_name}_encryption", + "id": bucket["Name"], + } + ) + + # Public Access Block + public_block = s3.get_public_access_block(Bucket=bucket["Name"]) + if public_block.get("PublicAccessBlockConfiguration"): + print(" - Public access block configured") + resources.append( + { + "type": "aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock", + "name": f"{resource_name}_public_block", + "id": bucket["Name"], + } + ) + + except s3.exceptions.ClientError as e: + # Configuration not set or permission denied + if "ServerSideEncryptionConfigurationNotFoundError" in str(e): + pass + elif "NoSuchPublicAccessBlockConfiguration" in str(e): + pass + else: + print(f" Warning: Could not check configurations: {e}") + + except Exception as e: + print(f"Error discovering S3 buckets: {e}", file=sys.stderr) + + return resources + + +def discover_iam_users(region: str, prefix: str = "nf-core-") -> List[Dict]: + """Discover IAM users matching prefix. + + Args: + region: AWS region (IAM is global but client needs region) + prefix: User name prefix filter + + Returns: + List of resource dictionaries for import + """ + print(f"Discovering IAM users with prefix '{prefix}'...") + + iam = boto3.client("iam", region_name=region) + resources = [] + + try: + users = iam.list_users()["Users"] + + for user in users: + if not user["UserName"].startswith(prefix): + continue + + resource_name = user["UserName"].replace(prefix, "").replace("-", "_") + print(f" Found user: {user['UserName']}") + + # User + resources.append( + { + "type": "aws:iam/user:User", + "name": f"{resource_name}_user", + "id": user["UserName"], + } + ) + + # Attached policies + try: + policies = iam.list_attached_user_policies(UserName=user["UserName"]) + for policy in policies["AttachedPolicies"]: + # Only custom policies (not AWS managed) + if not policy["PolicyArn"].startswith("arn:aws:iam::aws:"): + policy_name = policy["PolicyName"].replace("-", "_") + print(f" - Policy: {policy['PolicyName']}") + + resources.append( + { + "type": "aws:iam/policy:Policy", + "name": f"{policy_name}_policy", + "id": policy["PolicyArn"], + } + ) + + resources.append( + { + "type": "aws:iam/userPolicyAttachment:UserPolicyAttachment", + "name": f"{resource_name}_{policy_name}_attachment", + "id": f"{user['UserName']}/{policy['PolicyArn']}", + } + ) + + except Exception as e: + print(f" Warning: Could not check policies: {e}") + + except Exception as e: + print(f"Error discovering IAM users: {e}", file=sys.stderr) + + return resources + + +def discover_iam_roles(region: str, prefix: str = "nf-core-") -> List[Dict]: + """Discover IAM roles matching prefix. + + Args: + region: AWS region (IAM is global but client needs region) + prefix: Role name prefix filter + + Returns: + List of resource dictionaries for import + """ + print(f"Discovering IAM roles with prefix '{prefix}'...") + + iam = boto3.client("iam", region_name=region) + resources = [] + + try: + roles = iam.list_roles()["Roles"] + + for role in roles: + if not role["RoleName"].startswith(prefix): + continue + + resource_name = role["RoleName"].replace(prefix, "").replace("-", "_") + print(f" Found role: {role['RoleName']}") + + # Role + resources.append( + { + "type": "aws:iam/role:Role", + "name": f"{resource_name}_role", + "id": role["RoleName"], + } + ) + + # Attached policies + try: + policies = iam.list_attached_role_policies(RoleName=role["RoleName"]) + for policy in policies["AttachedPolicies"]: + # Only custom policies (not AWS managed) + if not policy["PolicyArn"].startswith("arn:aws:iam::aws:"): + policy_name = policy["PolicyName"].replace("-", "_") + print(f" - Policy: {policy['PolicyName']}") + + resources.append( + { + "type": "aws:iam/policy:Policy", + "name": f"{policy_name}_policy", + "id": policy["PolicyArn"], + } + ) + + resources.append( + { + "type": "aws:iam/rolePolicyAttachment:RolePolicyAttachment", + "name": f"{resource_name}_{policy_name}_attachment", + "id": f"{role['RoleName']}/{policy['PolicyArn']}", + } + ) + + except Exception as e: + print(f" Warning: Could not check policies: {e}") + + except Exception as e: + print(f"Error discovering IAM roles: {e}", file=sys.stderr) + + return resources + + +def generate_import_json(resources: List[Dict], output_file: str) -> None: + """Generate Pulumi import JSON file. + + Args: + resources: List of resource dictionaries + output_file: Output file path + """ + import_data = {"resources": resources} + + with open(output_file, "w") as f: + json.dump(import_data, f, indent=2) + + print(f"\n✓ Generated import file: {output_file}") + print(f"✓ Found {len(resources)} resources to import") + + +def main(): + """Main entry point.""" + parser = argparse.ArgumentParser( + description="Discover AWS resources for Pulumi import", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Discover all nf-core S3 buckets + python discover-aws-resources.py --region eu-west-1 --resource-type s3 + + # Discover all IAM resources + python discover-aws-resources.py --resource-type iam + + # Discover everything with custom prefix + python discover-aws-resources.py --prefix myproject- --resource-type all + + # Save to custom file + python discover-aws-resources.py --output custom-import.json + """, + ) + + parser.add_argument( + "--region", default="us-east-1", help="AWS region (default: us-east-1)" + ) + + parser.add_argument( + "--prefix", + default="nf-core-", + help="Resource name prefix filter (default: nf-core-)", + ) + + parser.add_argument( + "--resource-type", + choices=["s3", "iam-users", "iam-roles", "iam", "all"], + default="all", + help="Type of resources to discover (default: all)", + ) + + parser.add_argument( + "--output", + default="discovered-resources.json", + help="Output JSON file (default: discovered-resources.json)", + ) + + args = parser.parse_args() + + # Verify AWS credentials + try: + sts = boto3.client("sts", region_name=args.region) + identity = sts.get_caller_identity() + print(f"AWS Account: {identity['Account']}") + print(f"AWS User: {identity['Arn']}\n") + except Exception as e: + print(f"Error: Could not verify AWS credentials: {e}", file=sys.stderr) + sys.exit(1) + + # Discover resources + resources = [] + + if args.resource_type in ["s3", "all"]: + resources.extend(discover_s3_buckets(args.region, args.prefix)) + + if args.resource_type in ["iam-users", "iam", "all"]: + resources.extend(discover_iam_users(args.region, args.prefix)) + + if args.resource_type in ["iam-roles", "iam", "all"]: + resources.extend(discover_iam_roles(args.region, args.prefix)) + + if not resources: + print("\nNo resources found matching criteria.") + sys.exit(0) + + # Generate import file + generate_import_json(resources, args.output) + + # Print next steps + print("\nNext steps:") + print(f" 1. Review {args.output}") + print(f" 2. Run: uv run pulumi import -f {args.output} -o imported.py -y") + print(" 3. Add generated code to your Pulumi program") + print(" 4. Verify: uv run pulumi preview") + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/pulumi/new-project/SKILL.md b/.claude/skills/pulumi/new-project/SKILL.md new file mode 100644 index 00000000..f871bb27 --- /dev/null +++ b/.claude/skills/pulumi/new-project/SKILL.md @@ -0,0 +1,339 @@ +--- +name: Creating New Pulumi Projects +description: Initialize new Pulumi projects with proper structure, dependencies, and 1Password credential management. Use when starting a new infrastructure project, scaffolding Pulumi setup, or creating infrastructure-as-code projects. +--- + +# Creating New Pulumi Projects + +Scaffold new Pulumi projects with proper structure and configuration following nf-core/ops patterns. + +## When to Use + +Use this skill when: + +- Starting a new infrastructure project +- Creating infrastructure-as-code for a service +- Scaffolding Pulumi project structure +- Setting up new environment (AWS, GitHub, etc.) +- Need template for Pulumi project setup + +## Interactive Project Setup + +### 1. Gather Requirements + +Ask the user: + +- **Project name**: What should the project be called? (e.g., `co2_reports`, `megatests_infra`) +- **Cloud provider**: Which provider? (AWS, Azure, GCP, GitHub, etc.) +- **Stack name**: Initial stack name? (e.g., `development`, `AWSMegatests`) +- **AWS region** (if applicable): Which region? (e.g., `us-east-1`, `eu-north-1`) +- **Description**: Brief description of infrastructure purpose +- **Additional providers**: Need GitHub, 1Password, or other providers? + +### 2. Create Project Directory + +```bash +# Navigate to pulumi projects directory +cd ~/src/nf-core/ops/pulumi + +# Create project directory +mkdir {project_name} +cd {project_name} +``` + +### 3. Initialize Pulumi Project + +```bash +# Initialize with Python +pulumi new python --yes --name {project_name} --description "{description}" + +# Or manually create Pulumi.yaml (use template) +``` + +Use template from [templates/Pulumi.yaml](templates/Pulumi.yaml). + +### 4. Set Up Project Files + +Create the following files using templates: + +#### pyproject.toml + +Use [templates/pyproject.toml](templates/pyproject.toml) and customize: + +- Project name +- Description +- Required providers +- Python version + +#### .envrc + +Use [templates/envrc.template](templates/envrc.template) and customize: + +- AWS region (if applicable) +- 1Password vault references +- Additional environment variables + +#### **main**.py + +Use [templates/main_py.template](templates/main_py.template) as starting point: + +- Import required providers +- Add infrastructure resources +- Export useful outputs + +### 5. Install Dependencies + +```bash +# Install with uv +uv sync + +# Or with pip +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +### 6. Configure Credentials + +```bash +# Allow direnv to load .envrc +direnv allow + +# Verify credentials loaded +echo $AWS_ACCESS_KEY_ID +echo $PULUMI_CONFIG_PASSPHRASE +``` + +### 7. Initialize Stack + +```bash +# Select or create stack +uv run pulumi stack init {stack_name} + +# Set required configuration +uv run pulumi config set aws:region {region} + +# Set additional config as needed +uv run pulumi config set {key} {value} +``` + +### 8. Initial Deployment + +```bash +# Preview what will be created +uv run pulumi preview + +# Deploy (after user confirms) +uv run pulumi up +``` + +## Project Structure + +Standard structure for new projects: + +``` +project_name/ +├── .envrc # Environment variables (1Password integration) +├── .python-version # Python version (optional) +├── Pulumi.yaml # Project configuration +├── Pulumi.{stack}.yaml # Stack-specific configuration +├── __main__.py # Infrastructure definition +├── pyproject.toml # Python dependencies +├── uv.lock # Locked dependencies +├── README.md # Project documentation +└── .venv/ # Virtual environment (gitignored) +``` + +## Template Files + +### Minimal pyproject.toml + +```toml +[project] +name = "{project_name}" +version = "0.1.0" +description = "{description}" +readme = "README.md" +requires-python = ">=3.11" + +dependencies = [ + "pulumi>=3.0.0,<4.0.0", + "pulumi-aws>=6.0.0,<7.0.0", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" +``` + +### Minimal .envrc + +```bash +# Environment configuration for {project_name} +export OP_ACCOUNT=nf-core + +# Load 1Password integration +source_url "https://github.com/tmatilai/direnv-1password/raw/v1.0.1/1password.sh" \ + "sha256-4dmKkmlPBNXimznxeehplDfiV+CvJiIzg7H1Pik4oqY=" + +# AWS credentials from 1Password +from_op AWS_ACCESS_KEY_ID="op://Dev/Pulumi-AWS-key/access key id" +from_op AWS_SECRET_ACCESS_KEY="op://Dev/Pulumi-AWS-key/secret access key" + +# AWS Configuration +export AWS_REGION="{region}" +export AWS_DEFAULT_REGION="{region}" + +# Pulumi passphrase +from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password" +``` + +### Minimal **main**.py + +```python +"""Pulumi program for {project_name}.""" + +import pulumi +import pulumi_aws as aws + +# Get configuration +config = pulumi.Config() +aws_config = pulumi.Config("aws") +region = aws_config.require("region") + +# Example: Create an S3 bucket +bucket = aws.s3.Bucket( + "my-bucket", + bucket=f"{pulumi.get_project()}-{pulumi.get_stack()}", + acl="private", + tags={ + "Environment": pulumi.get_stack(), + "Project": pulumi.get_project(), + }, +) + +# Export outputs +pulumi.export("bucket_name", bucket.id) +pulumi.export("bucket_arn", bucket.arn) +``` + +## Common Providers + +### AWS + +```bash +# Install provider +uv add pulumi-aws + +# Configure +uv run pulumi config set aws:region us-east-1 +``` + +### GitHub + +```bash +# Install provider +uv add pulumi-github + +# Configure +uv run pulumi config set github:owner nf-core +uv run pulumi config set github:token --secret +``` + +### 1Password + +```bash +# Install provider +uv add pulumi-onepassword + +# Configure service account token +uv run pulumi config set pulumi-onepassword:service_account_token --secret + +# Or use .envrc: +export OP_SERVICE_ACCOUNT_TOKEN="your-token" +``` + +## Best Practices + +### 1. Use Consistent Naming + +Follow nf-core/ops conventions: + +- Project names: `{service}_{purpose}` (e.g., `co2_reports`, `megatests_infra`) +- Stack names: environment or purpose (e.g., `development`, `AWSMegatests`) +- Resource names: descriptive and prefixed with project/stack + +### 2. Document Everything + +Create comprehensive README.md: + +- What infrastructure is created +- How to deploy +- Required credentials +- Configuration options +- Troubleshooting tips + +### 3. Use .envrc for Credentials + +Always use 1Password + direnv pattern: + +- Never commit credentials +- Consistent credential loading +- Easy to update credentials + +### 4. Tag Resources + +Add tags to all resources: + +```python +tags={ + "Project": pulumi.get_project(), + "Environment": pulumi.get_stack(), + "ManagedBy": "Pulumi", +} +``` + +### 5. Export Useful Outputs + +Export values other stacks or users might need: + +```python +pulumi.export("vpc_id", vpc.id) +pulumi.export("subnet_ids", subnet_ids) +pulumi.export("endpoint_url", endpoint.url) +``` + +## Quick Start Checklist + +When creating a new project: + +- [ ] Create project directory +- [ ] Initialize Pulumi project (`pulumi new python`) +- [ ] Create pyproject.toml with dependencies +- [ ] Create .envrc with 1Password integration +- [ ] Write infrastructure code in **main**.py +- [ ] Install dependencies (`uv sync`) +- [ ] Allow direnv (`direnv allow`) +- [ ] Initialize stack (`pulumi stack init`) +- [ ] Set configuration (`pulumi config set`) +- [ ] Test preview (`pulumi preview`) +- [ ] Deploy (`pulumi up`) +- [ ] Create README.md documentation +- [ ] Commit to git + +## Advanced Patterns + +For more complex setups, see [reference.md](reference.md): + +- Multi-stack projects +- Component resources +- Stack references +- Custom providers +- Testing strategies +- CI/CD integration + +## Related Skills + +- **Deploy**: Preview and deploy infrastructure changes +- **Stack Management**: Manage stacks and configuration +- **Documentation**: Access provider-specific documentation diff --git a/.claude/skills/pulumi/new-project/reference.md b/.claude/skills/pulumi/new-project/reference.md new file mode 100644 index 00000000..8a08d73e --- /dev/null +++ b/.claude/skills/pulumi/new-project/reference.md @@ -0,0 +1,470 @@ +# Pulumi New Project Reference + +Advanced patterns for structuring and organizing Pulumi projects. + +## Table of Contents + +- [Project Organization](#project-organization) +- [Component Resources](#component-resources) +- [Multi-Stack Projects](#multi-stack-projects) +- [Testing Strategies](#testing-strategies) +- [CI/CD Integration](#cicd-integration) +- [Configuration Management](#configuration-management) +- [Provider Configuration](#provider-configuration) + +## Project Organization + +### Single-Stack Projects + +Simple projects with one environment: + +``` +simple_project/ +├── .envrc +├── Pulumi.yaml +├── Pulumi.development.yaml +├── __main__.py +└── pyproject.toml +``` + +### Multi-Stack Projects + +Projects with multiple environments: + +``` +multi_env_project/ +├── .envrc +├── Pulumi.yaml +├── Pulumi.development.yaml +├── Pulumi.staging.yaml +├── Pulumi.production.yaml +├── __main__.py +├── pyproject.toml +└── config/ + ├── development.py + ├── staging.py + └── production.py +``` + +### Component-Based Projects + +Large projects with reusable components: + +``` +large_project/ +├── .envrc +├── Pulumi.yaml +├── Pulumi.production.yaml +├── __main__.py +├── pyproject.toml +├── components/ +│ ├── __init__.py +│ ├── networking.py +│ ├── compute.py +│ └── storage.py +└── utils/ + ├── __init__.py + └── tags.py +``` + +## Component Resources + +Create reusable infrastructure components: + +### Basic Component + +```python +import pulumi +import pulumi_aws as aws +from pulumi import ComponentResource, ResourceOptions + + +class WebServer(ComponentResource): + def __init__(self, name: str, vpc_id: str, subnet_id: str, opts=None): + super().__init__("custom:WebServer", name, {}, opts) + + # Security group + self.security_group = aws.ec2.SecurityGroup( + f"{name}-sg", + vpc_id=vpc_id, + ingress=[ + aws.ec2.SecurityGroupIngressArgs( + protocol="tcp", + from_port=80, + to_port=80, + cidr_blocks=["0.0.0.0/0"], + ), + ], + opts=ResourceOptions(parent=self), + ) + + # EC2 instance + self.instance = aws.ec2.Instance( + f"{name}-instance", + instance_type="t3.micro", + subnet_id=subnet_id, + vpc_security_group_ids=[self.security_group.id], + opts=ResourceOptions(parent=self), + ) + + self.register_outputs({ + "instance_id": self.instance.id, + "public_ip": self.instance.public_ip, + }) + + +# Usage +web_server = WebServer("app", vpc_id=vpc.id, subnet_id=subnet.id) +pulumi.export("web_server_ip", web_server.instance.public_ip) +``` + +### Component with Configuration + +```python +from dataclasses import dataclass +from typing import Optional + + +@dataclass +class WebServerArgs: + vpc_id: str + subnet_id: str + instance_type: str = "t3.micro" + enable_monitoring: bool = False + tags: Optional[dict] = None + + +class WebServer(ComponentResource): + def __init__(self, name: str, args: WebServerArgs, opts=None): + super().__init__("custom:WebServer", name, {}, opts) + + tags = args.tags or {} + + # Create resources with args + self.instance = aws.ec2.Instance( + f"{name}-instance", + instance_type=args.instance_type, + subnet_id=args.subnet_id, + monitoring=args.enable_monitoring, + tags=tags, + opts=ResourceOptions(parent=self), + ) + + self.register_outputs({"instance_id": self.instance.id}) +``` + +## Multi-Stack Projects + +### Environment-Specific Configuration + +Create config files for each environment: + +```python +# config/base.py +BASE_CONFIG = { + "instance_type": "t3.micro", + "enable_monitoring": False, +} + +# config/production.py +from .base import BASE_CONFIG + +PRODUCTION_CONFIG = { + **BASE_CONFIG, + "instance_type": "m5.large", + "enable_monitoring": True, + "backup_retention_days": 30, +} + +# config/development.py +from .base import BASE_CONFIG + +DEVELOPMENT_CONFIG = { + **BASE_CONFIG, + "backup_retention_days": 7, +} +``` + +Use in **main**.py: + +```python +import pulumi +from config.development import DEVELOPMENT_CONFIG +from config.production import PRODUCTION_CONFIG + +stack = pulumi.get_stack() + +# Load config based on stack +if stack == "production": + env_config = PRODUCTION_CONFIG +elif stack == "staging": + env_config = STAGING_CONFIG +else: + env_config = DEVELOPMENT_CONFIG + +# Use config +instance = aws.ec2.Instance( + "app", + instance_type=env_config["instance_type"], + monitoring=env_config["enable_monitoring"], +) +``` + +### Stack-Specific Resources + +```python +import pulumi + +stack = pulumi.get_stack() + +# Only create monitoring in production +if stack == "production": + alarm = aws.cloudwatch.MetricAlarm( + "high-cpu", + comparison_operator="GreaterThanThreshold", + evaluation_periods=2, + metric_name="CPUUtilization", + namespace="AWS/EC2", + period=120, + statistic="Average", + threshold=80, + ) +``` + +## Testing Strategies + +### Unit Tests + +Test individual components: + +```python +# tests/test_webserver.py +import unittest +from unittest.mock import Mock, patch +import pulumi + + +class WebServerTests(unittest.TestCase): + @patch("pulumi.runtime.is_dry_run", return_value=True) + def test_webserver_creation(self, mock_dry_run): + # Test component creation + pass +``` + +### Integration Tests + +Test full deployments: + +```bash +#!/bin/bash +# test_deployment.sh + +set -e + +# Create test stack +pulumi stack init test-${RANDOM} + +# Deploy +pulumi up --yes + +# Verify outputs +BUCKET_NAME=$(pulumi stack output bucket_name) +aws s3 ls s3://${BUCKET_NAME} + +# Cleanup +pulumi destroy --yes +pulumi stack rm --yes +``` + +### Preview Tests + +Test that preview succeeds: + +```bash +# In CI/CD +uv run pulumi preview --expect-no-changes +``` + +## CI/CD Integration + +### GitHub Actions Workflow + +```yaml +name: Deploy Infrastructure + +on: + push: + branches: [main] + paths: + - "**.py" + - "Pulumi*.yaml" + - "pyproject.toml" + +jobs: + preview: + runs-on: ubuntu-latest + if: github.event_name == 'pull_request' + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install dependencies + run: | + pip install uv + uv sync + + - name: Preview changes + run: uv run pulumi preview --stack development + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }} + + deploy: + runs-on: ubuntu-latest + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install dependencies + run: | + pip install uv + uv sync + + - name: Deploy to production + run: uv run pulumi up --yes --stack production + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }} +``` + +### Pre-commit Hooks + +```yaml +# .pre-commit-config.yaml +repos: + - repo: local + hooks: + - id: pulumi-preview + name: Pulumi Preview + entry: uv run pulumi preview + language: system + pass_filenames: false + always_run: false + files: \.(py|yaml)$ +``` + +## Configuration Management + +### Hierarchical Configuration + +Load config from multiple sources: + +```python +import os +import pulumi +from pathlib import Path + +# 1. Default values +config = { + "instance_type": "t3.micro", + "region": "us-east-1", +} + +# 2. Pulumi config (stack-specific) +pulumi_config = pulumi.Config() +config["instance_type"] = pulumi_config.get("instance_type") or config["instance_type"] + +# 3. Environment variables +config["region"] = os.getenv("AWS_REGION", config["region"]) + +# 4. File-based config +config_file = Path("config.json") +if config_file.exists(): + import json + file_config = json.loads(config_file.read_text()) + config.update(file_config) +``` + +### Secret Management + +Best practices for secrets: + +```python +import pulumi + +config = pulumi.Config() + +# Use Pulumi secrets +database_password = config.require_secret("database_password") + +# Or load from 1Password via environment +import os +api_key = os.getenv("API_KEY") # Loaded by direnv from 1Password + +# Never hardcode secrets +# BAD: database_password = "hardcoded123" +``` + +## Provider Configuration + +### Multiple Provider Instances + +Use multiple AWS accounts or regions: + +```python +import pulumi_aws as aws + +# Default provider (us-east-1) +bucket_east = aws.s3.Bucket("bucket-east") + +# Secondary provider (eu-west-1) +provider_eu = aws.Provider("eu-west-1", + region="eu-west-1" +) + +bucket_eu = aws.s3.Bucket("bucket-eu", + opts=pulumi.ResourceOptions(provider=provider_eu) +) +``` + +### Assume Role + +```python +import pulumi_aws as aws + +# Provider with assumed role +provider_prod = aws.Provider("production", + region="us-east-1", + assume_role=aws.ProviderAssumeRoleArgs( + role_arn="arn:aws:iam::123456789:role/PulumiDeploy", + session_name="pulumi-deploy", + ), +) + +# Use provider +bucket = aws.s3.Bucket("prod-bucket", + opts=pulumi.ResourceOptions(provider=provider_prod) +) +``` + +## Best Practices + +1. **Component Resources**: Use for reusable infrastructure patterns +2. **Configuration Files**: Separate config from code +3. **Testing**: Test infrastructure code like application code +4. **CI/CD**: Automate deployments and previews +5. **Secrets**: Never commit secrets, use encryption or external stores +6. **Documentation**: Document components and configurations +7. **Tags**: Tag all resources consistently +8. **Outputs**: Export useful information for other stacks or users +9. **Dependencies**: Explicit dependencies improve reliability +10. **Monitoring**: Add observability from the start diff --git a/.claude/skills/pulumi/new-project/templates/Pulumi.yaml b/.claude/skills/pulumi/new-project/templates/Pulumi.yaml new file mode 100644 index 00000000..9ae698b2 --- /dev/null +++ b/.claude/skills/pulumi/new-project/templates/Pulumi.yaml @@ -0,0 +1,6 @@ +name: { { PROJECT_NAME } } +description: { { DESCRIPTION } } +runtime: python + +backend: + url: s3://nf-core-ops-pulumi-state?region=us-east-1&awssdk=v2 diff --git a/.claude/skills/pulumi/new-project/templates/envrc.template b/.claude/skills/pulumi/new-project/templates/envrc.template new file mode 100644 index 00000000..0edf41bb --- /dev/null +++ b/.claude/skills/pulumi/new-project/templates/envrc.template @@ -0,0 +1,19 @@ +# Environment configuration for {{PROJECT_NAME}} +# This file loads AWS credentials from 1Password + +export OP_ACCOUNT=nf-core + +# Load 1Password integration for direnv +source_url "https://github.com/tmatilai/direnv-1password/raw/v1.0.1/1password.sh" \ + "sha256-4dmKkmlPBNXimznxeehplDfiV+CvJiIzg7H1Pik4oqY=" + +# Load AWS credentials from 1Password +from_op AWS_ACCESS_KEY_ID="op://Dev/Pulumi-AWS-key/access key id" +from_op AWS_SECRET_ACCESS_KEY="op://Dev/Pulumi-AWS-key/secret access key" + +# AWS Configuration +export AWS_REGION="{{AWS_REGION}}" +export AWS_DEFAULT_REGION="{{AWS_REGION}}" + +# Load Pulumi passphrase from 1Password +from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password" diff --git a/.claude/skills/pulumi/new-project/templates/main_py.template b/.claude/skills/pulumi/new-project/templates/main_py.template new file mode 100644 index 00000000..4b966f5a --- /dev/null +++ b/.claude/skills/pulumi/new-project/templates/main_py.template @@ -0,0 +1,60 @@ +"""Pulumi program for {{PROJECT_NAME}}. + +{{DESCRIPTION}} +""" + +import pulumi +import pulumi_aws as aws + +# Get configuration +config = pulumi.Config() +aws_config = pulumi.Config("aws") +region = aws_config.require("region") +stack = pulumi.get_stack() +project = pulumi.get_project() + +# Common tags for all resources +common_tags = { + "Project": project, + "Environment": stack, + "ManagedBy": "Pulumi", +} + +# Example infrastructure - replace with your actual resources +# ============================================================ + +# Example: S3 Bucket +bucket = aws.s3.Bucket( + "main-bucket", + bucket=f"{project}-{stack}", + acl="private", + versioning=aws.s3.BucketVersioningArgs( + enabled=True, + ), + server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs( + rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs( + apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs( + sse_algorithm="AES256", + ), + ), + ), + tags=common_tags, +) + +# Block public access to bucket +bucket_public_access_block = aws.s3.BucketPublicAccessBlock( + "bucket-public-access-block", + bucket=bucket.id, + block_public_acls=True, + block_public_policy=True, + ignore_public_acls=True, + restrict_public_buckets=True, +) + +# Export useful outputs +# ====================== + +pulumi.export("bucket_name", bucket.id) +pulumi.export("bucket_arn", bucket.arn) +pulumi.export("region", region) +pulumi.export("stack", stack) diff --git a/.claude/skills/pulumi/new-project/templates/pyproject.toml b/.claude/skills/pulumi/new-project/templates/pyproject.toml new file mode 100644 index 00000000..51e1859a --- /dev/null +++ b/.claude/skills/pulumi/new-project/templates/pyproject.toml @@ -0,0 +1,33 @@ +[project] +name = "{{PROJECT_NAME}}" +version = "0.1.0" +description = "{{DESCRIPTION}}" +readme = "README.md" +requires-python = ">=3.11" + +dependencies = [ + "pulumi>=3.0.0,<4.0.0", + # Add provider dependencies as needed: + # "pulumi-aws>=6.0.0,<7.0.0", + # "pulumi-github>=6.0.0,<7.0.0", + # "pulumi-onepassword>=0.7.0,<1.0.0", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.uv] +dev-dependencies = [] + +[tool.ruff] +line-length = 100 +target-version = "py311" + +[tool.ruff.lint] +select = ["E", "F", "I", "N", "W"] +ignore = [] + +[tool.ruff.format] +quote-style = "double" +indent-style = "space" diff --git a/.claude/skills/pulumi/stack-management/SKILL.md b/.claude/skills/pulumi/stack-management/SKILL.md new file mode 100644 index 00000000..58af7724 --- /dev/null +++ b/.claude/skills/pulumi/stack-management/SKILL.md @@ -0,0 +1,233 @@ +--- +name: Managing Pulumi Stacks +description: Manage Pulumi stacks, view outputs, and configure stack settings. Use when switching stacks, viewing stack outputs, managing stack configuration, or working with multiple environments (dev, staging, production). +--- + +# Managing Pulumi Stacks + +Manage Pulumi stacks for different environments and view stack state and outputs. + +## When to Use + +Use this skill when: + +- Switching between stacks (dev, staging, production) +- Viewing stack outputs +- Managing stack configuration +- Listing available stacks +- Checking current stack state +- Working with multi-environment setups + +## Common Stack Operations + +### List Available Stacks + +```bash +# List all stacks in current project +uv run pulumi stack ls + +# Show current stack (marked with *) +uv run pulumi stack --show-name +``` + +### Switch Stacks + +```bash +# Switch to a different stack +uv run pulumi stack select + +# Examples: +uv run pulumi stack select development +uv run pulumi stack select production +uv run pulumi stack select AWSMegatests +``` + +### View Stack Outputs + +```bash +# Show all stack outputs +uv run pulumi stack output + +# Get specific output value +uv run pulumi stack output bucket_name + +# Get output as JSON +uv run pulumi stack output --json + +# Show secrets (requires passphrase) +uv run pulumi stack output --show-secrets +``` + +### View Stack Information + +```bash +# Show detailed stack info +uv run pulumi stack + +# Show stack with resource URNs +uv run pulumi stack --show-urns + +# Show stack resources +uv run pulumi stack --show-ids +``` + +## Configuration Management + +### View Configuration + +```bash +# Show all config for current stack +uv run pulumi config + +# Get specific config value +uv run pulumi config get aws:region + +# Get secret config value +uv run pulumi config get database_password --show-secrets +``` + +### Set Configuration + +```bash +# Set regular config value +uv run pulumi config set aws:region us-east-1 + +# Set secret config value (encrypted) +uv run pulumi config set database_password mypassword --secret + +# Set config from file +uv run pulumi config set-all --plaintext < config.json +``` + +### Remove Configuration + +```bash +# Remove config key +uv run pulumi config rm old_setting +``` + +## Stack Initialization + +### Create New Stack + +```bash +# Create and switch to new stack +uv run pulumi stack init + +# Examples: +uv run pulumi stack init development +uv run pulumi stack init staging +``` + +### Remove Stack + +**⚠️ Destructive operation - always confirm with user first!** + +```bash +# Remove stack (must be empty) +uv run pulumi stack rm + +# Force remove (dangerous!) +uv run pulumi stack rm --force +``` + +**Before removing:** + +1. Confirm with user this is intentional +2. Export stack state for backup +3. Run `pulumi destroy` first to remove resources +4. Verify stack is empty with `pulumi stack` + +## Quick Reference + +| Task | Command | +| ------------ | ---------------------------------------- | +| List stacks | `uv run pulumi stack ls` | +| Switch stack | `uv run pulumi stack select ` | +| Create stack | `uv run pulumi stack init ` | +| View outputs | `uv run pulumi stack output` | +| View config | `uv run pulumi config` | +| Set config | `uv run pulumi config set ` | +| Stack info | `uv run pulumi stack` | + +## Multi-Environment Workflow + +### Standard Stack Naming + +``` +development # For local development and testing +staging # Pre-production testing +production # Live production environment +``` + +### Environment-Specific Config + +Each stack can have different configuration: + +```bash +# Development stack +uv run pulumi stack select development +uv run pulumi config set instance_type t3.micro +uv run pulumi config set enable_monitoring false + +# Production stack +uv run pulumi stack select production +uv run pulumi config set instance_type m5.large +uv run pulumi config set enable_monitoring true +``` + +### Accessing Config in Code + +```python +import pulumi + +config = pulumi.Config() +instance_type = config.get("instance_type") or "t3.small" +enable_monitoring = config.get_bool("enable_monitoring") or False +``` + +## Troubleshooting + +Common issues and solutions: + +### Stack Already Exists + +``` +error: stack 'production' already exists +``` + +**Solution**: Use `pulumi stack select production` to switch to it. + +### Stack Not Found + +``` +error: no stack named 'staging' found +``` + +**Solution**: Create it with `pulumi stack init staging`. + +### Config Key Not Found + +``` +error: configuration key 'aws:region' not found +``` + +**Solution**: Set it with `pulumi config set aws:region us-east-1`. + +For more troubleshooting, see [troubleshooting.md](troubleshooting.md). + +## Advanced Patterns + +For complex stack management scenarios, see [reference.md](reference.md): + +- Stack tagging and organization +- Cross-stack references +- Stack exports and imports +- Stack migrations +- Organization-level stack management + +## Related Skills + +- **Deploy**: Preview and deploy infrastructure changes +- **New Project**: Initialize new Pulumi projects with proper structure +- **Documentation**: Access Pulumi provider documentation diff --git a/.claude/skills/pulumi/stack-management/reference.md b/.claude/skills/pulumi/stack-management/reference.md new file mode 100644 index 00000000..ffece66f --- /dev/null +++ b/.claude/skills/pulumi/stack-management/reference.md @@ -0,0 +1,412 @@ +# Pulumi Stack Management Reference + +Advanced stack management patterns and techniques. + +## Table of Contents + +- [Stack Organization](#stack-organization) +- [Cross-Stack References](#cross-stack-references) +- [Stack Tags](#stack-tags) +- [Stack Exports and Imports](#stack-exports-and-imports) +- [Stack Migrations](#stack-migrations) +- [Organization Management](#organization-management) +- [Configuration Strategies](#configuration-strategies) +- [Secret Management](#secret-management) + +## Stack Organization + +### Naming Conventions + +**Environment-based:** + +``` +myproject-dev +myproject-staging +myproject-production +``` + +**Region-based:** + +``` +myproject-us-east-1 +myproject-eu-west-1 +myproject-ap-southeast-1 +``` + +**Feature-based:** + +``` +myproject-networking +myproject-compute +myproject-data +``` + +**Combined:** + +``` +myproject-production-us-east-1 +myproject-staging-eu-west-1 +``` + +### Stack Hierarchy + +For complex infrastructures: + +``` +organization/ +├── shared/ +│ ├── networking-production +│ ├── networking-staging +│ └── security +├── product-a/ +│ ├── app-production +│ └── app-staging +└── product-b/ + ├── api-production + └── api-staging +``` + +## Cross-Stack References + +Share outputs between stacks: + +### Export from Source Stack + +```python +# In networking stack +import pulumi + +vpc = aws.ec2.Vpc("vpc", cidr_block="10.0.0.0/16") + +# Export for other stacks to use +pulumi.export("vpc_id", vpc.id) +pulumi.export("public_subnet_ids", [s.id for s in public_subnets]) +pulumi.export("private_subnet_ids", [s.id for s in private_subnets]) +``` + +### Import in Target Stack + +```python +# In application stack +import pulumi + +# Reference networking stack +networking_stack = pulumi.StackReference("organization/networking-stack/production") + +# Get outputs +vpc_id = networking_stack.get_output("vpc_id") +public_subnet_ids = networking_stack.get_output("public_subnet_ids") + +# Use in resources +instance = aws.ec2.Instance("app-server", + instance_type="t3.medium", + vpc_security_group_ids=[sg.id], + subnet_id=public_subnet_ids[0] +) +``` + +### Stack Reference Format + +``` +// +``` + +Examples: + +```python +pulumi.StackReference("acme-corp/networking/production") +pulumi.StackReference("my-org/databases/staging") +pulumi.StackReference("shared/security/global") +``` + +## Stack Tags + +Organize and filter stacks with tags: + +###Add Tags to Stack + +```bash +# Tag stack with environment +uv run pulumi stack tag set environment production + +# Tag with team ownership +uv run pulumi stack tag set team platform-team + +# Tag with cost center +uv run pulumi stack tag set cost-center engineering +``` + +### View Stack Tags + +```bash +# List all tags for current stack +uv run pulumi stack tag ls + +# Get specific tag value +uv run pulumi stack tag get environment +``` + +### Remove Tags + +```bash +# Remove a tag +uv run pulumi stack tag rm old-tag +``` + +### Using Tags + +Tags help with: + +- **Cost allocation**: Track spending by team or project +- **Automation**: Filter stacks in scripts +- **Organization**: Group related stacks +- **Compliance**: Track compliance requirements + +## Stack Exports and Imports + +### Export Stack State + +```bash +# Export stack to JSON +uv run pulumi stack export > stack-backup.json + +# Export with secrets decrypted +uv run pulumi stack export --show-secrets > stack-full-backup.json + +# Export specific version +uv run pulumi stack export --version 42 > stack-v42.json +``` + +### Import Stack State + +```bash +# Import stack state +uv run pulumi stack import --file stack-backup.json +``` + +### Use Cases + +**Backup before major changes:** + +```bash +uv run pulumi stack export > backup-$(date +%Y%m%d-%H%M%S).json +uv run pulumi up +``` + +**Migration between backends:** + +```bash +# Export from old backend +uv run pulumi stack export > migration.json + +# Switch backend +export PULUMI_BACKEND_URL="s3://new-backend" + +# Import to new backend +uv run pulumi stack import --file migration.json +``` + +## Stack Migrations + +### Rename Stack + +```bash +# Export current state +uv run pulumi stack export > old-stack.json + +# Create new stack +uv run pulumi stack init new-name + +# Import state +uv run pulumi stack import --file old-stack.json + +# Verify +uv run pulumi preview # Should show no changes + +# Remove old stack +uv run pulumi stack select old-name +uv run pulumi stack rm old-name --force +``` + +### Merge Stacks + +Combine multiple stacks: + +1. **Export both stacks** +2. **Manually merge JSON** (careful with dependencies) +3. **Import merged state** +4. **Run preview** to verify +5. **Remove old stacks** + +**Note**: Complex operation - test thoroughly in development first! + +### Split Stack + +Separate monolithic stack: + +1. **Create new stacks** for each component +2. **Move resources** to new code +3. **Use stack references** for dependencies +4. **Deploy new stacks** +5. **Remove resources** from old stack +6. **Destroy old stack** + +## Organization Management + +### Organization-Level Operations + +```bash +# List all stacks in organization +pulumi stack ls --organization my-org + +# Select stack with org prefix +uv run pulumi stack select my-org/project/stack +``` + +### Team Collaboration + +**Access Control:** + +- Use Pulumi Cloud for team access management +- Define roles: Admin, Editor, Viewer +- Audit access regularly + +**CI/CD Integration:** + +- Use service accounts for automation +- Rotate credentials regularly +- Limit permissions to minimum required + +## Configuration Strategies + +### Hierarchical Configuration + +**Project-level config** (`Pulumi.yaml`): + +```yaml +name: myproject +runtime: python +description: My infrastructure project +``` + +**Stack-level config** (`Pulumi..yaml`): + +```yaml +config: + aws:region: us-east-1 + myproject:instance_type: t3.medium + myproject:enable_monitoring: "true" +``` + +**Environment variables:** + +```bash +export PULUMI_CONFIG_PASSPHRASE="secret" +export AWS_REGION="us-east-1" +``` + +### Configuration Patterns + +**Shared defaults:** + +```python +# config.py +import pulumi + +config = pulumi.Config() + +# Defaults +INSTANCE_TYPE = config.get("instance_type") or "t3.small" +REGION = config.require("aws:region") +MONITORING = config.get_bool("enable_monitoring") or False +``` + +**Environment-specific overrides:** + +```python +import pulumi + +stack = pulumi.get_stack() + +if stack == "production": + instance_type = "m5.large" + enable_monitoring = True +elif stack == "staging": + instance_type = "t3.medium" + enable_monitoring = True +else: # development + instance_type = "t3.micro" + enable_monitoring = False +``` + +## Secret Management + +### Using Pulumi Secrets + +```bash +# Set secret value +uv run pulumi config set database_password mypassword --secret + +# Set secret from file +cat password.txt | uv run pulumi config set database_password --secret + +# View secret (requires passphrase) +uv run pulumi config get database_password --show-secrets +``` + +### Using 1Password + +Via .envrc and direnv: + +```bash +# .envrc +from_op DATABASE_PASSWORD="op://Dev/Database/password" +from_op API_KEY="op://Dev/API-Keys/key" +``` + +Then in Pulumi code: + +```python +import os +import pulumi + +# Get from environment (loaded by direnv from 1Password) +database_password = os.getenv("DATABASE_PASSWORD") + +# Or use Pulumi config +config = pulumi.Config() +api_key = config.get("api_key") or os.getenv("API_KEY") +``` + +### Secret Rotation + +```bash +# Update secret +uv run pulumi config set database_password new-password --secret + +# Deploy with new secret +uv run pulumi up +``` + +### Secrets in Stack Exports + +```bash +# Export without secrets (encrypted) +uv run pulumi stack export > backup.json + +# Export with decrypted secrets (keep secure!) +uv run pulumi stack export --show-secrets > backup-with-secrets.json +``` + +## Best Practices + +1. **Use consistent naming**: Follow naming conventions across all stacks +2. **Tag everything**: Add tags for organization and cost tracking +3. **Document dependencies**: Use stack references for clear relationships +4. **Regular exports**: Backup stack state before major changes +5. **Limit stack scope**: Don't make stacks too large +6. **Separate environments**: Use different stacks for dev/staging/prod +7. **Automate**: Use CI/CD for stack management +8. **Monitor**: Track stack health and resource counts +9. **Clean up**: Remove unused stacks regularly +10. **Secure secrets**: Use encryption and access controls diff --git a/.claude/skills/pulumi/stack-management/troubleshooting.md b/.claude/skills/pulumi/stack-management/troubleshooting.md new file mode 100644 index 00000000..ed5a569c --- /dev/null +++ b/.claude/skills/pulumi/stack-management/troubleshooting.md @@ -0,0 +1,564 @@ +# Pulumi Troubleshooting Guide + +Common issues and their solutions when working with Pulumi. + +## Table of Contents + +- [Authentication Issues](#authentication-issues) +- [State Backend Issues](#state-backend-issues) +- [Deployment Failures](#deployment-failures) +- [Configuration Problems](#configuration-problems) +- [Resource Issues](#resource-issues) +- [Performance Problems](#performance-problems) + +## Authentication Issues + +### AWS Credentials Not Found + +**Error:** + +``` +error: get credentials: failed to refresh cached credentials, no EC2 IMDS role found +error: operation error S3: GetObject, get identity: get credentials +``` + +**Solutions:** + +1. **Check environment variables:** + +```bash +echo $AWS_ACCESS_KEY_ID +echo $AWS_SECRET_ACCESS_KEY +echo $AWS_REGION +``` + +2. **If using direnv + 1Password:** + +```bash +# Check .envrc exists and is allowed +direnv allow + +# Verify credentials loaded +eval "$(direnv export bash)" +echo $AWS_ACCESS_KEY_ID +``` + +3. **Manually set credentials:** + +```bash +export AWS_ACCESS_KEY_ID="your-key" +export AWS_SECRET_ACCESS_KEY="your-secret" +export AWS_REGION="us-east-1" +``` + +### Pulumi Passphrase Missing + +**Error:** + +``` +error: failed to decrypt encrypted configuration value 'aws:secretKey': incorrect passphrase +``` + +**Solutions:** + +1. **Set passphrase:** + +```bash +export PULUMI_CONFIG_PASSPHRASE="your-passphrase" +``` + +2. **If using 1Password:** + +```bash +# Add to .envrc +from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password" +direnv allow +``` + +3. **Reset passphrase (if lost):** + +```bash +# Export stack without secrets +uv run pulumi stack export > stack-backup.json + +# Create new stack with new passphrase +uv run pulumi stack init new-stack + +# Import (you'll lose encrypted secrets) +uv run pulumi stack import --file stack-backup.json +``` + +### 1Password Not Responding + +**Error:** + +``` +direnv: error /path/to/.envrc is blocked. Run `direnv allow` to approve its content +``` + +**Solutions:** + +1. **Allow direnv:** + +```bash +direnv allow +``` + +2. **Check 1Password CLI:** + +```bash +# Test 1Password access +op whoami + +# If not signed in +eval $(op signin) +``` + +3. **Check service account token:** + +```bash +echo $OP_SERVICE_ACCOUNT_TOKEN +``` + +## State Backend Issues + +### Cannot Access S3 Backend + +**Error:** + +``` +error: read ".pulumi/meta.yaml": operation error S3: GetObject +``` + +**Solutions:** + +1. **Check AWS credentials** (see above) + +2. **Verify S3 bucket exists:** + +```bash +aws s3 ls s3://your-pulumi-state-bucket/ +``` + +3. **Check bucket permissions:** + +```bash +aws s3api get-bucket-policy --bucket your-pulumi-state-bucket +``` + +4. **Switch to local backend temporarily:** + +```bash +export PULUMI_BACKEND_URL="file://~/.pulumi" +uv run pulumi login file://~/.pulumi +``` + +### Stack Already Exists + +**Error:** + +``` +error: stack 'production' already exists +``` + +**Solution:** + +Switch to existing stack instead of creating: + +```bash +uv run pulumi stack select production +``` + +### Stack Not Found + +**Error:** + +``` +error: no stack named 'staging' found +``` + +**Solutions:** + +1. **List available stacks:** + +```bash +uv run pulumi stack ls +``` + +2. **Create the stack if needed:** + +```bash +uv run pulumi stack init staging +``` + +3. **Check you're in correct project:** + +```bash +cat Pulumi.yaml # Verify project name +``` + +## Deployment Failures + +### Resource Already Exists + +**Error:** + +``` +error: resource 'my-bucket' already exists +``` + +**Solutions:** + +1. **Import existing resource:** + +```bash +uv run pulumi import aws:s3/bucket:Bucket my-bucket existing-bucket-name +``` + +2. **Use different name:** + +```python +# Add unique suffix +bucket = aws.s3.Bucket(f"my-bucket-{pulumi.get_stack()}") +``` + +3. **Delete existing resource** (if safe): + +```bash +aws s3 rb s3://existing-bucket-name --force +``` + +### Insufficient Permissions + +**Error:** + +``` +error: AccessDenied: User is not authorized to perform: s3:PutObject +``` + +**Solutions:** + +1. **Check IAM permissions:** + +```bash +aws iam get-user +aws iam list-attached-user-policies --user-name your-user +``` + +2. **Add required permissions** to IAM policy + +3. **Use different credentials** with proper permissions + +### Resource Update Failed + +**Error:** + +``` +error: update failed: resource changes are not allowed +``` + +**Solutions:** + +1. **Use replacement instead:** + +```bash +uv run pulumi up --replace urn:pulumi:stack::project::Type::resource +``` + +2. **Check for protection:** + +```python +# Remove protect flag if set +resource = Resource("name", + opts=ResourceOptions(protect=False) +) +``` + +3. **Manual intervention required** - update resource manually, then refresh: + +```bash +uv run pulumi refresh +``` + +## Configuration Problems + +### Config Key Not Found + +**Error:** + +``` +error: configuration key 'aws:region' not found +``` + +**Solution:** + +Set the required config: + +```bash +uv run pulumi config set aws:region us-east-1 +``` + +### Secret Cannot Be Decrypted + +**Error:** + +``` +error: failed to decrypt encrypted configuration value +``` + +**Solutions:** + +1. **Set correct passphrase:** + +```bash +export PULUMI_CONFIG_PASSPHRASE="correct-passphrase" +``` + +2. **Re-encrypt with new passphrase:** + +```bash +# Export config without secrets +uv run pulumi config --show-secrets > config.txt + +# Change passphrase +export PULUMI_CONFIG_PASSPHRASE="new-passphrase" + +# Re-import configs as secrets +uv run pulumi config set key value --secret +``` + +### Wrong Stack Selected + +**Problem:** Making changes to wrong environment + +**Solution:** + +Always verify current stack before deploying: + +```bash +# Check current stack +uv run pulumi stack --show-name + +# Should output: production + +# If wrong, switch +uv run pulumi stack select production +``` + +**Best Practice:** Add stack name to shell prompt: + +```bash +# Add to .bashrc or .zshrc +export PS1="[$(pulumi stack --show-name 2>/dev/null || echo 'no-stack')] $PS1" +``` + +## Resource Issues + +### Dependency Cycle Detected + +**Error:** + +``` +error: cycle: a -> b -> c -> a +``` + +**Solution:** + +Break the cycle using explicit dependencies: + +```python +# Instead of circular dependencies +a = Resource("a", dependency=b.output) +b = Resource("b", dependency=c.output) +c = Resource("c", dependency=a.output) + +# Use explicit depends_on +a = Resource("a", opts=ResourceOptions(depends_on=[b])) +b = Resource("b", opts=ResourceOptions(depends_on=[c])) +c = Resource("c") # No dependency on a +``` + +### Resource Still in Use + +**Error:** + +``` +error: resource still has dependencies +``` + +**Solutions:** + +1. **Delete dependent resources first** +2. **Use targeted destroy:** + +```bash +uv run pulumi destroy --target dependent-resource +uv run pulumi destroy --target main-resource +``` + +3. **Force delete** (dangerous): + +```bash +uv run pulumi state delete urn:pulumi:stack::project::Type::resource --force +``` + +### Timeout Waiting for Resource + +**Error:** + +``` +error: timeout while waiting for resource to reach running state +``` + +**Solutions:** + +1. **Increase timeout:** + +```python +resource = Resource("name", + opts=ResourceOptions(custom_timeouts=CustomTimeouts( + create="30m", + update="20m", + delete="10m" + )) +) +``` + +2. **Check resource manually:** + +```bash +# For AWS resources +aws ec2 describe-instances --instance-ids i-xxxxx +``` + +3. **Cancel and retry:** + +```bash +uv run pulumi cancel +uv run pulumi up +``` + +## Performance Problems + +### Slow Deployments + +**Symptoms:** Deployments taking excessively long + +**Solutions:** + +1. **Increase parallelism:** + +```bash +uv run pulumi up --parallel 20 +``` + +2. **Split large stacks** into smaller ones + +3. **Use refresh less frequently:** + +```bash +# Skip refresh on preview +uv run pulumi preview --skip-refresh +``` + +4. **Enable performance logging:** + +```bash +export PULUMI_DEBUG_PROMISE_LEAKS=true +uv run pulumi up +``` + +### Large State File + +**Symptoms:** Slow operations, large .pulumi/ directory + +**Solutions:** + +1. **Clean up deleted resources:** + +```bash +uv run pulumi state delete urn:pulumi:stack::project::Type::old-resource --yes +``` + +2. **Split into multiple stacks** + +3. **Export and re-import to compact:** + +```bash +uv run pulumi stack export > stack.json +uv run pulumi stack rm --force +uv run pulumi stack init +uv run pulumi stack import --file stack.json +``` + +### Memory Issues + +**Error:** + +``` +JavaScript heap out of memory +``` + +**Solutions:** + +1. **Increase Node.js memory:** + +```bash +export NODE_OPTIONS="--max-old-space-size=4096" +``` + +2. **Reduce parallelism:** + +```bash +uv run pulumi up --parallel 5 +``` + +## Getting More Help + +### Enable Verbose Logging + +```bash +# Maximum verbosity +uv run pulumi up --logtostderr -v=9 + +# Save to file +uv run pulumi up --logtostderr -v=9 2>&1 | tee pulumi-debug.log +``` + +### Check Pulumi Version + +```bash +pulumi version +``` + +Update if needed: + +```bash +brew upgrade pulumi # macOS +# or +curl -fsSL https://get.pulumi.com | sh +``` + +### Community Resources + +- **Pulumi Slack**: https://slack.pulumi.com +- **GitHub Issues**: https://github.com/pulumi/pulumi/issues +- **Documentation**: https://www.pulumi.com/docs/ +- **Examples**: https://github.com/pulumi/examples + +### Creating Bug Reports + +When reporting issues: + +1. **Pulumi version**: `pulumi version` +2. **Minimal reproduction case** +3. **Full error output** with `-v=9` +4. **Stack export** (without secrets) +5. **Provider versions** + +```bash +# Gather debug info +pulumi version > debug-info.txt +uv run pulumi stack export >> debug-info.txt +uv run pulumi up --logtostderr -v=9 2>&1 | tee pulumi-error.log +```