Skip to content

feat: add ResourceOptimization workspace with conditional compute environment deployment#185

Open
edmundmiller wants to merge 5 commits into
mainfrom
feat/resource-optimization-workspace
Open

feat: add ResourceOptimization workspace with conditional compute environment deployment#185
edmundmiller wants to merge 5 commits into
mainfrom
feat/resource-optimization-workspace

Conversation

@edmundmiller
Copy link
Copy Markdown
Contributor

@edmundmiller edmundmiller commented Oct 22, 2025 •
edited
Loading

Summary

Adds support for the ResourceOptimization Seqera Platform workspace with conditional compute environment deployment capabilities for the multi-workspace architecture.

Changes

Core Infrastructure Improvements

  • Conditional Compute Environment Deployment: Workspaces can now selectively enable/disable CPU, GPU, and ARM compute environments via workspace_config.py
  • Workspace-Specific IAM Resources: IAM users and policies now include workspace name suffix to prevent conflicts between workspaces
  • Flexible S3 Infrastructure: Support both importing existing buckets and creating new ones

ResourceOptimization Workspace

Successfully deployed new workspace for Florian's resource optimization testing:

  • Workspace ID: 80750985193419
  • Compute Environments: CPU only (GPU and ARM disabled)
  • S3 Bucket: nf-core-resource-optimization
  • IAM Resources: TowerForge-AWSMegatests-ResourceOptimization (user + 3 policies)
  • Deployment Time: 1m2s (13 resources created)

Technical Details

  1. Updated compute_environments.py:

    • Added workspace_config parameter to deploy_seqera_environments_terraform()
    • Conditional deployment based on workspace_config["compute_environments"][env_type]["enabled"]
    • Updated get_compute_environment_ids_terraform() to handle missing environments gracefully

  2. Updated credentials.py:

    • Added workspace_name parameter to create_towerforge_credentials()
    • Workspace suffix applied to IAM user and policy names
    • Prevents conflicts when multiple workspaces exist

  3. Updated s3.py:

    • Added bucket_name and import_existing parameters
    • Supports creating new buckets for new workspaces
    • Maintains backward compatibility for existing AWSMegatests bucket import

  4. Updated constants.py:

    • Changed config file paths to absolute paths
    • Uses Path(__file__).parent.parent for shared directory reference

  5. Added Seqera SDK:

    • Copied pulumi_seqera v0.25.2 to both workspaces' sdks/ directories
    • Removed from pyproject.toml dependencies (not on PyPI)
    • Loaded via sys.path manipulation in __main__.py

  6. Documentation:

    • Added manual setup commands to pulumi_state/README.md
    • Documented 1Password credential retrieval
    • Included environment variable requirements and import commands

Deployment Output

compute_env_ids:
  cpu: teiXlOfccCoaPubY9sp6a

s3_bucket:
  arn: arn:aws:s3:::nf-core-resource-optimization
  name: nf-core-resource-optimization
  region: eu-west-1

workspace:
  name: ResourceOptimization
  organization: nf-core
  workspace_id: "80750985193419"

Testing

  • ✅ Pulumi preview successful (17 resources planned)
  • ✅ Deployment successful (13 resources created in 1m2s)
  • ✅ CPU compute environment created and visible in Seqera Platform
  • ✅ GPU and ARM environments correctly skipped
  • ✅ IAM resources created with workspace-specific names (no conflicts)
  • ✅ S3 bucket created successfully
  • ✅ Credentials uploaded to Seqera Platform

Related

edmundmiller and others added 5 commits October 21, 2025 16:22
@edmundmiller @claude
feat: add multi-workspace Seqera Platform structure
1f25214 
Create new pulumi/seqera_platform directory with support for multiple
independent Seqera Platform workspaces. This enables managing separate
workspaces with different configurations while sharing common code.

Structure:
- shared/: Reusable modules for providers, infrastructure, and integrations
- awsmegatests/: Full workspace with CPU, GPU, and ARM compute environments
- resource_optimization/: Focused workspace with CPU-only for resource testing

Each workspace is an independent Pulumi project with workspace-specific
configuration defined in workspace_config.py. The shared modules are
imported via sys.path manipulation to avoid code duplication.

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@edmundmiller @claude
feat: add Pulumi state storage bootstrap project
69e4191 
Create S3-backed state storage for all Pulumi projects. This bootstrap
project creates and manages the nf-core-pulumi-state bucket in eu-north-1.

Infrastructure:
- S3 bucket with versioning enabled (nf-core-pulumi-state)
- Server-side encryption (AES-256)
- Public access blocked
- Lifecycle policy (90-day version retention)
- Tags for resource tracking

Backend usage:
  pulumi login 's3://nf-core-pulumi-state?region=eu-north-1&awssdk=v2'

This project uses local backend to avoid circular dependency, while
all other projects use the S3 backend.

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@edmundmiller @claude
feat: add Pulumi state storage bootstrap project
3388276 
Create S3-backed state storage for all Pulumi projects. This bootstrap
project creates and manages the nf-core-pulumi-state bucket in eu-north-1.

Infrastructure:
- S3 bucket with versioning enabled
- Server-side encryption (AES-256)
- Public access blocked
- Lifecycle policy (90-day version retention)
- Tags for resource tracking

Backend usage:
  pulumi login 's3://nf-core-pulumi-state?region=eu-north-1&awssdk=v2'

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@edmundmiller
docs: add manual setup documentation for Pulumi state storage
abc64d0 
- Document AWS CLI commands for manual S3 bucket creation
- Add 1Password credential retrieval instructions
- Document environment variable requirements
- Add Pulumi import commands for existing bucket resources
- Provide troubleshooting guidance for bucket setup
@edmundmiller
feat: add ResourceOptimization workspace with conditional compute env…
e2d7ab9 
…ironment deployment

- Add support for workspace-specific configurations in multi-workspace setup
- Implement conditional compute environment deployment (CPU/GPU/ARM can be toggled per workspace)
- Add workspace-specific IAM resource naming to avoid conflicts between workspaces
- Add pulumi-seqera SDK (v0.25.2) to both awsmegatests and resource_optimization workspaces
- Update S3 infrastructure to support both import (existing buckets) and create (new buckets)
- Add absolute paths for seqerakit config files in constants
- Copy seqerakit configs to shared module for workspace reuse
- Update credentials.py to accept workspace_name parameter for unique IAM resources
- Remove pulumi-seqera from pyproject.toml dependencies (loaded via sdks/ directory)
- Document manual setup commands and 1Password credential retrieval in pulumi_state/README.md

Deployed ResourceOptimization workspace:
- Workspace ID: 80750985193419
- CPU compute environment only (GPU and ARM disabled)
- S3 bucket: nf-core-resource-optimization
- IAM resources: TowerForge-AWSMegatests-ResourceOptimization (user and policies)
- Successfully deployed 13 resources in 1m2s
@edmundmiller edmundmiller requested review from a team and maxulysse as code owners October 22, 2025 07:39
@edmundmiller edmundmiller self-assigned this Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxulysse maxulysse Awaiting requested review from maxulysse maxulysse is a code owner

@FloWuenne FloWuenne Awaiting requested review from FloWuenne

At least 1 approving review is required to merge this pull request.

Assignees

@edmundmiller edmundmiller

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@edmundmiller