Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/actions/download-prerequisites/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ runs:
using: "composite"
steps:
- name: Download the prerequisites bin
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
name: prerequisites-bin
path: bin
Expand All @@ -19,7 +19,7 @@ runs:
run: rm bin/executables.txt

- name: Download schema-embed.json
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
# Use a pattern to avoid failing if the artifact doesn't exist
pattern: schema-embed.*
Expand Down
2 changes: 1 addition & 1 deletion .github/actions/download-provider/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ runs:
steps:

- name: Download pulumi-resource-scm
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
pattern: pulumi-resource-scm-*-linux-amd64.tar.gz
path: ${{ github.workspace }}/bin
Expand Down
2 changes: 1 addition & 1 deletion .github/actions/download-sdk/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ runs:
using: "composite"
steps:
- name: Download ${{ inputs.language }} SDK
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
name: ${{ inputs.language }}-sdk.tar.gz
path: ${{ github.workspace}}/sdk/
Expand Down
12 changes: 12 additions & 0 deletions .github/actions/esc-action/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: "Load secrets"
description: |
This is a temporary action which assists with our migration to ESC. Instead
of surrounding every step that references secrets with an "if ESC" block, we
instead modify those steps to consume their secrets from this step's outputs.
Then, later, we can replace this action with esc-action to actually load
secrets from ESC.
inputs: {}
outputs: {}
runs:
using: "node20"
main: "index.js"
14 changes: 14 additions & 0 deletions .github/actions/esc-action/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
const fs = require("fs");

const file = process.env["GITHUB_OUTPUT"];
var stream = fs.createWriteStream(file, { flags: "a" });

for (const [name, value] of Object.entries(process.env)) {
try {
stream.write(`${name}<<EEEOOOFFF\n${value}\nEEEOOOFFF\n`); // << syntax accommodates multiline strings.
} catch (err) {
console.log(`error: failed to set output for ${name}: ${err.message}`);
}
}

stream.end();
10 changes: 5 additions & 5 deletions .github/actions/setup-tools/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,33 +59,33 @@ runs:

- name: Setup Node
if: inputs.tools == 'all' || contains(inputs.tools, 'nodejs')
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5
with:
node-version: 20.x
registry-url: https://registry.npmjs.org

- name: Setup DotNet
if: inputs.tools == 'all' || contains(inputs.tools, 'dotnet')
uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1
uses: actions/setup-dotnet@d4c94342e560b34958eacfc5d055d21461ed1c5d # v5.0.0
with:
dotnet-version: 8.0.x

- name: Setup Python
if: inputs.tools == 'all' || contains(inputs.tools, 'python')
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
with:
python-version: 3.11.8

- name: Setup Java
if: inputs.tools == 'all' || contains(inputs.tools, 'java')
uses: actions/setup-java@c5195efecf7bdfc987ee8bae7a71cb8b11521c00 # v4.7.1
uses: actions/setup-java@dded0888837ed1f317902acf8a20df0ad188d165 # v5.0.0
with:
cache: gradle
distribution: temurin
java-version: 11

- name: Setup Gradle
if: inputs.tools == 'all' || contains(inputs.tools, 'java')
uses: gradle/gradle-build-action@ac2d340dc04d9e1113182899e983b5400c17cda1 # v3
uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4.4.3
with:
gradle-version: 7.6
144 changes: 144 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# Pulumi scm Provider

The Pulumi scm provider is a Go-based Pulumi resource provider that bridges the Terraform provider to Pulumi. It generates SDKs for TypeScript/JavaScript, Python, .NET, Go, and Java. The provider uses the Terraform provider as an upstream source via git submodules.

Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.

## Working Effectively

### Prerequisites and Environment Setup
- All required dependencies are automatically installed via the `.github/workflows/copilot-setup-steps.yml` workflow
- This includes Go, Node.js, Python, .NET, Gradle, and all necessary Pulumi tools

### Initial Repository Setup
- Initialize the upstream submodule: `make upstream`

### Build Process
- **ALWAYS use `make` targets** - Never run custom commands unless explicitly told to
- **NEVER work directly in the `sdk/` folder** - All SDK generation and building is automated through `make`
- If a `make` target fails, there is something wrong with the environment setup, not the target itself

### Available Make Targets

#### Primary Build Targets:
- `make build` -- Build the provider and all SDKs
- `make provider` -- Build the provider binary
- `make schema` -- Generate the provider schema
- `make tfgen` -- Generate SDKs from schema
- `make upstream` -- Initialize upstream submodule

#### SDK Targets:
- `make build_sdks` -- Build all SDK packages
- `make generate_sdks` -- Generate all SDK source code
- `make build_nodejs` -- Build TypeScript/Node.js SDK
- `make build_python` -- Build Python SDK
- `make build_dotnet` -- Build .NET SDK
- `make build_go` -- Build Go SDK
- `make build_java` -- Build Java SDK

#### Development Targets:
- `make lint_provider` -- Lint provider Go code
- `make test_provider` -- Run provider unit tests

### Build Guidelines:
- **NEVER CANCEL** any build command once started - builds may take several minutes
- Set timeouts to 300+ seconds for build operations
- **DO NOT run tests in `examples/`** - They require cloud credentials and will run in PR workflows

## Repository Structure

### Key Directories:
- `provider/` -- Go provider implementation
- `sdk/` -- Generated SDKs for all languages
- `upstream/` -- Git submodule with the Terraform provider
- `scripts/` -- Build and utility scripts
- `examples/` -- Example Pulumi programs (test framework available but skipped)

### Important Files:
- `Makefile` -- Primary build orchestration with all available targets
- `provider/go.mod` -- Provider dependencies
- `.github/workflows/copilot-setup-steps.yml` -- Environment setup for AI coding agents
- `.github/workflows/` -- CI/CD pipelines

## Development Workflow

### Making Code Changes:
1. Initialize repository: `make upstream`
2. Make changes to provider code in `provider/`
3. Validate with: `make lint_provider`
4. Test with: `make test_provider`
5. Build provider: `make provider`
6. Generate and build SDKs: `make build_sdks`

### Validation Steps:
- Always use `make lint_provider` to lint provider code
- Use `make test_provider` to run provider unit tests
- Use `make build` to validate the full build process

### Working with SDKs:
- **NEVER work directly in `sdk/` folders** - All SDK operations are automated via `make` targets
- All SDKs are generated and built through `make` commands
- TypeScript SDK: Use `make build_nodejs`
- Python SDK: Use `make build_python`
- .NET SDK: Use `make build_dotnet`
- Go SDK: Use `make build_go`
- Java SDK: Use `make build_java`

## Validation Scenarios

### Code Quality Validation:
- Use `make lint_provider` to lint provider Go code
- Use `make test_provider` to run provider unit tests
- Use `make build` to validate full build process

### Manual Code Review:
- Check Go code follows standard patterns
- Validate resource definitions in `provider/resources.go`
- Ensure imports and dependencies are correct

## Common Tasks Reference

### Repository Root Contents:
```
.ci-mgmt.yaml -- CI management configuration
.devcontainer/ -- Dev container setup
.github/ -- GitHub workflows and templates
.gitmodules -- Git submodule configuration
.golangci.yml -- Go linter configuration
.mise.toml -- Mise tool configuration
CONTRIBUTING.md -- Contribution guidelines
Makefile -- Build orchestration with all available targets
README.md -- Project documentation
devbox.json -- Development environment
provider/ -- Go provider implementation
scripts/ -- Build utilities
sdk/ -- Generated SDKs (managed via make targets)
upstream/ -- Terraform provider submodule
```

### Common File Operations:
- **Provider source**: `provider/resources.go` -- Resource definitions
- **Provider tests**: `provider/resources_test.go` -- Unit tests
- **Generated SDKs**: All in `sdk/` directory, managed via `make` targets only

### Common Development Tasks:
- Run provider tests: `make test_provider`
- Build provider: `make provider`
- Generate schema: `make schema`
- Build all SDKs: `make build_sdks`

## Build Expectations

- Provider builds: 1-3 minutes depending on system
- SDK generation: 2-5 minutes for all SDKs
- Individual SDK builds: 30 seconds to 2 minutes each
- Full build (`make build`): 5-10 minutes total

Set timeouts of 300+ seconds for build operations and NEVER CANCEL running builds.

## Critical Reminders

- **ALWAYS** use `make` targets - never run custom commands unless explicitly instructed
- **NEVER** work directly in `sdk/` folders - use `make` targets for all SDK operations
- **DO NOT** run tests in `examples/` - they require cloud credentials
- **FOCUS** on `make` targets for all development, building, and validation tasks
25 changes: 17 additions & 8 deletions .github/workflows/build_provider.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ on:
{"os": "linux", "arch": "arm64"},
{"os": "darwin", "arch": "amd64"},
{"os": "darwin", "arch": "arm64"},
{"os": "windows", "arch": "amd64"}
{"os": "windows", "arch": "amd64"},
{"os": "windows", "arch": "arm64"}
]
}

Expand All @@ -33,11 +34,17 @@ jobs:
strategy:
fail-fast: true
matrix: ${{ fromJSON(inputs.matrix) }}
permissions:
contents: read
id-token: write # For ESC secrets.
steps:
- name: Checkout Repo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
persist-credentials: false
persist-credentials: false
- id: esc-secrets
name: Map environment to ESC outputs
uses: ./.github/actions/esc-action
# Without ldid cross-compiling Node binaries on a Linux worker intended to work on darwin-arm64 fails to sign the
# binaries properly and they do not work as expected. See https://github.com/pulumi/pulumi-awsx/issues/1490
- uses: MOZGIII/install-ldid-action@v1
Expand Down Expand Up @@ -71,6 +78,8 @@ jobs:
go-provider-${{ matrix.platform.os }}-${{ matrix.platform.arch }}-
- name: Prepare local workspace before restoring previously built
run: make prepare_local_workspace
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Restore prerequisites
uses: ./.github/actions/download-prerequisites
- name: Restore makefile progress
Expand All @@ -80,11 +89,11 @@ jobs:
- name: Build provider
run: make "provider-${{ matrix.platform.os }}-${{ matrix.platform.arch }}"
env:
AZURE_SIGNING_CLIENT_ID: ${{ secrets.AZURE_SIGNING_CLIENT_ID }}
AZURE_SIGNING_CLIENT_SECRET: ${{ secrets.AZURE_SIGNING_CLIENT_SECRET }}
AZURE_SIGNING_TENANT_ID: ${{ secrets.AZURE_SIGNING_TENANT_ID }}
AZURE_SIGNING_KEY_VAULT_URI: ${{ secrets.AZURE_SIGNING_KEY_VAULT_URI }}
SKIP_SIGNING: ${{ secrets.AZURE_SIGNING_CLIENT_ID == '' && secrets.AZURE_SIGNING_CLIENT_SECRET == '' && secrets.AZURE_SIGNING_TENANT_ID == '' && secrets.AZURE_SIGNING_KEY_VAULT_URI == '' }}
AZURE_SIGNING_CLIENT_ID: ${{ steps.esc-secrets.outputs.AZURE_SIGNING_CLIENT_ID }}
AZURE_SIGNING_CLIENT_SECRET: ${{ steps.esc-secrets.outputs.AZURE_SIGNING_CLIENT_SECRET }}
AZURE_SIGNING_TENANT_ID: ${{ steps.esc-secrets.outputs.AZURE_SIGNING_TENANT_ID }}
AZURE_SIGNING_KEY_VAULT_URI: ${{ steps.esc-secrets.outputs.AZURE_SIGNING_KEY_VAULT_URI }}
SKIP_SIGNING: ${{ steps.esc-secrets.outputs.AZURE_SIGNING_CLIENT_ID == '' && secrets.AZURE_SIGNING_CLIENT_SECRET == '' && secrets.AZURE_SIGNING_TENANT_ID == '' && secrets.AZURE_SIGNING_KEY_VAULT_URI == '' }}

- name: Package provider
run: make provider_dist-${{ matrix.platform.os }}-${{ matrix.platform.arch }}
Expand Down
Loading
Loading