Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# Contributing to pulumitest

This guide provides detailed information about the pulumitest library's architecture, testing patterns, and development practices.

## Build Commands
- Run all tests in pulumitest module: `go test -v ./...`
- Run a single test: `go test -v -run TestName`
- Run with coverage: `go test -v -coverprofile="coverage.txt" -coverpkg=./... ./...`
- Run with race detection: `go test -race ./...`
- Check for lint issues: `golangci-lint run`
- From root: `go test -v github.com/pulumi/providertest/pulumitest`

## Architecture

The `pulumitest` module is the core testing library for Pulumi programs and providers. It wraps Pulumi's Automation API with testing-specific defaults and conveniences.

### Core Components

**PulumiTest** (`pulumiTest.go`)
- Main test harness that manages the full test lifecycle
- Handles program copying, dependency installation, stack creation/destruction
- Stores context, working directory, options, and current stack reference
- Created via `NewPulumiTest(t, source, ...opts)` with functional options pattern

**PT Interface** (`testingT.go`)
- Thin wrapper around Go's `testing.T` that provides test-specific methods
- All operations accept PT as first parameter for proper test reporting
- Helper methods call `t.Helper()` to ensure correct failure line numbers in test output

**Options System** (`opttest/opttest.go`)
- Functional options pattern via `opttest.Option` interface
- Key options: `AttachProvider`, `AttachProviderServer`, `AttachProviderBinary`, `TestInPlace`, `SkipInstall`, `SkipStackCreate`, `YarnLink`, `GoModReplacement`, `DotNetReference`, `LocalProviderPath`
- Options are deeply copied to allow independent modification when using `CopyToTempDir()`
- Default passphrase: "correct horse battery staple" for deterministic encryption

### Operations

**Stack Lifecycle** (`newStack.go`, `installStack.go`, `destroy.go`)
- `Install(t)`: Runs `pulumi install` to restore dependencies
- `NewStack(t, name, ...opts)`: Creates new stack with local backend by default, sets as current stack
- `InstallStack(t, name)`: Convenience combining Install + NewStack
- `Destroy(t)`: Destroys resources and removes stack (automatic via `t.Cleanup()`)
- Auto-destroy behavior configurable via `optnewstack.DisableAutoDestroy()` or `EnableAutoDestroy()`

**Pulumi Operations** (`up.go`, `preview.go`, `refresh.go`, `destroy.go`, `import.go`)
- `Up(t, ...opts)`: Runs `pulumi up`, returns `auto.UpResult`
- `Preview(t, ...opts)`: Runs `pulumi preview`, returns `auto.PreviewResult`
- `Refresh(t, ...opts)`: Runs `pulumi refresh`, returns `auto.RefreshResult`
- `Destroy(t)`: Destroys current stack
- `Import(t, resourceType, name, id, ...opts)`: Imports existing resource into state
- All operations accept `optrun.Option` for runtime configuration

**Test Utilities**
- `CopyToTempDir(t, ...opts)`: Creates independent copy in temp directory for isolated testing
- `UpdateSource(t, newSourcePath)`: Replaces program files with new version for multi-step tests
- `SetConfig(t, key, value)`: Sets stack configuration values
- `ExportStack(t)`: Exports stack state as deployment JSON
- `ImportStack(t, deployment)`: Imports stack state from deployment JSON
- `GrpcLog(t)`: Retrieves gRPC log for provider calls made during test
- `Run(t, fn, ...opts)`: Execute function with optional state caching and option layering

### Provider Attachment

The library supports multiple ways to configure providers for testing:

**Attach In-Process Server** (`opttest.AttachProviderServer`)
- Start `pulumirpc.ResourceProviderServer` implementation in-process via goroutine
- Allows debugging from test into provider code
- Factory function receives `PulumiTest` context

**Attach Local Binary** (`opttest.AttachProviderBinary`)
- Start pre-built provider binary and attach via `PULUMI_DEBUG_PROVIDERS`
- Path can be directory (assumes `pulumi-resource-<name>`) or full binary path

**Attach Downloaded Plugin** (`opttest.AttachDownloadedPlugin`)
- Downloads specific plugin version via `pulumi plugin install`
- Starts and attaches the downloaded provider

**Local Provider Path** (`opttest.LocalProviderPath`)
- Sets `plugins.providers` in Pulumi.yaml for providers that don't support attachment
- Provider is started by Pulumi engine, not attached




### Subdirectories

**opttest/** - Options for PulumiTest construction and stack creation
**optrun/** - Options for Up/Preview/Refresh/Destroy operations
**optnewstack/** - Options for NewStack (auto-destroy configuration)
**assertup/** - Assertions for Up results (`HasNoDeletes`, `HasNoChanges`, etc.)
**assertpreview/** - Assertions for Preview results
**assertrefresh/** - Assertions for Refresh results
**changesummary/** - Types for analyzing resource change summaries
**sanitize/** - Utilities for sanitizing sensitive data in logs and snapshots

### File Organization

- Core types: `pulumiTest.go`, `testingT.go`
- Operations: `up.go`, `preview.go`, `refresh.go`, `destroy.go`, `import.go`
- Stack management: `newStack.go`, `installStack.go`, `install.go`
- Utilities: `copy.go`, `updateSource.go`, `setConfig.go`, `exportStack.go`, `importStack.go`
- gRPC logging: `grpcLog.go`, `grpcLog_test.go`
- File system: `fs_unix.go`, `fs_windows.go`, `tempdir.go`
- Project file handling: `pulumiYAML.go`, `csproj.go`
- Command execution: `execCmd.go`, `run.go`
- Cleanup: `cleanup.go`

## Code Patterns

**Helper Functions**
- Always call `t.Helper()` in functions that accept PT to ensure correct test failure line numbers
- Return errors when operation can fail; panic only for programmer errors
- Accept variadic options as last parameters

**Testing**
- Use `t.Parallel()` for tests that can run concurrently
- Use `t.Run()` for sub-tests to organize test cases
- Cleanup via `t.Cleanup()` ensures resources freed even on test failure

**Context Management**
- Test context created from `t.Deadline()` if available
- Context cancelled via `t.Cleanup()` for automatic resource cleanup
- Provider factories receive context to handle graceful shutdown

**Options Pattern**
- All options implement `Option` interface with `Apply(*Options)` method
- Options are composable and order-independent (where possible)
- Use `Defaults()` to reset options to initial state
- Options deeply copied for independent test instances

## Troubleshooting

### .NET/C# Issues

**Target Framework Not Found**
- Error: `Framework 'Microsoft.NETCore.App', version 'X.X.X' not found`
- Solution: Install the required .NET SDK version
- Check installed versions: `dotnet --list-sdks`

**Project Reference Resolution**
- Ensure referenced projects use compatible target frameworks
- Use absolute paths or paths relative to the test working directory
- The test framework automatically resolves relative paths to absolute paths

**NuGet Package Version Mismatch**
- Pulumi .NET SDK versioning differs from CLI versioning
- Latest stable SDK: 3.90.0 (as of writing)
- Check NuGet.org for current versions

**Common Test Failures**
- `pulumi install` fails: Check .csproj package versions are available on NuGet
- Build fails with missing types: Verify all project references are correctly added
- Stack creation hangs: Check for `PULUMI_AUTOMATION_API_SKIP_VERSION_CHECK=true` in CI environments
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ Library for testing Pulumi providers by running Pulumi programs.

The library is composed of several modules. The most important of these is the [`pulumitest`](./pulumitest/) module. This is a library designed for testing any Pulumi program within a Go test. It extends the Go [Automation API](https://www.pulumi.com/automation/) with defaults appropriate for local testing such as using temporary directories for state.

Here's a short example of how to use pulumitest:
## Quick Start

Here's a basic example of how to use pulumitest:

```go
import (
Expand All @@ -23,7 +25,7 @@ func TestPulumiProgram(t *testing.T) {
}
```

Refer to [the full documentation](./pulumitest/README.md) for a complete walkthrough of the features.
For detailed usage patterns, examples, and configuration options, see [the pulumitest documentation](./pulumitest/README.md).

## Attaching In-Process Providers

Expand Down Expand Up @@ -93,3 +95,4 @@ The `providers` module provides additional utilities for `pulumitest` when build
The `grpclog` module contains types and functions for reading, querying and writing Pulumi's grpc log format (normally living in a `grpc.json` file).

The `replay` module has methods for exercising specific provider gRPC methods directly and from existing log files.

51 changes: 51 additions & 0 deletions pulumitest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,34 @@ NewPulumiTest(t, "test_dir",
opttest.GoModReplacement("github.com/pulumi/pulumi-my-provider/sdk/v3", "..", "sdk"))
```

### .NET/C# - Project References

For .NET/C#, we support adding local project references to the `.csproj` file. This is useful when testing with a locally built SDK that hasn't been published to NuGet.

The reference can be specified using the `DotNetReference` test option:

```go
// Path can point to a .csproj file or a directory containing one
NewPulumiTest(t, "test_dir",
opttest.DotNetReference("Pulumi.Aws", "..", "pulumi-aws", "sdk", "dotnet"))
```

The `DotNetReference` option adds a `<ProjectReference>` element to the test program's `.csproj` file, and the framework automatically resolves relative paths to absolute paths.

### .NET/C# Examples

```go
// Basic .NET test
test := NewPulumiTest(t, "path/to/csharp/project")
up := test.Up(t)

// Test with local SDK reference
test := NewPulumiTest(t,
"path/to/csharp/project",
opttest.DotNetReference("Pulumi.Aws", "../pulumi-aws/sdk/dotnet"),
)
```

## Additional Operations

### Update Source
Expand All @@ -149,6 +177,27 @@ Set a variable in the stack's config:
test.SetConfig(t, "gcp:project", "pulumi-development")
```

## Testing Patterns

### Default Behavior
- Programs are copied to a temporary directory (OS-specific or `PULUMITEST_TEMP_DIR`)
- Dependencies are installed automatically unless `SkipInstall()` is used
- A stack named "test" is created automatically unless `SkipStackCreate()` is used
- Local backend is used in the temp directory unless `UseAmbientBackend()` is used
- Stacks are automatically destroyed on test completion
- Temp directories are retained on failure for debugging (configurable via environment variables)

### gRPC Logging
- Enabled by default, written to `grpc.json` in the working directory
- Disable with `opttest.DisableGrpcLog()`
- Access via `test.GrpcLog(t)` which returns parsed log entries
- Supports sanitization of secrets before writing to disk

### Multi-Step Tests
- Use `UpdateSource(t, path)` to replace program files between operations
- Useful for testing update behavior, replacements, etc.
- Example pattern: `Up()` → `UpdateSource()` → `Up()` → assert changes

## Environment Variables

The behavior of pulumitest can be adjusted through use of certain environment variables:
Expand All @@ -160,6 +209,8 @@ The behavior of pulumitest can be adjusted through use of certain environment va
| `PULUMITEST_RETAIN_FILES_ON_FAILURE` | Can be set explicitly to `true` or `false`. Defaults to `true` locally and `false` in CI environments. |
| `PULUMITEST_SKIP_DESTROY_ON_FAILURE` | Skips the automatic attempt to destroy a stack even after a test failure. This defaults to `false`. If set to true, the files will also be retained unless `PULUMITEST_RETAIN_FILES_ON_FAILURE` set to `false`. |
| `PULUMITEST_TEMP_DIR` | Changes the default temp directory from the OS-specific system location. |
| `PULUMI_CONFIG_PASSPHRASE` | Override default passphrase (defaults to "correct horse battery staple") |
| `PULUMI_BACKEND_URL` | Override default local backend |

## Asserts

Expand Down
125 changes: 125 additions & 0 deletions pulumitest/csproj.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
package pulumitest

import (
"encoding/xml"
"fmt"
"os"
"path/filepath"
"strings"
)

// xmlNode represents a generic XML node that preserves all structure, attributes, and content.
// This approach allows us to manipulate .csproj files without losing any data.
type xmlNode struct {
XMLName xml.Name
Attrs []xml.Attr `xml:"-"`
Content []byte `xml:",innerxml"`
Nodes []xmlNode `xml:",any"`
}

func (n *xmlNode) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
n.XMLName = start.Name
n.Attrs = start.Attr
type node xmlNode
return d.DecodeElement((*node)(n), &start)
}

func (n *xmlNode) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
start.Name = n.XMLName
start.Attr = n.Attrs
if err := e.EncodeToken(start); err != nil {
return err
}
if len(n.Nodes) > 0 {
for _, node := range n.Nodes {
if err := e.Encode(&node); err != nil {
return err
}
}
} else if len(n.Content) > 0 {
if err := e.EncodeToken(xml.CharData(n.Content)); err != nil {
return err
}
}
return e.EncodeToken(start.End())
}

// findCsprojFile finds a .csproj file in the given directory
func findCsprojFile(dir string) (string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return "", fmt.Errorf("failed to read directory %s: %w", dir, err)
}

for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".csproj") {
return filepath.Join(dir, entry.Name()), nil
}
}

return "", fmt.Errorf("no .csproj file found in directory %s", dir)
}

// addProjectReferences adds ProjectReference elements to a .csproj file
func addProjectReferences(csprojPath string, references map[string]string) error {
// Read and parse the .csproj file
data, err := os.ReadFile(csprojPath)
if err != nil {
return fmt.Errorf("failed to read .csproj file: %w", err)
}

var root xmlNode
if err := xml.Unmarshal(data, &root); err != nil {
return fmt.Errorf("failed to parse .csproj XML: %w", err)
}

// Build ProjectReference nodes
var refNodes []xmlNode
for _, refPath := range references {
absPath, err := filepath.Abs(refPath)
if err != nil {
return fmt.Errorf("failed to get absolute path for %s: %w", refPath, err)
}

// Check if the path exists
info, err := os.Stat(absPath)
if err != nil {
return fmt.Errorf("reference path does not exist: %s: %w", absPath, err)
}

// If it's a directory, look for a .csproj file in it
if info.IsDir() {
absPath, err = findCsprojFile(absPath)
if err != nil {
return fmt.Errorf("failed to find .csproj in directory: %w", err)
}
}

refNodes = append(refNodes, xmlNode{
XMLName: xml.Name{Local: "ProjectReference"},
Attrs: []xml.Attr{{Name: xml.Name{Local: "Include"}, Value: absPath}},
})
}

// Create new ItemGroup node with ProjectReferences
itemGroup := xmlNode{
XMLName: xml.Name{Local: "ItemGroup"},
Nodes: refNodes,
}

// Add the new ItemGroup to the root
root.Nodes = append(root.Nodes, itemGroup)

// Marshal back to XML
output, err := xml.MarshalIndent(&root, "", " ")
if err != nil {
return fmt.Errorf("failed to marshal .csproj XML: %w", err)
}

// Write back to file
if err := os.WriteFile(csprojPath, output, 0644); err != nil {
return fmt.Errorf("failed to write .csproj file: %w", err)
}

return nil
}
Loading