Skip to content

feat: add ai configs resources#404

Open
monsagri wants to merge 26 commits intomainfrom
ffeldberg/test-skill-on-ai-configs-resources
Open

feat: add ai configs resources#404
monsagri wants to merge 26 commits intomainfrom
ffeldberg/test-skill-on-ai-configs-resources

Conversation

@monsagri
Copy link
Copy Markdown
Contributor

@monsagri monsagri commented Mar 18, 2026

This is an alternative PR to #402, made using Claude and the 'terraform-provider-add-resource' skill I created earlier.

Functionality is tested against a locally built version of the provider. You can see the stuff it's created in the Engineering Blitz Account here https://app.launchdarkly.com/projects/ai-config-local-test/ai-configs?env=test&selected-env=test

Bit more detail below in a writeup of the work done provided by Claude. And in the spirit of transparency - this fun little jaunt cost LaunchDarkly $74.90 (partly on me due to using big Opus)

# AI Config Resources - Development Log

## Overview

Added Terraform provider support for LaunchDarkly AI Configs: 4 new resources, 3 data sources, acceptance tests, examples, CI updates, and auto-generated docs.

## Prompts Used

### 1. Initial request
> Please use the terraform provider add resource skill in order to add support for AI Config resources and data sources to the provider. Use our public facing documentation etc. as needed to inform your work https://launchdarkly.com/docs/home/ai-configs https://launchdarkly.com/docs/api/ai-configs

### 2. Scope clarification (asked by Claude)
Claude asked which resources to implement and whether variations should be inline or separate. Response:
> Implement all of these, when looking at variations it'd be best to follow existing patterns for our flag resource which also supports variations. I believe separate resources make the most sense, but make sure to compare against both the flag resource and potential guidance from our docs.

### 3. Build command correction
> It looks like you didn't run make fmt and make generate - are those mentioned in the skill for creating a new resource in the provider? I also can't see any new tests, we want both unit acceptance tests (including updates to the CI workflow to ensure acceptance tests are run) - is that mentioned in the skill?

### 4. Skill discovery
> There is a skill for adding a new resource to the provider - it's called terraform-provider-add-resource - please double check

This pointed Claude to the correct `terraform-provider-add-resource` skill, which defined the full checklist (Steps 0-10) including `make fmt`, `make generate`, example files, acceptance tests, and CI matrix updates.
### 5. Development log request
> Can you give me a writeup on the development process we've followed...

### 6. Cross-branch comparison
> Can you compare the solution you've come up with on this branch to another implementation on the branch devin/1773079106-add-ai-config-resource and compare/contrast/assess each implementation?

Claude compared both branches across 9 dimensions (scope, architecture, schema design, variation handling, test depth, CI, docs, code quality, completeness). See `IMPLEMENTATION_COMPARISON.md` for the full analysis.

### 7. Address identified gaps
> Can you address the gaps in our branch that you identified?

Added 5 new `ai_config` test scenarios ported from the comparison findings: `TestAccAIConfig_WithMode`, `TestAccAIConfig_WithMaintainer`, `TestAccAIConfig_WithTeamMaintainer`, `TestAccAIConfig_WithEvaluationMetric`, `TestAccAIConfig_RemoveOptionalFields`. Verified `"judge"` mode doesn't exist in the Go API client — skipped.

### 8. CI lint fix
> I've just checked the CI lint logs and there's only one error... func `validateJsonString` is unused

Removed the unused `validateJsonString()` function and its `validation` import from `json_helper.go`.

### 9. CI InternalValidate fix
> Another issue from CI... resource launchdarkly_model_config: No Update defined, must set ForceNew on: []string{"tags"} / data source launchdarkly_model_config: provider is a reserved field name

Two fixes: (1) Added `ForceNew: true` to the `tags` field on `model_config` by overriding the `tagsSchema()` result. (2) Renamed the reserved `provider` attribute to `model_provider` across `keys.go`, helper, resource, tests, and examples.

### 10. Local testing
> Now please build the current version of the provider and attempt to test it using the local-testing folder

Built provider with `make build`, created `local-testing/ai_config_test.tf` with all 4 resources + 3 data sources + outputs. `terraform plan` succeeded — all resources planned correctly. Apply requires a live API token.

## Skill Used: `terraform-provider-add-resource`

The `terraform-provider-add-resource` skill (v1.0.0, from `launchdarkly-agent-skills`) defines a 10-step process for adding new resources to the provider:

| Step | What |
|------|------|
| 0 | Understand the API surface (OpenAPI spec, Go client types) |
| 1 | Add schema constants to `keys.go` (with `//gofmts:sort`) |
| 2 | Create helper file (`<name>_helper.go`) with shared schema + read logic |
| 3 | Create resource file with CRUD + Exists + Import |
| 4 | Create data source file (thin wrapper around shared read) |
| 5 | Register in `provider.go` ResourcesMap/DataSourcesMap |
| 6 | Create `examples/resources/` and `examples/data-sources/` with `resource.tf`, `import.sh`, `data-source.tf` |
| 7 | Doc templates (optional - `make generate` auto-generates from schema) |
| 8 | Write acceptance tests (create/import/update steps, CheckDestroy) |
| 9 | Add test cases to `.github/workflows/test.yml` matrix |
| 10 | Run `make fmt`, `make build`, `make generate` |

The skill also includes a `references/patterns.md` with full code templates for helpers, resources, data sources, and tests.

## What Was Built

### Resources
- `launchdarkly_ai_config` — core AI Config (key, name, mode, description, tags)
- `launchdarkly_ai_config_variation` — variation as separate child resource (messages, model, tools, instructions)
- `launchdarkly_model_config` — project-level model configuration (ForceNew only, no PATCH API)
- `launchdarkly_ai_tool` — project-level AI tool definitions (JSON schema, custom params)

### Data Sources
- `launchdarkly_ai_config`, `launchdarkly_model_config`, `launchdarkly_ai_tool`

### Key Design Decisions
- **Variations as separate resources** rather than inline blocks, following the `feature_flag` / `feature_flag_environment` pattern
- **Standard API client** (`client.ld.AIConfigsApi`) — the v22 Go client doesn't require beta headers for AI Config endpoints
- **Model config is immutable** — no PATCH API exists, so all fields are `ForceNew: true`
- **JSON fields** (`model`, `params`, `schema_json`, `custom_parameters`) use `TypeString` with semantic diff suppression to avoid false diffs from key ordering

### Files Changed/Created
- 11 new Go source files (helpers, resources, data sources)
- 7 new test files
- 11 new example files (`resource.tf`, `import.sh`, `data-source.tf`)
- Modified: `keys.go`, `provider.go`, `test.yml`
- Auto-generated: `docs/resources/*.md`, `docs/data-sources/*.md`

Comparison overview of the branches by claude

  Branch A (ours) wins on scope (4 resources vs 1), CI integration, code conventions, and overall completeness. Branch B (Devin) wins on ai_config test depth — it has 6 focused test scenarios covering mode, maintainers, evaluation metrics, and
  optional field removal, while ours only tests the basic create/update/import flow.

  Key gaps to address on our branch:
  1. Port Branch B's deeper ai_config test scenarios (mode, maintainers, evaluation metrics, optional removal)
  2. Consider adding "judge" to mode validation if the API supports it

  Key gaps in Branch B:
  1. Missing 3 of 4 resources (variation, model_config, ai_tool) — users can't actually configure AI behavior
  2. No CI workflow updates — tests silently don't run
  3. Non-idiomatic Go naming (Ai vs AI)
  4. No ConflictsWith on maintainer fields

@monsagri monsagri requested review from a team as code owners March 18, 2026 15:36
@monsagri monsagri changed the title Ffeldberg/test skill on ai configs resources feat: add ai configs resources Mar 18, 2026
monsagri and others added 6 commits March 18, 2026 16:38
S1009: len() for nil maps is defined as zero, so the nil check is unnecessary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The AIConfigPost constructor sets description="" by default, which the
API rejects as a 400. Clear it to nil so it's omitted from JSON unless
explicitly set. Also add description to all test HCL templates and
success_criteria to the evaluation metric test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
monsagri and others added 11 commits March 19, 2026 09:47
The GenericOpenAPIError.Error() only returns the HTTP status, not the
body. Use handleLdapiErr which includes the body in the error string.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add "judge" to allowed mode values (API requires it for evaluation metrics)
- Fix evaluation metric test: use $ld:ai:judge: prefixed key format
- Fix is_inverted always being set on read (was nil when false, causing
  import state mismatch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@monsagri monsagri requested a review from Copilot March 19, 2026 20:51
@monsagri monsagri requested a review from a team March 19, 2026 20:51
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Terraform provider support for LaunchDarkly AI Configs, including resources/data sources for AI configs, variations, tools, and model configs, plus acceptance tests, examples, generated docs, and CI updates to run the new acceptance test suites.

Changes:

  • Introduces new resources/data sources for ai_config, ai_config_variation, ai_tool, and model_config using the v22 Go SDK AI Configs API.
  • Adds shared schema/read helpers and JSON validation/diff-suppression helpers for JSON-encoded fields.
  • Adds acceptance tests, examples, generated docs, and expands the CI test matrix to execute the new acceptance tests.

Reviewed changes

Copilot reviewed 40 out of 42 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
launchdarkly/resource_launchdarkly_ai_config.go Implements launchdarkly_ai_config CRUD + import.
launchdarkly/ai_config_helper.go Shared schema + read logic for AI Config resource/data source.
launchdarkly/data_source_launchdarkly_ai_config.go Adds launchdarkly_ai_config data source.
launchdarkly/resource_launchdarkly_ai_config_test.go Acceptance tests for AI Config resource scenarios.
launchdarkly/data_source_launchdarkly_ai_config_test.go Acceptance tests for AI Config data source.
launchdarkly/resource_launchdarkly_ai_config_variation.go Implements launchdarkly_ai_config_variation CRUD + import.
launchdarkly/ai_config_variation_helper.go Schema + read/flatten/expand helpers for AI Config variations.
launchdarkly/resource_launchdarkly_ai_config_variation_test.go Acceptance tests for AI Config variation scenarios.
launchdarkly/resource_launchdarkly_ai_tool.go Implements launchdarkly_ai_tool CRUD + import.
launchdarkly/ai_tool_helper.go Shared schema + read logic for AI tool resource/data source.
launchdarkly/data_source_launchdarkly_ai_tool.go Adds launchdarkly_ai_tool data source.
launchdarkly/resource_launchdarkly_ai_tool_test.go Acceptance tests for AI tool resource scenarios.
launchdarkly/data_source_launchdarkly_ai_tool_test.go Acceptance tests for AI tool data source.
launchdarkly/resource_launchdarkly_model_config.go Implements launchdarkly_model_config create/read/delete + import (immutable model config API).
launchdarkly/model_config_helper.go Shared schema + read logic for model config resource/data source.
launchdarkly/data_source_launchdarkly_model_config.go Adds launchdarkly_model_config data source.
launchdarkly/resource_launchdarkly_model_config_test.go Acceptance tests for model config resource scenarios.
launchdarkly/data_source_launchdarkly_model_config_test.go Acceptance tests for model config data source.
launchdarkly/json_helper.go Adds JSON validation + semantic diff suppression helpers for JSON-encoded schema fields.
launchdarkly/keys.go Adds new schema key constants used by the AI Configs resources/data sources.
launchdarkly/provider.go Registers the new resources and data sources with the provider.
examples/resources/launchdarkly_ai_config/resource.tf Example usage for AI Config resource.
examples/resources/launchdarkly_ai_config/import.sh Import example for AI Config resource.
examples/resources/launchdarkly_ai_config_variation/resource.tf Example usage for AI Config variation resource.
examples/resources/launchdarkly_ai_config_variation/import.sh Import example for AI Config variation resource.
examples/resources/launchdarkly_ai_tool/resource.tf Example usage for AI tool resource.
examples/resources/launchdarkly_ai_tool/import.sh Import example for AI tool resource.
examples/resources/launchdarkly_model_config/resource.tf Example usage for model config resource.
examples/resources/launchdarkly_model_config/import.sh Import example for model config resource.
examples/data-sources/launchdarkly_ai_config/data-source.tf Example usage for AI Config data source.
examples/data-sources/launchdarkly_ai_tool/data-source.tf Example usage for AI tool data source.
examples/data-sources/launchdarkly_model_config/data-source.tf Example usage for model config data source.
docs/resources/ai_config.md Generated documentation for AI Config resource.
docs/resources/ai_config_variation.md Generated documentation for AI Config variation resource.
docs/resources/ai_tool.md Generated documentation for AI tool resource.
docs/resources/model_config.md Generated documentation for model config resource.
docs/data-sources/ai_config.md Generated documentation for AI Config data source.
docs/data-sources/ai_tool.md Generated documentation for AI tool data source.
docs/data-sources/model_config.md Generated documentation for model config data source.
.github/workflows/test.yml Adds new acceptance test patterns to the CI matrix.
.gitignore Ignores .claude/ and normalizes .DS_Store entry formatting.
go.sum Updates module checksums due to dependency graph changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…st cleanup

- Fix GetOk on bool/string zero-values in ai_config create+update
- Add ConflictsWith on model vs model_config_key in variation schema
- Add CustomizeDiff for is_inverted requiring evaluation_metric_key
- Convert ValidateFunc to ValidateDiagFunc for JSON fields
- Add sleep before read in variation create/update for version propagation
- Create ai_config_variation data source with tests and docs
- Add inline model acceptance test for variations
- Deduplicate test project helper and cooldown into shared ai_test_helpers_test.go
- Add CheckDestroy to all data source tests
- Alphabetize provider.go registration
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants