Ignore changes to DDE blueprints list to mitigate noise

[doshitan]

Ticket

Related to #1027

Changes

Due to (presumed) underlying provider limitations, when specifying blueprints to use, the BDA project resource will always show a diff between the IaC and current state.

Operating assumption is the list of blueprints won't change frequently after initially setting things up. So default to ignoring changes there to workaround (likely) provider issues which would break useful functionality like the check-infra-deploy-status.yml workflow.

Added docs for the manual steps involved why you do want to update blueprints and since no docs existed for the feature in general, added some basics there too.

In addition, the blueprint resources should have create_before_destroy=true Terraform lifecycle setting so the BDA project is updated to drop the blueprint before the blueprint itself is destroyed, which would cause errors otherwise.

Testing

As a part of navapbc/platform-test#278, making no config (or any other) changes and just running make infra-update-app-service APP_NAME=app-documentai ENVIRONMENT=dev

Before (no ignore_changes):

After (with ignore_changes):

Other attributes that change still show up

Adding/removing blueprints process

When adding a new custom template (tester.json here), the template resource itself is created:

But that's it, the project config is not updated:

Unless you follow the instructions, commenting out the ignore_changes line:

The diff is confusing due to the ordering bug, but the new templates does show up in the project:

The same applies when adding an existing ARN and adding/removing blueprints (evidence left out here for brevity).

Checking infra state

All the check-infra-deploy-status.yml workdlow does is run terraform plan -detailed-exitcode against the root modules, which just exits non-zero if there are state changes. Existing evidence shows there are state changes in a service's root module without this fix, and no state changes with it. But

Relevant section from ./bin/infra-deploy-status-check-configs:

Then run: terraform -chdir="infra/app-documentai/service" plan -input=false -detailed-exitcode -var="environment_name=dev" simulating the CI check.

With ignore_changes:

Without ignore_changes:

[sean-navapbc]

• Verify what happens when you DO want to change blueprints (follow the documented process)
• Confirm other attributes of the resource still show diffs correctly
• Test that the check-infra-deploy-status.yml workflow now passes

[doshitan]

• Verify what happens when you DO want to change blueprints (follow the documented process)
• Confirm other attributes of the resource still show diffs correctly
• Test that the check-infra-deploy-status.yml workflow now passes

@sean-navapbc Added please take a look.

[sean-navapbc]

Pragmatic Workaround Review

This PR implements a reasonable workaround for the awscc provider list ordering bug (hashicorp/terraform-provider-awscc#3058). The approach is sound and well-documented, but the manual update process has some operational risks.

✅ What This PR Does Well

1. Excellent documentation - The new document-data-extraction.md file is thorough and well-structured
2. Root cause transparency - Links to the upstream provider issue
3. Escape hatch provided - Acknowledges users can remove ignore_changes if they prefer
4. Targeted fix - Only ignores the problematic attribute, not the entire resource

⚠️ Primary Concerns

1. Human Error Risk (MEDIUM)

The documented workaround requires developers to:

1. Comment out line in module
2. Deploy
3. Remember to uncomment line

Risk: Developers will forget step 3, leaving the ignore block commented out indefinitely. This defeats the entire purpose of the fix.

2. No Drift Detection (LOW-MEDIUM)

Once ignore_changes is active, there's no way to verify that the actual AWS state matches what you intended. If someone manually changes blueprints in the console, Terraform won't catch it.

3. Blueprint Removal Behavior

If you remove a blueprint from config, will ignore_changes prevent it from being removed from AWS? Worth testing to avoid orphaned resources.

I've posted inline suggestions to improve robustness and documentation.

[sean-navapbc]

Additional Enhancement Suggestions

1. Add Example Blueprint File

Consider adding an example custom blueprint JSON file to help users understand the structure:

Create: infra/modules/document-data-extraction/examples/sample-blueprint.json

{
  "name": "invoice-extraction",
  "description": "Extract structured data from invoices",
  "schema": {
    "type": "object",
    "properties": {
      "invoice_number": {
        "type": "string",
        "description": "Invoice identifier"
      },
      "date": {
        "type": "string",
        "format": "date",
        "description": "Invoice date"
      },
      "total_amount": {
        "type": "number",
        "description": "Total invoice amount"
      }
    },
    "required": ["invoice_number", "date", "total_amount"]
  }
}

Then reference this in the documentation under the "Custom blueprints" section.

---

2. Optional: Create Verification Helper Script

For teams that frequently need to verify blueprint state, consider adding a script:

Create: scripts/verify-dde-blueprints.sh

#!/bin/bash
# Verify deployed DDE blueprints match Terraform configuration

set -euo pipefail

APP_NAME=${1:-}
ENVIRONMENT=${2:-}

if [[ -z "$APP_NAME" || -z "$ENVIRONMENT" ]]; then
  echo "Usage: $0 <app_name> <environment>"
  exit 1
fi

PROJECT_NAME="${APP_NAME}-${ENVIRONMENT}-project"

echo "Fetching deployed blueprints for $PROJECT_NAME..."
DEPLOYED_BLUEPRINTS=$(aws bedrock-data-automation describe-project \
  --project-name "$PROJECT_NAME" \
  --query 'customOutputConfiguration.blueprints[*].blueprintArn' \
  --output json | jq -r '.[] | select(. != null)' | sort)

echo ""
echo "Deployed Blueprints:"
echo "$DEPLOYED_BLUEPRINTS"
echo ""
echo "Compare with configured blueprints in:"
echo "  infra/$APP_NAME/app-config/env-config/document_data_extraction.tf"
echo ""
echo "⚠️  Note: Differences may exist due to ignore_changes in Terraform"

Make it executable: chmod +x scripts/verify-dde-blueprints.sh

This gives operators a quick way to check for drift without relying on Terraform.

[sean-navapbc]

Testing Recommendation

Before merging, please test the blueprint removal scenario to verify ignore_changes behavior:

Test Case: Blueprint Removal

1. Start with a DDE project that has 2-3 blueprints configured
2. Deploy successfully
3. Remove one blueprint from the blueprints config list
4. Run terraform plan (without commenting out ignore_changes)
5. Expected behavior: Plan should show no changes (blueprint removal ignored)
6. Comment out custom_output_configuration.blueprints line
7. Run terraform plan again
8. Expected behavior: Plan should now show the blueprint being removed

Why This Matters

If ignore_changes prevents blueprint removals from being applied (even when commented out), it could lead to:

• Orphaned blueprints consuming resources
• IAM policies granting access to blueprints that shouldn't exist
• Confusion when deployed state doesn't match configuration

If the test reveals issues with removals, consider documenting this specific limitation in the "Updating blueprints" section.

---

Related question: Does the awscc_bedrock_blueprint resource (lines 65-72) also need ignore_changes, or does it remain stable? The issue description only mentions the project resource.

[sean-navapbc]

Review Summary

✅ Review Complete - Posted 2 inline suggestions + 2 enhancement comments

Inline Suggestions (can be applied with one click):

1. Stronger warning in code - Adds reminder to uncomment after blueprint updates
2. Limitations & verification section - Documents tradeoffs and provides verification commands

Additional Recommendations (in comments):

3. Example blueprint file - Help users understand custom blueprint structure
4. Verification helper script - Quick drift detection without Terraform
5. Test blueprint removal - Verify ignore_changes doesn't block deletions

What I Like About This PR:

• Targets root cause (provider bug) with pragmatic workaround
• Documentation is thorough and well-organized
• Provides escape hatch for teams that don't need the silence
• Links to upstream issue for tracking

View the full review: #1028

[sean-navapbc]

I'm approving as is with recommendations below

[doshitan]

#1028 (comment)

1. Add Example Blueprint File

This already exists in the location mentioned in the existing docs: https://github.com/navapbc/template-infra/blob/main/infra/%7B%7Bapp_name%7D%7D/service/document-data-extraction-blueprints/template_blueprint.json

2. Optional: Create Verification Helper Script

Possibly as further mitigation in #1027 as a part of some automated detection and targeted state change if upstream doesn't fix the underlying issue soon.

#1028 (comment)

Related question: Does the awscc_bedrock_blueprint resource (lines 65-72) also need ignore_changes, or does it remain stable? The issue description only mentions the project resource.

It does not, but has some interactions with the BDA project that have been called out.