Skip to content

BDA module test#237

Closed
damianj wants to merge 51 commits intomainfrom
bda-module
Closed

BDA module test#237
damianj wants to merge 51 commits intomainfrom
bda-module

Conversation

@damianj
Copy link

@damianj damianj commented Nov 6, 2025

Ticket

navapbc/template-infra#986

Changes

Adding BDA module for testing before integration into template-infra.

Context for reviewers

This is just to create the BDA resources, i.e., the project and blueprints, any integrations with how the project will be triggered are not in scope for this PR.

Testing

2026-01-23

platform-test-bda-invocation-p-237.mov

2026-02-19

platform-test-bda-invocation-p-237-2026-02-19.mov

Preview environment for app

♻️ Environment destroyed ♻️

Preview environment for app-nextjs

♻️ Environment destroyed ♻️

Preview environment for app-rails

♻️ Environment destroyed ♻️

Preview environment for app-flask

♻️ Environment destroyed ♻️

Copy link
Contributor

@doshitan doshitan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good start! Some refactoring and questions throughout, but I think the main thing we should do is ensure the module exports the necessary info such that it could be used in a service.

Also, could you post some examples of the changes working in the “Testing” section of the PR description? Like a short video from the AWS console or whatever of calling the created BDA project with example input and seeing the output.

We'll want to avoid product/vendor-specific names for modules (following the style guide). I couldn't think of anything better than document_data_extraction, so that's what I'm suggesting here, but open to other options ("document" doesn't obviously capture the potential scope of images/videos, but feels like the most common usecase).

Refactorings to that end:

@@ -0,0 +1,48 @@
module "bda_input_bucket" {
source = "../../modules/storage"
name = "bda-test-input-app-flask"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we create new buckets, the naming should follow the existing convention and uniqueness constraints, which the simpliest way would probably be something like:

Suggested change
name = "bda-test-input-app-flask"
name = "${local.bucket_name}-bda-input"

Re-using the existing storage bucket name (which should handle most of the conventions) and tack on the specific purpose. Ideally we'd use it something more clear like -document-data-extraction-, but that's like 40% the max length for a bucket name, so should probably default to something shorter.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@doshitan We should use separate buckets. Rather than the existing storage bucket. Reasons here:

@doshitan We should 100% use separate input/output buckets for BDA. Reasons include, but are not limited to:

  1. Physical separation of resources

    • Input bucket: BDA read-only
    • Output bucket: BDA write-only
  2. Retention policies

    • Input bucket: longer retention for compliance/regulatory reasons
    • Output bucket: shorter retention for processed results
  3. Cost/storage management

    • Input bucket: transition to Infrequent Access after 30 days, Glacier > 90 days, etc.
    • Output bucket: standard storage class with n-day automatic deletion
  4. Monitoring/alerts:

    • Per-bucket S3 metrics and CloudWatch alerts

- Add enable_document_data_extraction boolean
- Add environment variables and bucket policies to service
- Add additional outputs for BDA project/blueprint
Copy link
Contributor

@doshitan doshitan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like most of my previous in-line comments haven't been addressed. I think most critically could use an answer to #237 (comment)

- Move infra/modules/document-data-extraction/resources/bedrock-data-automation/* to infra/modules/document-data-extraction/resources
- Rename infra/app-flask/service/blueprints to document-data-extraction-blueprints
- Remove bda_ prefix from infra/modules/document-data-extraction/resources/variables.tf and infra/app-flask/service/document_data_extraction.tf
- Update infra/app-flask/app-config/env-config/document_data_extraction.tf name and path
- Added local.prefix to bucket names and BDA project name
- Updated environment variables to use prefixed bucket names
Copy link
Contributor

@doshitan doshitan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can tighten up the document data extract module interface and outputs a bit. And we should sync on the bucket setup for BDA in general.

There are still some older unresolved comments:

override_configuration = {
document = {
splitter = {
state = var.override_config_state
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of the variable and its use seem a little inconsistent. Should like override_configuration = var.override_config?

Also what is the purpose/function of override_configuration? Like is there a less BDA-specific name we could give the module variable?

Copy link

@laurencegoolsby laurencegoolsby Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- Create KMS key
- Remove KMS configuration parameters from module interface
- Add Bedrock Data Automation access_policy_arn output for service integration
- Update blueprints_map to use map(string) tags instead of complex objects
- Remove enabled_blueprint logic in favor of reading blueprints directory
- Update README
Copy link
Contributor

@doshitan doshitan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should post some evidence of the setup working. Probably easiest is to add a little test function in the app-flask code that invokes the BDA project with a file uploaded (like via the AWS console) to a location BDA can read from?

for blueprint in fileset(local.document_data_extraction_config.blueprints_path, "*") :
split(".", blueprint)[0] => {
schema = file("${local.document_data_extraction_config.blueprints_path}/${blueprint}")
type = "DOCUMENT"
Copy link
Contributor

@doshitan doshitan Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As opposed to image, video, or audio right? And it looks like you can only have one custom blueprint for these other types right? Have we found value in custom blueprints for any of these?

And so not in scope, but if projects wanted to have custom blueprints for multiple kinds of images/audio, they'd have to create multiple BDA projects, hmm.

But JPG, PNG, TIFF are counted as "documents" not "images" right?

Copy link

@laurencegoolsby laurencegoolsby Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@doshitan All of our blueprints were defined with a DOCUMENT modality rather than IMAGE.

JPG, PNG, and TIFF documents can match to blueprints with either a DOCUMENT or IMAGE modality. In other words, if you upload a jpg, BDA can choose a "document" blueprint if it's the best fit.

File Types Default Routing Behavior
PNG Semantic classifier; either Image or Document
JPEG Semantic classifier; either Image or Document
PDF, TIFF Document
MP4, MOV Video
AMR, FLAC, M4A, MP3, OGG, WEBM, WAV Audio

https://docs.aws.amazon.com/bedrock/latest/userguide/bda-routing-enablement.html?utm_source=chatgpt.com#bda-standard-routing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it looks like there's also a difference in behavior using InvokeDataAutomationAsync and InvokeDataAutomation, the latter rejecting JPG/PNG that get detected as DOCUMENT? Feels like that nuance should be captured somewhere. infra/modules/document-data-extraction/README.md doesn't feel quite right, but a usage guide would be good. I guess for now the usage guide will be the API app we build to use this module.

Also let's put a short comment here summarizing type thing? Something like "JPG/PNG can be processed as DOCUMENT or IMAGE types, but IMAGE types can only have a single custom blueprint so generally the blueprints will be for the DOCUMENT type"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@doshitan Comment added!

…ment data extraction integration

- Create /test-document-data-extraction endpoint in Flask app
- Update boto3/botocore to support latest BDA API (1.38.0+)
- Fix Terraform outputs to include bucket name prefixes
- Add DDE environment variables to local.env template
- Consolidate blueprints_path and aws_managed_blueprints into single blueprints list
- Support mixed list of file paths and ARNs in blueprints variable
- Add glob pattern expansion in service layer for blueprint files
- Remove InvokeModel and InvokeModelWithResponseStream permissions (not needed for BDA)
@laurencegoolsby
Copy link

@doshitan
Addressed feedback:

  • Consolidated blueprints_path and aws_managed_blueprints into single blueprints list
  • Removed InvokeModel/InvokeModelWithResponseStream permissions (not needed for BDA)
  • Updated README
  • Tested - BDA still working with refactored blueprint configuration

Re: Storage module - BDA requires storage module changes regardless of encryption approach. Awaiting your feedback on how to proceed.

@laurencegoolsby
Copy link

CI app-flask PR Environment Checks / update / Update environment (pull_request)Failing after 21s

Currently receiving an AccessDeniedException for Bedrock Blueprint operations. This appears to be related to missing bedrock and cloudformation permissions in the CI/CD role. These services were added to infra/project-config/aws_services.tf in the latest commit, but perhaps project-config changes need to be applied before the service deployment will succeed?

cc @doshitan

@doshitan
Copy link
Contributor

@laurencegoolsby the changes from #238 need re-applied against the account layer.

@laurencegoolsby
Copy link

@laurencegoolsby the changes from #238 need re-applied against the account layer.

Updated!

@laurencegoolsby laurencegoolsby self-requested a review February 17, 2026 16:42
Copy link
Contributor

@doshitan doshitan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Service test still look to be failing after re-run: https://github.com/navapbc/platform-test/actions/runs/21992515262/job/64041863229?pr=237

As well as warning:

  TestService 2026-02-18T17:48:59Z logger.go:67: ╷
  TestService 2026-02-18T17:48:59Z logger.go:67: │ Warning: Reference to undefined provider
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │   on document_data_extraction.tf line 33, in module "dde_input_bucket":
  TestService 2026-02-18T17:48:59Z logger.go:67: │   33:     aws = aws.dde
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ There is no explicit declaration for local provider name "aws" in
  TestService 2026-02-18T17:48:59Z logger.go:67: │ module.dde_input_bucket, so Terraform is assuming you mean to pass a
  TestService 2026-02-18T17:48:59Z logger.go:67: │ configuration for "hashicorp/aws".
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ If you also control the child module, add a required_providers entry named
  TestService 2026-02-18T17:48:59Z logger.go:67: │ "aws" with the source address "hashicorp/aws".
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ (and 2 more similar warnings elsewhere)

@laurencegoolsby
Copy link

Service test still look to be failing after re-run: https://github.com/navapbc/platform-test/actions/runs/21992515262/job/64041863229?pr=237

As well as warning:

  TestService 2026-02-18T17:48:59Z logger.go:67: ╷
  TestService 2026-02-18T17:48:59Z logger.go:67: │ Warning: Reference to undefined provider
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │   on document_data_extraction.tf line 33, in module "dde_input_bucket":
  TestService 2026-02-18T17:48:59Z logger.go:67: │   33:     aws = aws.dde
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ There is no explicit declaration for local provider name "aws" in
  TestService 2026-02-18T17:48:59Z logger.go:67: │ module.dde_input_bucket, so Terraform is assuming you mean to pass a
  TestService 2026-02-18T17:48:59Z logger.go:67: │ configuration for "hashicorp/aws".
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ If you also control the child module, add a required_providers entry named
  TestService 2026-02-18T17:48:59Z logger.go:67: │ "aws" with the source address "hashicorp/aws".
  TestService 2026-02-18T17:48:59Z logger.go:67: │ 
  TestService 2026-02-18T17:48:59Z logger.go:67: │ (and 2 more similar warnings elsewhere)

@doshitan

CI test is failing with ECS service stability timeout:

TestService 2026-02-19T16:20:22Z logger.go:67: Apply complete! Resources: 74 added, 0 changed, 0 destroyed.
TestService 2026-02-19T16:20:22Z logger.go:67:
TestService 2026-02-19T16:20:22Z logger.go:67: Outputs:
TestService 2026-02-19T16:20:22Z logger.go:67:
TestService 2026-02-19T16:20:22Z logger.go:67: application_log_group = "service/t-1vt6i8-app-flask-dev"
TestService 2026-02-19T16:20:22Z logger.go:67: application_log_stream_prefix = "t-1vt6i8-app-flask-dev"
TestService 2026-02-19T16:20:22Z logger.go:67: image_tag = "b4adda952bd5a6511fb60c95b13f103919a88a54"
TestService 2026-02-19T16:20:22Z logger.go:67: migrator_role_arn = "arn:aws:iam::430004246987:role/t-1vt6i8-app-flask-dev-migrator"
TestService 2026-02-19T16:20:22Z logger.go:67: migrator_username = "migrator"
TestService 2026-02-19T16:20:22Z logger.go:67: service_cluster_name = "t-1vt6i8-app-flask-dev"
TestService 2026-02-19T16:20:22Z logger.go:67: service_endpoint = "https://t-1vt6i8-app-flask-dev-140916984.us-east-1.elb.amazonaws.com"
TestService 2026-02-19T16:20:22Z logger.go:67: service_name = "t-1vt6i8-app-flask-dev"
Wait for service to be stable
TestService 2026-02-19T16:20:22Z logger.go:67: Running command aws with args [ecs wait services-stable --cluster t-1vt6i8-app-flask-dev --services t-1vt6i8-app-flask-dev]
TestService 2026-02-19T16:30:25Z logger.go:67:
TestService 2026-02-19T16:30:25Z logger.go:67: Waiter ServicesStable failed: Max attempts exceeded
command.go:33:
Error Trace: /home/runner/go/pkg/mod/github.com/gruntwork-io/mailto:terratest@v0.48.2/modules/shell/command.go:33
/home/runner/work/platform-test/platform-test/infra/test/infra_test.go:61
/home/runner/work/platform-test/platform-test/infra/test/infra_test.go:53
Error: Received unexpected error:
error while running command: exit status 255;
Waiter ServicesStable failed: Max attempts exceeded
Test: TestService

The manual deployment in p-237 workspace is healthy and running. This seems to suggest the CI failure is independent from the PR. Any chance this is related to the quota issues mentioned in Slack or is it something else entirely?


Re: the Terraform provider warning - it does not appear to be a blocking issue as the deploy succeeded. I can add an explicit AWS provider declaration if desired.

@doshitan
Copy link
Contributor

The manual deployment in p-237 workspace is healthy and running. This seems to suggest the CI failure is independent from the PR. Any chance this is related to the quota issues mentioned in Slack or is it something else entirely?

I'd re-run the infra tests and watch the logs/deployment events for the infra test.

Re: the Terraform provider warning - it does not appear to be a blocking issue as the deploy succeeded. I can add an explicit AWS provider declaration if desired.

Yes, I think we can a required_providers block to modules/storage. We haven't typically done it, but I don't think the extra explicitness really hurts. Alternatively modules/storage could take a region param, but we don't typically do that either. Being explicit with required provider feels better.

@laurencegoolsby
Copy link

laurencegoolsby commented Feb 19, 2026

The manual deployment in p-237 workspace is healthy and running. This seems to suggest the CI failure is independent from the PR. Any chance this is related to the quota issues mentioned in Slack or is it something else entirely?

I'd re-run the infra tests and watch the logs/deployment events for the infra test.

Re: the Terraform provider warning - it does not appear to be a blocking issue as the deploy succeeded. I can add an explicit AWS provider declaration if desired.

Yes, I think we can a required_providers block to modules/storage. We haven't typically done it, but I don't think the extra explicitness really hurts. Alternatively modules/storage could take a region param, but we don't typically do that either. Being explicit with required provider feels better.

  1. Updated infra/modules/document-data-extraction/resources/providers.tf to include required_providers.
  2. Added screenshot from 2/19/2026

@doshitan
Copy link
Contributor

Yes, I think we can a required_providers block to modules/storage. We haven't typically done it, but I don't think the extra explicitness really hurts. Alternatively modules/storage could take a region param, but we don't typically do that either. Being explicit with required provider feels better.

Updated infra/modules/document-data-extraction/resources/providers.tf to include required_providers.

As mentioned, modules/storage needs the provider block. Warning still present:

TestService 2026-02-19T20:35:39Z logger.go:67: ╷
  TestService 2026-02-19T20:35:39Z logger.go:67: │ Warning: Reference to undefined provider
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │   on document_data_extraction.tf line 33, in module "dde_input_bucket":
  TestService 2026-02-19T20:35:39Z logger.go:67: │   33:     aws = aws.dde
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ There is no explicit declaration for local provider name "aws" in
  TestService 2026-02-19T20:35:39Z logger.go:67: │ module.dde_input_bucket, so Terraform is assuming you mean to pass a
  TestService 2026-02-19T20:35:39Z logger.go:67: │ configuration for "hashicorp/aws".
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ If you also control the child module, add a required_providers entry named
  TestService 2026-02-19T20:35:39Z logger.go:67: │ "aws" with the source address "hashicorp/aws".
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ (and one more similar warning elsewhere)

And is the testing up-to-date/have we validated the functionality? The current demo looks to be from January?

But if those are all good, should update the template PR to get ready to merge.

@laurencegoolsby
Copy link

laurencegoolsby commented Feb 20, 2026

Yes, I think we can a required_providers block to modules/storage. We haven't typically done it, but I don't think the extra explicitness really hurts. Alternatively modules/storage could take a region param, but we don't typically do that either. Being explicit with required provider feels better.

Updated infra/modules/document-data-extraction/resources/providers.tf to include required_providers.

As mentioned, modules/storage needs the provider block. Warning still present:

TestService 2026-02-19T20:35:39Z logger.go:67: ╷
  TestService 2026-02-19T20:35:39Z logger.go:67: │ Warning: Reference to undefined provider
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │   on document_data_extraction.tf line 33, in module "dde_input_bucket":
  TestService 2026-02-19T20:35:39Z logger.go:67: │   33:     aws = aws.dde
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ There is no explicit declaration for local provider name "aws" in
  TestService 2026-02-19T20:35:39Z logger.go:67: │ module.dde_input_bucket, so Terraform is assuming you mean to pass a
  TestService 2026-02-19T20:35:39Z logger.go:67: │ configuration for "hashicorp/aws".
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ If you also control the child module, add a required_providers entry named
  TestService 2026-02-19T20:35:39Z logger.go:67: │ "aws" with the source address "hashicorp/aws".
  TestService 2026-02-19T20:35:39Z logger.go:67: │ 
  TestService 2026-02-19T20:35:39Z logger.go:67: │ (and one more similar warning elsewhere)

And is the testing up-to-date/have we validated the functionality? The current demo looks to be from January?

But if those are all good, should update the template PR to get ready to merge.

  • Provider warnings resolved: Added required_providers blocks to both document-data-extraction and storage modules in platform-test and template-infra (#989)
  • Testing validated: Executed document POST test with fresh deployment. Screen recording added to description
  • Template PR updated: template-infra PR #989 now includes matching provider configuration changes.

laurencegoolsby added a commit to navapbc/template-infra that referenced this pull request Feb 23, 2026
Fixes #986

## Changes

Add a new `document-data-extraction` module that provides AWS Bedrock
Data Automation (BDA) resources for document data extraction workflows.

### DDE Module
- Add BDA project and profile resources
- Add custom blueprint support from JSON schemas
- Add AWS-managed blueprint support
- Add IAM access policies with bedrock permissions
- Add cross-region inference profile support
- Add standard and custom output configuration options

### Storage Module
- Uses KMS encryption (storage module provides the encryption key)

### Template Files
- Update service and env-config files to use new module interface
- Add `aws_managed_blueprints` parameter

## Important Notes
- **BDA uses its own internal service role** - This module does not
create a custom IAM role for BDA
- **S3 bucket encryption** - S3 buckets used with BDA should use the KMS
encryption key provided by the storage module
- **Lambda permissions** - Lambda functions invoking BDA must have S3
permissions for both input and output buckets directly attached to their
execution role

## Testing

Tested in navapbc/platform-test#237
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants