[New Resource]: aws_bedrockagentcore_online_evaluation_config

[thenom]

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the library.

Changes to Security Controls

No changes to security controls.

Description

Adds a new resource aws_bedrockagentcore_online_evaluation_config for managing Amazon Bedrock AgentCore Online Evaluation Configurations.

Online evaluation configurations continuously monitor agent performance by sampling live traffic from CloudWatch logs and applying evaluators (built-in or custom) to assess agent quality in production.

This resource supports:

• CloudWatch log group data source configuration
• Up to 10 evaluators (built-in and custom)
• Sampling configuration (0.01% to 100%)
• Trace filters with string, boolean, and numeric values
• Session timeout configuration
• Execution status control (ENABLED/DISABLED)
• Tagging

Relations

Closes #47210

References

• AWS Documentation: Online Evaluations
• API Reference: CreateOnlineEvaluationConfig

Output from Acceptance Testing

Unable to run acceptance tests in our environment. Tests have been written following existing patterns in the bedrockagentcore package and we'd appreciate the maintainers running them on their side.

% make testacc TESTS=TestAccBedrockAgentCoreOnlineEvaluationConfig PKG=bedrockagentcore

[github-actions]

Community Guidelines

This comment is added to every new Pull Request to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

• Please vote on this Pull Request by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
• Please see our prioritization guide for additional information on how the maintainers handle prioritization.
• Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Pull Request and do not help prioritize the request.

Pull Request Authors

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[thenom]

Hi, just wondered if there was any movement on this one?

[lonmar65]

Is there any movement on this? This will be really useful for a project I'm currently working on.

[ewbankkit]

Keep getting

% make testacc TESTARGS='-run=TestAccBedrockAgentCoreOnlineEvaluationConfig_basic' PKG=bedrockagentcore
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
make: Validating schemas
ok      github.com/hashicorp/terraform-provider-aws/internal/provider/sdkv2     5.807s
ok      github.com/hashicorp/terraform-provider-aws/internal/provider/framework 5.870s
make: Running acceptance tests on branch: 🌿 HEAD 🌿...
TF_ACC=1 go1.26.3 test ./internal/service/bedrockagentcore/... -v -count 1 -parallel 20  -run=TestAccBedrockAgentCoreOnlineEvaluationConfig_basic -timeout 360m -vet=off -buildvcs=false
2026/05/20 15:44:44 Creating Terraform AWS Provider (SDKv2-style)...
2026/05/20 15:44:44 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
    online_evaluation_config_test.go:46: Step 1/2 error: Error running apply: exit status 1

        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test,
          on main_gen.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_6228945248530521769"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"

--- FAIL: TestAccBedrockAgentCoreOnlineEvaluationConfig_basic (18.32s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/bedrockagentcore   24.270s
FAIL
make: *** [testacc] Error 1

including in the AWS Console.

[ewbankkit]

That's a CloudWatch Transaction Search thing: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Enable-TransactionSearch.html.
Luckily we can now configure that via #47961 & #47975.

[ewbankkit]

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccBedrockAgentCoreOnlineEvaluationConfig_' PKG=bedrockagentcore ACCTEST_PARALLELISM=3
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
make: Validating schemas
ok      github.com/hashicorp/terraform-provider-aws/internal/provider/sdkv2     (cached)
ok      github.com/hashicorp/terraform-provider-aws/internal/provider/framework (cached)
make: Running acceptance tests on branch: 🌿 HEAD 🌿...
TF_ACC=1 go1.26.3 test ./internal/service/bedrockagentcore/... -v -count 1 -parallel 3  -run=TestAccBedrockAgentCoreOnlineEvaluationConfig_ -timeout 360m -vet=off -buildvcs=false
2026/05/20 16:58:47 Creating Terraform AWS Provider (SDKv2-style)...
2026/05/20 16:58:47 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_basic
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_basic
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_List_basic
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_List_basic
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_List_includeResource
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_List_includeResource
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_disappears
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_disappears
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_tags
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_tags
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_update
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_update
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_executionStatus
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_executionStatus
=== RUN   TestAccBedrockAgentCoreOnlineEvaluationConfig_filters
=== PAUSE TestAccBedrockAgentCoreOnlineEvaluationConfig_filters
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_basic
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_disappears
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_executionStatus
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_disappears (26.82s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_filters
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_basic (39.57s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_update
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_filters (25.04s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_List_includeResource
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_executionStatus (55.03s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_basic
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_update (37.12s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_includeResource (27.03s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_List_basic
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_basic (29.32s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride
=== NAME  TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride
    acctest.go:1731: skipping test for aws/us-west-2: Error running apply: exit status 1

        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test[1],
          on main.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_7451763984349650737_1"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"


        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test[0],
          on main.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_7451763984349650737_0"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"

--- SKIP: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride (15.64s)
=== CONT  TestAccBedrockAgentCoreOnlineEvaluationConfig_tags
=== NAME  TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride
    acctest.go:1731: skipping test for aws/us-west-2: Error running apply: exit status 1

        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test,
          on main_gen.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_3239526875872362415"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"

--- SKIP: TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride (15.53s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_basic (27.10s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_tags (53.49s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/bedrockagentcore   151.579s

[thenom]

Thats great, thanks for that @ewbankkit, Do you need me to look into those test failures?

[ewbankkit]

@thenom No need. Those errors are because the alternate Region does not have Transaction Search enabled (and we are skipping tests that return that error). Aiming to get this one merged for nest week's release.

[jar-b]

LGTM 🎉

% make t K=bedrockagentcore T=TestAccBedrockAgentCoreOnlineEvaluationConfig_
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
make: Running acceptance tests on branch: 🌿 f-agentcore-evaluations-config 🌿...
TF_ACC=1 go1.26.3 test ./internal/service/bedrockagentcore/... -v -count 1 -parallel 20 -run='TestAccBedrockAgentCoreOnlineEvaluationConfig_'  -timeout 360m -vet=off
2026/05/22 10:32:25 Creating Terraform AWS Provider (SDKv2-style)...
2026/05/22 10:32:25 Initializing Terraform AWS Provider (SDKv2-style)...

=== NAME  TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride
    acctest.go:1731: skipping test for aws/us-west-2: Error running apply: exit status 1

        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test,
          on main_gen.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_6103120789618582800"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"

=== NAME  TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride
    acctest.go:1731: skipping test for aws/us-west-2: Error running apply: exit status 1

        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test[0],
          on main.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_7281814430189491881_0"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"


        Error: creating Bedrock AgentCore Online Evaluation Config

          with aws_bedrockagentcore_online_evaluation_config.test[1],
          on main.tf line 4, in resource "aws_bedrockagentcore_online_evaluation_config" "test":
           4: resource "aws_bedrockagentcore_online_evaluation_config" "test" {

        ID: "tf_acc_test_7281814430189491881_1"
        Cause: operation error Bedrock AgentCore Control:
        CreateOnlineEvaluationConfig, , ValidationException: One or more specified
        log groups do not exist"

--- SKIP: TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_regionOverride (20.85s)
--- SKIP: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_regionOverride (20.86s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_includeResource (37.78s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_disappears (41.00s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_filters (41.48s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_List_basic (43.45s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_basic (45.81s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_update (54.32s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_Identity_basic (56.69s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_executionStatus (71.80s)
--- PASS: TestAccBedrockAgentCoreOnlineEvaluationConfig_tags (74.36s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/bedrockagentcore   81.537s

[jar-b]

Thanks for your contribution, @thenom! 👍

[github-actions]

Warning

This Issue has been closed, meaning that any additional comments are much easier for the maintainers to miss. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.