integ-tests: assertion chaining do not fail as expected

[nmussy]

Describe the bug

The various implementations of IApiCall (awsApiCall, invokeFunction and httpApiCall) are used to run assertions on integration test after their deployment. These API or HTTP calls can be reused to run multiple assertions over a single result, either by chaining them or storing the result in a variable and reusing it. Unfortunately, it seems that only the last assertion evaluated for an IApiCall instance seem to determine whether the assertion will pass or fail the test.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Each assertions made to a IApiCall construct should be evaluated, and the tests should only pass if every one is valid

Current Behavior

When running multiple assertions on the same IApiCall, only the last assertion determines whether it fails or passes.

Reproduction Steps

Given the following integration test, based off of integ.assertions.ts:

import { App, CfnResource, Stack } from 'aws-cdk-lib';
import { ExpectedResult, IntegTest } from '../../../lib';

const app = new App();
const stack = new Stack(app, 'Assertions');

const ssmParameter = new CfnResource(stack, 'Utf8Parameter', {
  type: 'AWS::SSM::Parameter',
  properties: {
    Type: 'String',
    Value: 'ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ!"#¤%&/()=?`´^*+~_-.,:;<>|',
  },
});

const integ = new IntegTest(app, 'AssertionsTest', {
  testCases: [stack],
});

The following assertions fail, as expected:

integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));

integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));

integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
  .expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }))
  .expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));
  
integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
  .assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'))
  .assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));

But the following assertions pass, as their last expect/assertAtPath calls pass:

integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
  .expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }));

integ.assertions.awsApiCall('SSM', 'getParameter', {
  Name: ssmParameter.ref,
  WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
  .assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'));

We can see the same issue occurring with httpApiCall and invokeFunction, as these assertions pass erroneously:

integ.assertions.httpApiCall(
  'https://httpbin.org/status/403',
).expect(ExpectedResult.objectLike({ invalid: 'object' }))
  .expect(ExpectedResult.objectLike({ status: 403 }));

const targetFunc = new lambda.Function(stack, 'TargetFunc', {
  code: lambda.Code.fromInline('exports.handler = async (event, context) => { return { foo: "bar" }; };'),
  handler: 'index.handler',
  runtime: lambda.Runtime.NODEJS_LATEST,
});

integ.assertions.invokeFunction({
  functionName: targetFunc.functionName,
  invocationType: InvocationType.EVENT,
  payload: JSON.stringify({ days: 1 }),
}).expect(ExpectedResult.objectLike({ invalid: 'value' }))
  .expect(ExpectedResult.objectLike({ StatusCode: 202 }))
  .waitForAssertions({
    interval: Duration.seconds(30),
    totalTimeout: Duration.minutes(90),
  });

Possible Solution

The brute and safe solution would be to render these assertion functions effective singletons, by throwing an error if the instance of the IApiCall construct is used more than once. This would allow us to at least root out all potential silently failing tests.

Additional Information/Context

We have multiple instances of multiple assertions being made to a single IApiCall, such as:

aws-cdk/packages/@aws-cdk-testing/framework-integ/test/aws-iam/test/integ.imported-role.ts

Lines 65 to 80 in 034679a

	const test = new integ.IntegTest(app, 'ImportedRoleTest', {
	testCases: [roleStack, thirdStack],
	assertionStack,
	});
	test.assertions
	.awsApiCall('IAM', 'listRolePolicies', { RoleName: role.roleName })
	.assertAtPath('PolicyNames.0', integ.ExpectedResult.stringLikeRegexp('^Policyintegiamimportedrole1Role.{8}$'))
	.assertAtPath('PolicyNames.1', integ.ExpectedResult.stringLikeRegexp('^Policyintegiamimportedrole1Rolex+.{8}$'))
	.assertAtPath('PolicyNames.2', integ.ExpectedResult.stringLikeRegexp('^Policyintegiamimportedrole2Role.{8}$'))
	.assertAtPath('PolicyNames.3', integ.ExpectedResult.stringLikeRegexp('^Policyintegiamimportedrole2Roley+.{8}$'));
	test.assertions
	.awsApiCall('IAM', 'listAttachedRolePolicies', { RoleName: roleToBeImported.roleName })
	.assertAtPath('AttachedPolicies.0.PolicyName', integ.ExpectedResult.stringLikeRegexp('^MyCustomManagedPolicy[0-9]$'))
	.assertAtPath('AttachedPolicies.1.PolicyName', integ.ExpectedResult.stringLikeRegexp('^MyCustomManagedPolicy[0-9]$'));

This might be a regression, but I cannot confirm that this was ever working properly

CDK CLI Version

v2.173.2 (034679a)

Framework Version

No response

Node.js Version

v20.11.1

OS

macOS

Language

TypeScript

Language Version

TypeScript ~5.5.2 (internal version)

Other information

Originally reported by @brandondahler in #32575 (comment)

[khushail]

Looks like CDK Team is already looking into this issue. Marking as appropriate for the traction by the team.

cc: @shikha372 , @HBobertz