sam validate may break when combined with other transforms

[gdw2]

Description

I'm running two transforms on my template, a custom transform and AWS::Serverless. My first transform leverages syntax that breaks sam validation. While I don't necessarily expect sam to validate unordinary syntax, I would expect a way to bypass validation when running sam package. I supposed I may be able to deploy a template with the sam transform, but without using the sam cli.

Steps to reproduce

Run 'sam validate' on the following simplified template. Sam does not like resources composed of lists. This may not be valid sam template now, but it is after transformation.

AWSTemplateFormatVersion: "2010-09-09"
Transform: ['InlineJS-1_0', 'AWS::Serverless-2016-10-31']

Resources:
  'Test':
    - Name: Test1                                                                                                                                                                                                                                                                 - Name: Test2

Observed result

$ sam validate --debug --profile gwarner7 --region us-west-2 --template /home/gwarner/code/bep/modules/lono-inline/examples/greg-loop-js2.yaml
Using SAM Template at /home/gwarner/code/bep/modules/lono-inline/examples/greg-loop-js2.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2019-11-21 20:59:27 Found credentials in shared credentials file: ~/.aws/credentials
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam validate', 'duration': 1945, 'exitReason': 'AttributeError', 'exitCode': 255, 'requestId': 'b0f95160-41bb-4873-8732-a273b358843b', 'installationId': 'bed69871-b653-4c4f-97b2-27b52db02413', 'sessionId': '47aea76c-e669-4a7b-944e-4f385ea87235', 'executionEnvironment': 'CLI', 'pyversion': '3.7.3', 'samcliVersion': '0.31.1'}}]}
Telemetry response: 200
Traceback (most recent call last):
  File "/home/gwarner/.pyenv/versions/greg-sam/bin/sam", line 10, in <module>
    sys.exit(cli())
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 93, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/commands/validate/validate.py", line 25, in cli
    do_cli(ctx, template)  # pragma: no cover
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/commands/validate/validate.py", line 45, in do_cli
    validator.is_valid()
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/commands/validate/lib/sam_template_validator.py", line 57, in is_valid
    self._replace_local_codeuri()
  File "/home/gwarner/.pyenv/versions/3.7.3/envs/greg-sam/lib/python3.7/site-packages/samcli/commands/validate/lib/sam_template_validator.py", line 85, in _replace_local_codeuri
    resource_type = resource.get("Type")
AttributeError: 'list' object has no attribute 'get'

Expected result

Template to validate.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux
2. sam --version:
   SAM CLI, version 0.31.1

[jfuss]

@gdw2 Thanks for the report, this is super interesting. Right now, we are assuming that resources have a Type property [1]. Looks like your custom resource does not have this, which then fails without SAM CLI. Is this template deployable? I can't find documentation but I thought resources in the template need to have specific properties (Type being one of them).

[1]

LogicalId:
  Type: Blah
  Properties:
    ...

[kanaka]

The final template needs to be deployable, yes. But CFN macro functionality means that as long as the final output of the transform macros is valid, then it is deployable. For example, I've done something like the following:

AWSTemplateFormatVersion: "2010-09-09"
Transform: ['LispEvaluator']
Lisp: "Lisp code"

The LispEvaluator macro evaluates the Lisp code and returns a modified template with the "Lisp" top level key removed, and a valid Resources section added.

So for the SAM use-case, I think that the template validation is happening too early. I think the checks should either happen right before the Serverless macro is executed or after all transforms are executed. Or probably most likely it would be some mix of both: any validations needed for Serverless should definitely happen prior to the Serverless transform, any validations that are specifically for CFN but not needed by Serverless should happen after all transforms. But certainly it can't be assumed that the starting state of the template is valid CFN template given that early macros can completely transform the structure of the input.

[iancward]

We have a SAM template that pulls in the API Swagger docs (DefinitionBody) from an external file (using Fn::Transform for AWS::Include) and when I attempt to run sam validate, I get an error that the resources is invalid because validate doesn't seem to know how to make sense of the Include transform.

This is using SAM CLI, version 1.24.0.

Error: [InvalidResourceException('ApiGateway', "Unable to add Cors configuration because 'DefinitionBody' does not contain a valid Swagger definition.")] ('ApiGateway', "Unable to add Cors configuration because 'DefinitionBody' does not contain a valid Swagger definition.")

[morganpartee]

Hey just wanted to add, this is still a problem in SAM 1.35.0.

In my case I have a custom transform for making UTC eventbridge rules from local time cron statements like:

      Events:
        Weekday:
          ChicagoTimer: cron(0 9-17 ? * MON-FRI *)
        Weekend:
          ChicagoTimer: cron(0 9 ? * SAT,SUN *)

Which deploys correctly, but fails sam validate. Also when running sam build/deploy, custom transforms aren't rendered into the .aws-sam/build/template.yaml file, which makes troubleshooting a little more annoying.

sam validate fails with:

Error: [InvalidResourceException('<FunctionName>', 'Event with id [Weekday] is invalid. Resource dict has missing or invalid value for key Type. Event Type is: None.')] ('stuckLoanChecker', 'Event with id [Weekday] is invalid. Resource dict has missing or invalid value for key Type. Event Type is: None.')

[sriram-mv]

Is this still an issue? I know there was a merge within the serverless-application-model repo wrt order of transforms logic a while ago.