Open
Description
Context
After introducing enhanced validation in #1368 we have received a number of bug reports about false diagnostics:
- fix: Work around 'unreliable' input data for Registry modules #1456
- Resolve version correctly for backend schema #1453
- decoder: allow objects in
for_each
hcl-lang#333 - fix output block precondition in 1.2 schema terraform-schema#282
- schema: Fix schema merging for
terraform_remote_state
data source terraform-schema#272 - schema: Make
cloud
organization
optional terraform-schema#271 - (fix) decoder: Ensure empty objects & tuples still get collected as targets hcl-lang#330
- schema: Declare
DefaultValue
for relevant attributes terraform-schema#270 - schema: Introduce v1.3 proxy related fields for provisioner
connection
terraform-schema#269 - fix: Ensure
target_platform
is recognised in nestedconnection
block terraform-schema#268
and some more continue to trickle in:
config
attribute not expected interraform_remote_state
data source vscode-terraform#1606- Variable references are not recognised in depends_on vscode-terraform#1598
- provider block for github shows as invalid for an empty app_auth block vscode-terraform#1597
- Incorrectly flagging nested dynamic blocks as errors vscode-terraform#1594
- Variable with empty list causes "No declaration found" on submodules vscode-terraform#1592
Proposal
Instead of letting our users discover and report such cases of false diagnostics we can be more proactive and run our validation logic against publicly available repositories of Terraform code on GitHub, to perform a form of smoke testing of the validation logic.
This can in turn provide us with higher confidence in the validation logic and the schema, and detect at least some, if not most, of the bugs before they even reach users.
- Identify some popular repositories with Terraform code on GitHub and the Terraform Registry
- Create a script (Go code) which can run all existing validation logic
- Context-less validation (
SchemaModuleValidation
,SchemaVariablesValidation
) - Context-dependent validation (
ReferenceValidation
)
- Context-less validation (
- Run script against those repositories