`terraform test`: dump the complete state file when a destroy operation fails.

[lorengordon]

Terraform Version

terraform 1.6

Use Cases

I'm interested in using the test command, in the upcoming terraform 1.6 release. We currently have a similar workflow using the tftest pytest library. One thing we've found useful is that tftest leaves the state behind when the destroy action fails. This gives us the opportunity to investigate what happened, update the config or permissions if needed, and re-execute the destroy to cleanup any lingering resources.

Currently, the test command maintains state only in memory. I am curious what this means for destroy-time errors. Do the resources just remain lingering? Is there any way to inspect the failure and cleanup?

Attempted Solutions

n/a

Proposal

Perhaps on error, write the state to a file?

References

No response

[liamcervante]

Hi @lorengordon, thanks for the feedback.

Currently, terraform test will provide a set of diagnostics that describes any state left behind if the destroy action fails. I do like the idea of potentially writing out a file. I will investigate this, and see how practical it is.

Thanks again!

[liamcervante]

I've considered this a bit further and I will say that we won't see any changes here for the launch of v1.6. The test command at the moment doesn't have the ability to recover or use state files, so printing out the state file won't really give additional information over the diagnostics that are already produced. It's also not straightforward how Terraform should handle state files from alternate modules and/or registry modules and this needs more thought. We'd want to do this properly instead of trying to rush something in place for v1.6.

I did quickly implement #33809 which means the test command will return a non-zero return code if the destroy operation fails, making it easier to detect as a fail case.

I will leave this issue open though, as we are investigating approaches to custom state management within test files and it may be that with fine-grained state management we can actually use any dumped state files to make the tidy up process easier. Potentially, we'll see this arriving in Terraform v1.7.

[lorengordon]

I suppose I was figuring I'd be able to use the state file locally, with just terraform destroy. But there are probably some details of how test works that I haven't considered.

[liamcervante]

Yeah, I think that works for the very simple case. But it gets complicated with edge cases as the test command could be tearing down infrastructure that it has created based on other modules that are held in the registry, terraform test could be executing within a CI pipeline or in TFC, there could already be state files in the local directory already.

I really like this idea as a feature, but I'd want to do it properly and realistically that means giving it a full development cycle, and as a bonus that means we can make it work with the other state operations we've had as suggestions which just means it'll be even better when it does launch. All changes we make have to be backwards/forwards compatible and I wouldn't want to lock this out of future integrations because we just rushed and solved the simple case.

[lorengordon]

Oh yes, I don't think it's that urgent for us. I honestly wasn't expecting an implementation of this request for the 1.6 release. Sorry if I wasn't clear on that. Really appreciate the work and attention you've put into the question. Thanks again!

[albertorm95]

Will there be an intention on this, when you have to manually destroy ~30 aws resources terraform test stop being easy to use

[liamcervante]

Hi @albertorm95, terraform test should already be attempting to destroy the resources it creates - is there a particular reason that this isn't working for you?

I'd imagined what we implemented here would be about making a rare edge case easier to recover from rather than something that would be used as part of a regular test cycle. Ideally, Terraform would successfully clean up on every test execution rather than it failing and dumping the state file every time.

Maybe you could post your use case into the forums, and we could try and get the destroy operation to execute successfully?

[albertorm95]

Hello! @liamcervante

What about an scenario where the test is executed on a remote system and for any reason the system failed before destroying or in the destroy process, those resources will be there without a state that represent them

If there is state file on the test we will able to reference it and execute a manual destroy 🤔