RFC: Dry runs and atomic tasks

[zonzujiro]

Hey, guys!

While working on generator I have faced two issues, which I want try to solve and I want to hear your opinion.

In general, it would be great to solve them only in my project, just to avoid situation where I can brake something in widely used util. :) But if this is not possible and you don't mind, I can try to provide couple features to node-plop.

Problems

1. When node-plop will run his list of actions and something will happen in the middle of modifications, it will throw error, we will have half-finished task and broken result.
2. During a development of the commands, we need to re-run generator and then remove generated files, which is annoying.

Possible solution

For both cases we can try to use some in-memory fs instead of real fs first, like memory-fs of memfs.

General logic can look like this:

1. Create copy of targetDir in memory
2. Run actions
3. In the end of operations copy result of all operations to real file system - replace files, like an idea.

Pros:
When something happens during running of tasks actual file system will stay the same and dry run will be also possible.

Cons:
I have never worked with above mentioned NPM packages and I am not sure, that they will cover all things node-plop needs. So probably if I will implement this feature in node-plop, it should be under the the --atomic probably :)

[crutchcorn]

I am so sorry it's taken me so long to get back to this. I got the email and then had a bunch of stuff hit me at once.

I like the general idea of memfs usage (as memory-fs is depreciated), but I guess my problem is that we'd run into memory problems when dealing with large groups of files. I mean, it's not the end of the world to lose the ability to do dry runs.

I know the original idea was to always use memfs, but I'd rather implement it only when --dry or some other option is passed. I'd like to keep the current behavior as-is

[zonzujiro]

@crutchcorn no problem! Thank you for an answer anyway :)

I know the original idea was to always use memfs, but I'd rather implement it only when --dry or some other option is passed. I'd like to keep the current behavior as-is

Well, it was my idea also, because --dry can have a lot of specific logic.

I have tried to implement robust POC in my free time and faced several issues:

If we just use memfs, then when node-plop will try to get templates from templateFiles it will not found them

So we should:

1. collect all files from templateFiles of all actions and put them into a memory file system
2. copy targetDir content of all actions into memory FS
3. and only then start... "plopping" (😃 )

Probably, the first bullet is not so important because we can only write the changes we need. I have faced this issue only because I have tried to implement dry run from outside the node-plop, so I don't have a possibility to insert something in the middle of the running of actions.

Dry runs, by default, will work only with standart actions

Inside custom actions, developers can use fs or other tools like jscodeshift or eslint and we can't control paths they are using.

So, saying if I have functionality like: generate file -> run eslint on it the last action will fail, files are not in the actual file system. And this thing makes dry runs almost meaningless.

So we need to provide a way to run custom actions on fake FS and memfs has volumes.
Maybe we can pass volume as an argument inside custom action and say to developer: "From this moment you on your own", WDYT? It can be part of NodePlopAPI actually.

Summary

1. We creating memfs volume
2. We iterate through actions and check the existence of target paths in the file system. If such a path has content - we should copy this content into the volume
3. We running actions using volume for writing
4. If we face custom actions, we pass volume as an argument and expecting, that developer will do his action on this volume, not on the real filesystem
5. When everything is done - we make a copy of the content of volume into the real filesystem,

[zonzujiro]

Also, it is more about transitive operations, not only dry run - if something during actions will fail, file system will be left untouched.

Dry runs is a bonus of this functionality. As well as possibility to use snapshots for testing

[zonzujiro]

@crutchcorn hey! Don't want to be pushy, but can you give me a short review on the proposal? :)

[crutchcorn]

So we should:

1. collect all files from templateFiles of all actions and put them into a memory file system
2. copy targetDir content of all actions into memory FS
3. and only then start... "plopping" (😃 )

This makes sense to me. I'm still hesitant on copying all files into memory, as I'm worried about memory limitations for larger projects, so I'd love to see an escape hatch if possible (like a flag that says --noAtomicOps or something to disbable memfs usage)

Custom Actions

I beleve that we can easily handle custom actions by appending our own custom fs into the NodePlopAPI API and creating documentation on how to use it and how Plop then handles it.

After all, it'll help us fully reflect normal usage in "dry" runs. That way, we can instruct custom actions to interface with plop.fs instead of just fs. The flags we pass to node-plop will handle if these are fs or memfs and if they mirror back to the real fs or not. This seems to match your thoughts of implementation as well

Regardless, I'll want to make sure that we have plenty of test coverage here

All in all, things look good as you've outlined them

[zonzujiro]

All this feature will be under the flag. Default behaviour will stay untouched.

Ok. Cool. Then I will start work when I will have free time.

Thank you!

[crutchcorn]

No no, thank YOU @zonzujiro! :D Your contribution (and discussion) is invaluable to the project :) Sorry for having taken so long to review

[crutchcorn]

Transfering this issue to https://github.com/plopjs/plop/ since we're migrating all of node-plop to that repo as a monorepo

[crutchcorn]

Re-opening as this issue is not complete