Ability to Resume Progress After Unexpected Termination

[PseudoResonance]

DAIN is working very well for me on an AMD GPU, but my power went out suddenly, and now I am unable to resume progress, and the program starts over again, losing many hours of work.

Is there any way of resuming where it left off, or perhaps can a feature be added to allow the program to continue even after an unexpected termination?

[routineLife1]

May need to be an automated script

[PseudoResonance]

Yes, I was thinking maybe I could make a script to manually feed in the files, but also would it not be slower? Because doesn't this program have to initialize each time it starts up, so by calling it over and over for individual frames, it would have to reinitialize each time.

I was thinking maybe if there was a -start_number parameter, like with ffmpeg, then the program could be restarted manually at any index. It could also automatically check the output folder for the last known frame and start there.

EDIT: If anyone is wondering, the solution I am using at the moment is to move a small amount (maybe 2000 frames) into a separate input folder, have DAIN process those, then use a PowerShell script to automatically rename all the outputted frames so that the frame number lines up properly. DAIN will process input files starting at any rame number, but will always start outputting at frame 1, so I have to shift the number until it lines up. In this way, I can process the full video in smaller batches and merge everything together afterwards.

Here is the simple PowerShell script I am using if anyone is interested. I simply input the directory of the output folder, and the frame number that the first file should be. It then goes backwards through all the files and renames them. The output files must be in the default DAIN-ncnn-Vulkan format, of %08d, and no other files should be present in the directory. It is quite a bad solution, but it does work.

$dir = Read-Host 'Directory'
$init = Read-Host 'Initial value of first file'
$initVal = [int]$init

$files = Get-ChildItem $dir
$total = $files.Count
for ($i=$total - 1; $i -ge 0; $i--) {
	$name = $files[$i].Basename
	$ext = $files[$i].Extension
	$nameVal = ([int]$name) - 1
	$finalVal = '{0:d8}' -f ($nameVal + $initVal)
	Rename-Item -Path $files[$i].FullName -NewName "$finalVal$ext"
}
Write-Output "Done renaming $total files"
cmd /c pause | out-null

[backcountrymountains]

This will help me, thanks!

[supercom32]

This would be an extremely helpful feature, especially if your talking about a job which spans multiple days. The easiest way to implement this idea would be to check if the destination file already exists, and if so, do not process anything and move on to the next item.

Unfortunately, my C++ is rather limited, and I'm not exactly sure where this work is occurring. But all you'd need to do is find out where each frame is being processed, add a small file check, and if an output file exists, continue to the next entry in your iteration loop rather than doing the actual work to compute it.

Perhaps if there is someone more experienced with C++ they could make the changes required?

[PseudoResonance]

I haven't tested it, but checking every file if it exists is probably a huge waste of time. I think you're better off just having a starting index parameter and assuming that all prior files exist.

[supercom32]

In my humble opinion, I would agree that setting a starting index is the most direct approach, but is probably not very user friendly. The reason being, it assumes the user is knowlegable enough to know which index they need to use to start/resume their job. Given that input frames are not always 1:1 with output frames, people need to count how many output frames are present, figure out what interpolation ratio they were using, and then determine which input frame the process died at.

By skipping image processing if an output file already exists, it makes resuming a job automatic with no user intervention, involves the least amount of code changes, and is reasonably fast (especially when modern computers can iterate over a simple no-op loop with incredible speed).

Again, in my humble opinion, as It is nether right or wrong.

[PseudoResonance]

Yeah your idea very well might be better, but you can also write a file to the output, name it something like task.json, and store the index in there. If you're interpolating 3 frames between every 2 real frames and 2 already exist, you are probably fine just overwriting those 2 to better ensure they're correct and continuing on. Such an index could be used as a fallback if there's no index specified in the command line arguments.

[supercom32]

To be frank, any alternative at this point would be tremendously helpful. Unfortunately, I'm not well versed enough in C/C++ to promote a change myself. Perhaps if someone who is knowledgeable in the language could chime in, that would be helpful.