Multithreading over multiple servers

[snurbnacnud]

Hi Markus,

Your script is really helping me in preparing our environment for the migration to EXO.

Nor really an issue, but I am wondering how exactly you managed to get the multithreading to work over multiple servers. I am not new to PowerShell, but the runspaces are new territory. ;-) I would like to do something similar to clean up orphaned sids in our environment.

Maybe you can give me some pointers on how to achieve this?

Best regards,

Duncan

[GruberMarkus]

Hi Duncan,

I am not a runspaces expert, but I'll show you my learning path.

Good starting points are:

• Microsoft's Doctor Scripto article series at https://devblogs.microsoft.com/scripting/beginning-use-of-powershell-runspaces-part-1/
• The Microsoft learning article about PowerShell Runspaces at https://learn.microsoft.com/en-us/powershell/scripting/developer/hosting/creating-multiple-runspaces?view=powershell-7.2
• Boe Prox's blog article at https://learn-powershell.net/2013/04/19/sharing-variables-and-live-objects-between-powershell-runspaces/

The first two articles cover the basic setup of runspaces. Boe Prox's article covers this, too, but goes more into detail how to share data with and between runspaces. After having set up your first few runspaces, it becomes easier and the focus switches to designing the data structures behind them- especially, if runspaces can modify data shared between them and the main script.

If you can use PowerShell 7, the Foreach-Object -Parallel construct often makes parallelization even easier. You find details at https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.2.

In Export-RecipientPermissions, the 'Combine temporary export files' task has an optional sub-task 'Pre-combine files', which could act as a learning example.
To save RAM, the script produces lots and lots of temporary result files, each with a certain number in name. At the end, these temporary result files need to be combined to one single file, keeping the sequence according to their numbers.
As there is only one target file, you can not efficiently parallelize this job: Only one job can write to a file at the same time.
So, the scripts splits up the whole list of files into a certain number of chunks, each consisting of a set of contiguous files. For each of these junks, a separate runspace is created which combines his own list of files into another temporary file. These separate runspaces run in parallel.
After the separate runspaces are completed, a single-thread job just has to combine the few new temporary result files to the final single file.
Although more steps need to be done, this is much faster than the single-thread process used before.