fast going train

Running ForEach in parallel on Windows Powershell 5 (and older)

What we are looking at today is something I have seen people ask for time and time again but never seen a good pre-built solution for. I used to have my own “manual” and clunky way of doing this but then when looking for something completely different I just happened to stumble upon this little beauty of a function being used internally in the ChocoOneGet module. And let me tell you it is just amazing in its simplicity!

https://www.powershellgallery.com/packages/ChocoOneGet/0.4.0/Content/Internal%5CForEach-Parallel.ps1

Internally this function uses runspaces but you really don’t need to know anything about how it does it’s magic to use it so that is all I am going to say about that. If you are interested in finding out exactly how it works let me know and I might do another post explaining the code inside the function.

So how does it work from a user perspective then?

Actually it is really simple, it works pretty much the same as Foreach-Object but much much faster. Lets take an example. Below you have two loops that run the Test-Connection command 16 times, I will get back to that number later on. One of these loops use regular old ForEach-Object and the other one is using ForEach-Parallel. Then in the box below you have the output of this little experiment.

#Load the function
.\ForEach-Parallel.ps1


"Regular ForEach"
Measure-Command {
    1..16| ForEach-Object {
        Test-Connection "scriptingnerd.com"
    }
}


"ForEach-Parallel"
Measure-Command {
    1..16| ForEach-Parallel -ScriptBlock {
        Test-Connection "scriptingnerd.com"
    }
}
Regular ForEach
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 52
Milliseconds      : 244
Ticks             : 522449026
TotalDays         : 0,000604686372685185
TotalHours        : 0,0145124729444444
TotalMinutes      : 0,870748376666667
TotalSeconds      : 52,2449026
TotalMilliseconds : 52244,9026

ForEach-Parallel
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 498
Ticks             : 34982846
TotalDays         : 4,04894050925926E-05
TotalHours        : 0,000971745722222222
TotalMinutes      : 0,0583047433333333
TotalSeconds      : 3,4982846
TotalMilliseconds : 3498,2846

Wow, you see that? The regular ForEach took over 52 seconds while the Parallel one took just under 3.5 seconds. That’s amazing and just imagine how much this could speed up your already existing scripts. Instead of having to wait for all tasks one by one your waiting time would always be that of the slowest task. That is, unless you do 17 or more tasks.

Wait what, what do you mean?
Simple, by default the function will only start 16 tasks at a time, once one of them is finished it will run task 17 and so forth. In the example above this would lead to a doubling of the parallel time. The limit is there to avoid overwhelming your system but if you think it can handle the load you can change how many tasks should be allowed to run at the same time by passing in the parameter -MaxRunspaces. Like this

#Load the function
.\ForEach-Parallel.ps1

"ForEach-Parallel (Default)"
Measure-Command {
    1..17| ForEach-Parallel -ScriptBlock {
        Test-Connection "scriptingnerd.com"
    }
}

"ForEach-Parallel (Extra runspaces)"
Measure-Command {
    1..17| ForEach-Parallel -ScriptBlock {
        Test-Connection "scriptingnerd.com"
    } -MaxRunspaces 17
}
Foreach-Parallel (Default)
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 6
Milliseconds      : 679
Ticks             : 66795728
TotalDays         : 7,73098703703704E-05
TotalHours        : 0,00185543688888889
TotalMinutes      : 0,111326213333333
TotalSeconds      : 6,6795728
TotalMilliseconds : 6679,5728

Foreach-Parallel (Extra runspaces)
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 595
Ticks             : 35955956
TotalDays         : 4,16156898148148E-05
TotalHours        : 0,000998776555555556
TotalMinutes      : 0,0599265933333333
TotalSeconds      : 3,5955956
TotalMilliseconds : 3595,5956

Ok, this sound great, so what’s the catch?
Honestly, I haven’t found anything that stopped me from using it yet, should I find one in the future I will come back and update this. That said there is one thing you should be aware of, if you are used to the parallel flag of Powershell 7 then know that this behaves a bit different. Instead of running everything and throw it back to you as soon as it finish this function will wait until everything is finished before giving it back, this also means that it will be returned in the same order you sent it in, not in the order it finished.

Leave a Reply

Your email address will not be published. Required fields are marked *