Building a PowerShell Troubleshooting Toolkit Revisited

Recently we posted an article about a PowerShell script you could use to build a USB key that contains free troubleshooting and diagnostic utilities. Those scripts relied on a list of links that were processed sequentially by the Invoke-Webrequest cmdlet. The potential downside is that it might take a bit of time to completely download everything. Wouldn’t it be much nicer if we you could download files, let’s say in batches of five?

Unfortunately, there are no cmdlets or parameters that you can use to throttle a set of commands. You could try to create some sort of throttling mechanism with PowerShell’s job infrastructure. If you are proficient with .NET, then you could try your hand at runspaces and runspace pools. Frankly, those make my head hurt, and I don’t expect IT pros to have to be .NET developers to use PowerShell. Fortunately, there is an alternative that I think is a good compromise between usability and systems programming: workflow.
PowerShell 3.0 brought us the ability to create workflows in PowerShell script. The premise of a workflow is that you can orchestrate a series of activities that can run unattended on 10, 100, or 1,000 remote computers. I don’t have space here to fully explain workflows. There is a chapter on workflow in the PowerShell in Depth book from Manning. But one of the great features in my opinion is the ability to execute multiple commands simultaneously.
In a workflow, you can use a ForEach construct with the –Parallel parameter.

Foreach -parallel ($item in $items) {
do-something $item
}

The –Parallel parameter only works in a workflow. If there are 10 items, then the Do-Something command will run all 10 at the same time. Or you can throttle the number of simultaneous commands.

Foreach –parallel –throttle 5 ($item in $items) {
do-something $item
}


Now only five commands will run at a time. As one command finishes, the next one in the queue is processed. You don’t have to write any complicated .NET code to take advantage of this feature. Use a workflow. And there’s no rule that says I have to use a workflow with a remote computer. I can create a workflow and run it locally, which is what I’ve done with my original script.

#requires -version 4.0
#create a USB tool drive using a PowerShell Workflow
Workflow Get-MyToolsWF {
<#
.Synopsis
Download tools from the Internet.
.Description
This PowerShell workflow will download a set of troubleshooting and diagnostic tools from the Internet. The path will typically be a USB thumbdrive. If you use the -Sysinternals parameter, then all of the SysInternals utilities will also be downloaded to a subfolder called Sysinternals under the given path.
You can limit the number of concurrent downloads with the ThrottleLimit parameter which has a default value of 5.
.Example
PS C:> Get-MyToolsWF -path G: -sysinternals
Download all tools from the web and the Sysinternals utilities. Save to drive G:.
.Notes
Last Updated: September 30, 2014
Version     : 1.0
  ****************************************************************
  * DO NOT USE IN A PRODUCTION ENVIRONMENT UNTIL YOU HAVE TESTED *
  * THOROUGHLY IN A LAB ENVIRONMENT. USE AT YOUR OWN RISK.  IF   *
  * YOU DO NOT UNDERSTAND WHAT THIS SCRIPT DOES OR HOW IT WORKS, *
  * DO NOT USE IT OUTSIDE OF A SECURE, TEST SETTING.             *
  ****************************************************************
.Link
Invoke-WebRequest
#>
[cmdletbinding()]
Param(
[Parameter(Position=0,Mandatory=$True,HelpMessage="Enter the download path")]
[ValidateScript({Test-Path $_})]
[string]$Path,
[switch]$Sysinternals,
[int]$ThrottleLimit=5
)
Write-Verbose -Message "$(Get-Date) Starting $($workflowcommandname)"
Function _download {
    [cmdletbinding()]
    param([string]$Uri,[string]$Path)
    $out = Join-path -path $path -child (split-path $uri -Leaf)
    Write-Verbose -Message  "Downloading $uri to $out"
    #hash table of parameters to splat to Invoke-Webrequest
    $paramHash = @{
     UseBasicParsing = $True
     Uri = $uri
     OutFile = $out
     DisableKeepAlive = $True
     ErrorAction = "Stop"
    }
    Try {
       Invoke-WebRequest @paramHash
       Get-Item -Path $out
        }
    Catch {
        Write-Warning -Message "Failed to download $uri. $($_.exception.message)"
    }
    } #end function
Sequence {
<#
csv data of downloads
The product should be a name or description of the tool.
The URI is a direct download link. The link must end in the executable file name (or zip or msi).
The file will be downloaded and saved locally using the last part of the URI.
#>
$csv = @"
product,uri
HouseCallx64,http://go.trendmicro.com/housecall8/HousecallLauncher64.exe
HouseCallx32,http://go.trendmicro.com/housecall8/HousecallLauncher.exe
"RootKit Buster x32",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180.exe
"Rootkit Buster x64",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180x64.exe
RUBotted,http://files.trendmicro.com/products/rubotted/RUBottedSetup.exe
"Hijack This",http://go.trendmicro.com/free-tools/hijackthis/HiJackThis.exe
WireSharkx64,http://wiresharkdownloads.riverbed.com/wireshark/win64/Wireshark-win64-1.12.1.exe
WireSharkx32,http://wiresharkdownloads.riverbed.com/wireshark/win32/Wireshark-win32-1.12.1.exe
"WireShark Portable",http://wiresharkdownloads.riverbed.com/wireshark/win32/WiresharkPortable-1.12.1.paf.exe
SpyBot,http://spybotupdates.com/files/spybot-2.4.exe
CCleaner,http://download.piriform.com/ccsetup418.exe
"Malware Bytes",http://data-cdn.mbamupdates.com/v2/mbam/consumer/data/mbam-setup-2.0.2.1012.exe
"Emisoft Emergency Kit",http://download11.emsisoft.com/EmsisoftEmergencyKit.zip
"Avast! Free AV",http://files.avast.com/iavs5x/avast_free_antivirus_setup.exe
"McAfee Stinger x32",http://downloadcenter.mcafee.com/products/mcafee-avert/Stinger/stinger32.exe
"McAfee Stinger x64",http://downloadcenter.mcafee.com/products/mcafee-avert/Stinger/stinger64.exe
"Microsoft Fixit Portable",http://download.microsoft.com/download/E/2/3/E237A32D-E0A9-4863-B864-9E820C1C6F9A/MicrosoftFixit-portable.exe
"Cain and Abel",http://www.oxid.it/downloads/ca_setup.exe
"@
#convert CSV data into objects
#save converted objects to a variable seen in the entire workflow
$workflow:download  = $csv | ConvertFrom-Csv
} #end sequence
Sequence {
foreach -parallel -throttle $ThrottleLimit ($item in $download) {
    Write-Verbose -message "Downloading $($item.product)"
    _download -Uri $item.uri -Path $path
} #foreach item
} #end sequence
Sequence {
#region SysInternals
if ($Sysinternals) {
    #test if subfolder exists and create it if missing
    $subfolder = Join-Path -Path $path -ChildPath Sysinternals
    if (-Not (Test-Path -Path $subfolder)) {
        New-item -ItemType Directory -Path $subfolder
    }
    #get the page
    $sysint = Invoke-WebRequest "http://live.sysinternals.com/Tools" -DisableKeepAlive -UseBasicParsing
    #get the links
    $links = $sysint.links | Select -Skip 1
    foreach -parallel -throttle $ThrottleLimit ($item in $links) {
     #download files to subfolder
     $uri = "http://live.sysinternals.com$($item.href)"
     Write-Verbose -message "Downloading $uri"
     _download -Uri $uri -Path $subfolder
    } #foreach
} #if SysInternals
} #end sequence
Write-verbose -message "$(Get-Date) Finished $($workflowcommandname)"
} #end workflow

There are rules for a workflow so I had to make a few adjustments. Workflows aren’t designed to run from pipelined input so the CSV data is embedded in the script. Otherwise, you should recognize most of the code. The key difference is that downloads now happen in parallel and throttled.

foreach -parallel -throttle $ThrottleLimit ($item in $download) {
  Write-Verbose -message "Downloading $($item.product)"
  _download -Uri $item.uri -Path $path
} #foreach item

I do the same thing for the SysInternals links.

foreach -parallel -throttle $ThrottleLimit ($item in $links) {
 #download files to subfolder
 $uri = "http://live.sysinternals.com$($item.href)"
 Write-Verbose -message "Downloading $uri"
 _download -Uri $uri -Path $subfolder
} #foreach


Once the workflow is loaded into my session, which you can do by dot-sourcing the script, I can run it the same way I would any command.

PS C:> Get-MyToolsWF -path G: -sysinternals

Use the –Verbose cmdlet if you want to see more detail. Using the workflow and parallel processing, I can download everything in about five minutes! Personally, I don’t find much use for workflows, but I love this parallel feature and hope that someday we’ll see it as part of the main PowerShell language.