Testing URIs and URLs with PowerShell

Recently I posted some revisions to the tools I use to create a troubleshooting toolkit. My preferred methodology is to maintain download information in a CSV file.


My CSV file looks like this:

product,uri
HouseCallx64,http://go.trendmicro.com/housecall8/HousecallLauncher64.exe
HouseCallx32,http://go.trendmicro.com/housecall8/HousecallLauncher.exe
"RootKit Buster x32",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180.exe
"Rootkit Buster x64",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180x64.exe
RUBotted,http://files.trendmicro.com/products/rubotted/RUBottedSetup.exe

Naturally new versions of my selected tools will become available over time, which usually means that the older links will no longer work. In some cases, vendors are smart enough to use one link and simply change what the link points to. But sadly not everyone does that, so I need some way to test if the Uniform Resource Identifier (URI) is still valid. I don’t want to try and download the file. I simply want to test. It turns out we can still use Invoke-WebRequest. The trick is to specify a different method other than the default GET. There is an HTTP method called HEAD, which will answer a web request with basically all of the information except the actual body. The HEAD method is designed for exactly what I need to do.

invoke-webrequest http://go.trendmicro.com/housecall8/HousecallLauncher64.exe -DisableKeepAlive -UseBasicParsing -Method head
Using the Invoke-WebRequest cmdlet in PowerShell to test the Uniform Resource Identifier (URI). (Image Credit: Jeff Hicks)
Using the Invoke-WebRequest cmdlet in PowerShell to test the Uniform Resource Identifier (URI). (Image Credit: Jeff Hicks)

The response is very quick because it is not trying to download the body, i.e. file. All I get back are the headers that contain the relevant information I need. I can retrieve details from either the RawContent or BaseResponse properties.

Using the RawContent property to obtain detailed information in PowerShell. (Image Credit: Jeff Hicks)
Using the RawContent property to obtain detailed information in PowerShell. (Image Credit: Jeff Hicks)

This makes it easy to get details about the file I might try to download.

$a.BaseResponse | Select ResponseURI,ContentLength,ContentType,LastModified
Using the BaseResponse property to obtain more detailed information in PowerShell. (Image Credit: Jeff Hicks)
Using the BaseResponse property to obtain more detailed information in PowerShell. (Image Credit: Jeff Hicks)


Let me point out that even though I am using this cmdlet with the HEAD method to test my CSV file, you can use it for any URI or URL as a very fast-testing mechanism. To make your life easier, I went ahead and created a PowerShell tool in the form of an advanced function.

#requires -version 4.0
Function Test-URI {
<#
.Synopsis
Test a URI or URL
.Description
This command will test the validity of a given URL or URI that begins with either http or https. The default behavior is to write a Boolean value to the pipeline. But you can also ask for more detail.
Be aware that a URI may return a value of True because the server responded correctly. For example this will appear that the URI is valid.
test-uri -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip
But if you look at the test in detail:
ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip
ContentLength : 23070
ContentType   : text/html
LastModified  : 1/19/2015 11:34:44 AM
Status        : 200
You'll see that the content type is Text and most likely a 404 page. By comparison, this is the desired result from the correct URI:
PS C:\> test-uri -detail -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip
ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip
ContentLength : 2863977
ContentType   : application/x-zip-compressed
LastModified  : 12/31/2014 1:48:34 PM
Status        : 200
.Example
PS C:\> test-uri https://petri.com
True
.Example
PS C:\> test-uri https://petri.com -detail
ResponseUri   : https://petri.com/
ContentLength : -1
ContentType   : text/html; charset=UTF-8
LastModified  : 1/19/2015 12:14:57 PM
Status        : 200
.Example
PS C:\> get-content D:\temp\uris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"}
ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip
ContentLength : 23070
ContentType   : text/html
LastModified  : 1/19/2015 11:34:44 AM
Status        : 200
ResponseURI   : http://download.bleepingcomputer.com/grinler/rkill
ContentLength :
ContentType   :
LastModified  :
Status        : 404
Test a list of URIs and filter for those that are not OK or where the type is not an application.
.Notes
Last Updated: January 19, 2015
Version     : 1.0
Learn more about PowerShell:
Essential PowerShell Learning Resources
**************************************************************** * DO NOT USE IN A PRODUCTION ENVIRONMENT UNTIL YOU HAVE TESTED * * THOROUGHLY IN A LAB ENVIRONMENT. USE AT YOUR OWN RISK. IF * * YOU DO NOT UNDERSTAND WHAT THIS SCRIPT DOES OR HOW IT WORKS, * * DO NOT USE IT OUTSIDE OF A SECURE, TEST SETTING. * **************************************************************** .Link Invoke-WebRequest #> [cmdletbinding(DefaultParameterSetName="Default")] Param( [Parameter(Position=0,Mandatory,HelpMessage="Enter the URI path starting with HTTP or HTTPS", ValueFromPipeline,ValueFromPipelineByPropertyName)] [ValidatePattern( "^(http|https)://" )] [Alias("url")] [string]$URI, [Parameter(ParameterSetName="Detail")] [Switch]$Detail, [ValidateScript({$_ -ge 0})] [int]$Timeout = 30 ) Begin { Write-Verbose -Message "Starting $($MyInvocation.Mycommand)" Write-Verbose -message "Using parameter set $($PSCmdlet.ParameterSetName)" } #close begin block Process { Write-Verbose -Message "Testing $uri" Try { #hash table of parameter values for Invoke-Webrequest $paramHash = @{ UseBasicParsing = $True DisableKeepAlive = $True Uri = $uri Method = 'Head' ErrorAction = 'stop' TimeoutSec = $Timeout } $test = Invoke-WebRequest @paramHash if ($Detail) { $test.BaseResponse | Select ResponseURI,ContentLength,ContentType,LastModified, @{Name="Status";Expression={$Test.StatusCode}} } #if $detail else { if ($test.statuscode -ne 200) { #it is unlikely this code will ever run but just in case Write-Verbose -Message "Failed to request $uri" write-Verbose -message ($test | out-string) $False } else { $True } } #else quiet } Catch { #there was an exception getting the URI write-verbose -message $_.exception if ($Detail) { #most likely the resource is 404 $objProp = [ordered]@{ ResponseURI = $uri ContentLength = $null ContentType = $null LastModified = $null Status = 404 } #write a matching custom object to the pipeline New-Object -TypeName psobject -Property $objProp } #if $detail else { $False } } #close Catch block } #close Process block End { Write-Verbose -Message "Ending $($MyInvocation.Mycommand)" } #close end block } #close Test-URI Function

By default the function will give you a true/false answer if the URI exists. Please be careful as some sites might still return a status code of 200 indicating OK. For example, the following returns true even though I fudged the original URI to make it invalid in terms of downloading:

test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip

I added a –Detail parameter to write a custom object to the pipeline to make it easier to identify files from my CSV that can’t be downloaded,.

test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip -Detail
Using test-url with the -Detail parameter. (Image Credit: Jeff Hicks)
Using test-url with the -Detail parameter. (Image Credit: Jeff Hicks)

I know that the ContentType should be “application” something. Most likely this is pointing to a 404 page. My Test-URI gives me the information I need. I can then use PowerShell to do something with it.

get-content D:\temp\uris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"}

I created a test file with some intentionally broken links so that you could see the results
011915 2220 TestingURIs5

I would then know to research these links and update my source file. I want to stress that I didn’t include this functionality in Test-URI. If I had, then Test-URI would be limited, and I’d be unable to use it for other URL or URI testing. When creating PowerShell tools and scripts, you don’t want to code yourself into a corner. Write objects to the pipeline and use existing PowerShell commands to work with the results as I did above.

Related Article: