Last Update: Sep 04, 2024 | Published: Mar 02, 2015
Recently I posted some revisions to the tools I use to create a troubleshooting toolkit. My preferred methodology is to maintain download information in a CSV file.
My CSV file looks like this:
product,uri
HouseCallx64,http://go.trendmicro.com/housecall8/HousecallLauncher64.exe
HouseCallx32,http://go.trendmicro.com/housecall8/HousecallLauncher.exe
"RootKit Buster x32",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180.exe
"Rootkit Buster x64",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180x64.exe
RUBotted,http://files.trendmicro.com/products/rubotted/RUBottedSetup.exe
Naturally new versions of my selected tools will become available over time, which usually means that the older links will no longer work. In some cases, vendors are smart enough to use one link and simply change what the link points to. But sadly not everyone does that, so I need some way to test if the Uniform Resource Identifier (URI) is still valid. I don’t want to try and download the file. I simply want to test. It turns out we can still use Invoke-WebRequest. The trick is to specify a different method other than the default GET. There is an HTTP method called HEAD, which will answer a web request with basically all of the information except the actual body. The HEAD method is designed for exactly what I need to do.
invoke-webrequest http://go.trendmicro.com/housecall8/HousecallLauncher64.exe -DisableKeepAlive -UseBasicParsing -Method head
The response is very quick because it is not trying to download the body, i.e. file. All I get back are the headers that contain the relevant information I need. I can retrieve details from either the RawContent or BaseResponse properties.
This makes it easy to get details about the file I might try to download.
$a.BaseResponse | Select ResponseURI,ContentLength,ContentType,LastModified
Let me point out that even though I am using this cmdlet with the HEAD method to test my CSV file, you can use it for any URI or URL as a very fast-testing mechanism. To make your life easier, I went ahead and created a PowerShell tool in the form of an advanced function.
#requires -version 4.0 Function Test-URI { <# .Synopsis Test a URI or URL .Description This command will test the validity of a given URL or URI that begins with either http or https. The default behavior is to write a Boolean value to the pipeline. But you can also ask for more detail. Be aware that a URI may return a value of True because the server responded correctly. For example this will appear that the URI is valid. test-uri -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip But if you look at the test in detail: ResponseUri : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip ContentLength : 23070 ContentType : text/html LastModified : 1/19/2015 11:34:44 AM Status : 200 You'll see that the content type is Text and most likely a 404 page. By comparison, this is the desired result from the correct URI: PS C:> test-uri -detail -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip ResponseUri : http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip ContentLength : 2863977 ContentType : application/x-zip-compressed LastModified : 12/31/2014 1:48:34 PM Status : 200 .Example PS C:> test-uri https://petri.com True .Example PS C:> test-uri https://petri.com -detail ResponseUri : https://petri.com/ ContentLength : -1 ContentType : text/html; charset=UTF-8 LastModified : 1/19/2015 12:14:57 PM Status : 200 .Example PS C:> get-content D:tempuris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"} ResponseUri : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip ContentLength : 23070 ContentType : text/html LastModified : 1/19/2015 11:34:44 AM Status : 200 ResponseURI : http://download.bleepingcomputer.com/grinler/rkill ContentLength : ContentType : LastModified : Status : 404 Test a list of URIs and filter for those that are not OK or where the type is not an application. .Notes Last Updated: January 19, 2015 Version : 1.0 Learn more about PowerShell:Essential PowerShell Learning Resources**************************************************************** * DO NOT USE IN A PRODUCTION ENVIRONMENT UNTIL YOU HAVE TESTED * * THOROUGHLY IN A LAB ENVIRONMENT. USE AT YOUR OWN RISK. IF * * YOU DO NOT UNDERSTAND WHAT THIS SCRIPT DOES OR HOW IT WORKS, * * DO NOT USE IT OUTSIDE OF A SECURE, TEST SETTING. * **************************************************************** .Link Invoke-WebRequest #> [cmdletbinding(DefaultParameterSetName="Default")] Param( [Parameter(Position=0,Mandatory,HelpMessage="Enter the URI path starting with HTTP or HTTPS", ValueFromPipeline,ValueFromPipelineByPropertyName)] [ValidatePattern( "^(http|https)://" )] [Alias("url")] [string]$URI, [Parameter(ParameterSetName="Detail")] [Switch]$Detail, [ValidateScript({$_ -ge 0})] [int]$Timeout = 30 ) Begin { Write-Verbose -Message "Starting $($MyInvocation.Mycommand)" Write-Verbose -message "Using parameter set $($PSCmdlet.ParameterSetName)" } #close begin block Process { Write-Verbose -Message "Testing $uri" Try { #hash table of parameter values for Invoke-Webrequest $paramHash = @{ UseBasicParsing = $True DisableKeepAlive = $True Uri = $uri Method = 'Head' ErrorAction = 'stop' TimeoutSec = $Timeout } $test = Invoke-WebRequest @paramHash if ($Detail) { $test.BaseResponse | Select ResponseURI,ContentLength,ContentType,LastModified, @{Name="Status";Expression={$Test.StatusCode}} } #if $detail else { if ($test.statuscode -ne 200) { #it is unlikely this code will ever run but just in case Write-Verbose -Message "Failed to request $uri" write-Verbose -message ($test | out-string) $False } else { $True } } #else quiet } Catch { #there was an exception getting the URI write-verbose -message $_.exception if ($Detail) { #most likely the resource is 404 $objProp = [ordered]@{ ResponseURI = $uri ContentLength = $null ContentType = $null LastModified = $null Status = 404 } #write a matching custom object to the pipeline New-Object -TypeName psobject -Property $objProp } #if $detail else { $False } } #close Catch block } #close Process block End { Write-Verbose -Message "Ending $($MyInvocation.Mycommand)" } #close end block } #close Test-URI Function
By default the function will give you a true/false answer if the URI exists. Please be careful as some sites might still return a status code of 200 indicating OK. For example, the following returns true even though I fudged the original URI to make it invalid in terms of downloading:
test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip
I added a –Detail parameter to write a custom object to the pipeline to make it easier to identify files from my CSV that can’t be downloaded,.
test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip -Detail
I know that the ContentType should be “application” something. Most likely this is pointing to a 404 page. My Test-URI gives me the information I need. I can then use PowerShell to do something with it.
get-content D:tempuris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"}
I created a test file with some intentionally broken links so that you could see the results
I would then know to research these links and update my source file. I want to stress that I didn’t include this functionality in Test-URI. If I had, then Test-URI would be limited, and I’d be unable to use it for other URL or URI testing. When creating PowerShell tools and scripts, you don’t want to code yourself into a corner. Write objects to the pipeline and use existing PowerShell commands to work with the results as I did above.
Related Article: