PowerShell

Testing URIs and URLs with PowerShell

Recently I posted some revisions to the tools I use to create a troubleshooting toolkit. My preferred methodology is to maintain download information in a CSV file.

My CSV file looks like this:

product,uri
HouseCallx64,http://go.trendmicro.com/housecall8/HousecallLauncher64.exe
HouseCallx32,http://go.trendmicro.com/housecall8/HousecallLauncher.exe
"RootKit Buster x32",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180.exe
"Rootkit Buster x64",http://files.trendmicro.com/products/rootkitbuster/RootkitBusterV5.0-1180x64.exe
RUBotted,http://files.trendmicro.com/products/rubotted/RUBottedSetup.exe

Naturally new versions of my selected tools will become available over time, which usually means that the older links will no longer work. In some cases, vendors are smart enough to use one link and simply change what the link points to. But sadly not everyone does that, so I need some way to test if the Uniform Resource Identifier (URI) is still valid. I don’t want to try and download the file. I simply want to test. It turns out we can still use Invoke-WebRequest. The trick is to specify a different method other than the default GET. There is an HTTP method called HEAD, which will answer a web request with basically all of the information except the actual body. The HEAD method is designed for exactly what I need to do.

invoke-webrequest http://go.trendmicro.com/housecall8/HousecallLauncher64.exe -DisableKeepAlive -UseBasicParsing -Method head
Using the Invoke-WebRequest cmdlet in PowerShell to test the Uniform Resource Identifier (URI). (Image Credit: Jeff Hicks)
Using the Invoke-WebRequest cmdlet in PowerShell to test the Uniform Resource Identifier (URI). (Image Credit: Jeff Hicks)

The response is very quick because it is not trying to download the body, i.e. file. All I get back are the headers that contain the relevant information I need. I can retrieve details from either the RawContent or BaseResponse properties.

Sponsored Content

What is “Inside Microsoft Teams”?

“Inside Microsoft Teams” is a webcast series, now in Season 4 for IT pros hosted by Microsoft Product Manager, Stephen Rose. Stephen & his guests comprised of customers, partners, and real-world experts share best practices of planning, deploying, adopting, managing, and securing Teams. You can watch any episode at your convenience, find resources, blogs, reviews of accessories certified for Teams, bonus clips, and information regarding upcoming live broadcasts. Our next episode, “Polaris Inc., and Microsoft Teams- Reinventing how we work and play” will be airing on Oct. 28th from 10-11am PST.

Using the RawContent property to obtain detailed information in PowerShell. (Image Credit: Jeff Hicks)
Using the RawContent property to obtain detailed information in PowerShell. (Image Credit: Jeff Hicks)

This makes it easy to get details about the file I might try to download.

$a.BaseResponse | Select ResponseURI,ContentLength,ContentType,LastModified
Using the BaseResponse property to obtain more detailed information in PowerShell. (Image Credit: Jeff Hicks)
Using the BaseResponse property to obtain more detailed information in PowerShell. (Image Credit: Jeff Hicks)

Let me point out that even though I am using this cmdlet with the HEAD method to test my CSV file, you can use it for any URI or URL as a very fast-testing mechanism. To make your life easier, I went ahead and created a PowerShell tool in the form of an advanced function.

#requires -version 4.0

Function Test-URI {
<#
.Synopsis
Test a URI or URL
.Description
This command will test the validity of a given URL or URI that begins with either http or https. The default behavior is to write a Boolean value to the pipeline. But you can also ask for more detail.

Be aware that a URI may return a value of True because the server responded correctly. For example this will appear that the URI is valid.

test-uri -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip

But if you look at the test in detail:

ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip
ContentLength : 23070
ContentType   : text/html
LastModified  : 1/19/2015 11:34:44 AM
Status        : 200

You'll see that the content type is Text and most likely a 404 page. By comparison, this is the desired result from the correct URI:

PS C:\> test-uri -detail -uri http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip

ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo6_3_0.zip
ContentLength : 2863977
ContentType   : application/x-zip-compressed
LastModified  : 12/31/2014 1:48:34 PM
Status        : 200

.Example
PS C:\> test-uri https://petri.com
True
.Example
PS C:\> test-uri https://petri.com -detail

ResponseUri   : https://petri.com/
ContentLength : -1
ContentType   : text/html; charset=UTF-8
LastModified  : 1/19/2015 12:14:57 PM
Status        : 200
.Example
PS C:\> get-content D:\temp\uris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"}

ResponseUri   : http://files.snapfiles.com/localdl936/CrystalDiskInfo7_2_0.zip
ContentLength : 23070
ContentType   : text/html
LastModified  : 1/19/2015 11:34:44 AM
Status        : 200

ResponseURI   : http://download.bleepingcomputer.com/grinler/rkill
ContentLength : 
ContentType   : 
LastModified  : 
Status        : 404

Test a list of URIs and filter for those that are not OK or where the type is not an application.
.Notes
Last Updated: January 19, 2015
Version     : 1.0

Learn more about PowerShell:
http://jdhitsolutions.com/blog/essential-powershell-resources/

  ****************************************************************
  * DO NOT USE IN A PRODUCTION ENVIRONMENT UNTIL YOU HAVE TESTED *
  * THOROUGHLY IN A LAB ENVIRONMENT. USE AT YOUR OWN RISK.  IF   *
  * YOU DO NOT UNDERSTAND WHAT THIS SCRIPT DOES OR HOW IT WORKS, *
  * DO NOT USE IT OUTSIDE OF A SECURE, TEST SETTING.             *
  ****************************************************************

.Link
Invoke-WebRequest
#>

[cmdletbinding(DefaultParameterSetName="Default")]
Param(
[Parameter(Position=0,Mandatory,HelpMessage="Enter the URI path starting with HTTP or HTTPS",
ValueFromPipeline,ValueFromPipelineByPropertyName)]
[ValidatePattern( "^(http|https)://" )]
[Alias("url")]
[string]$URI,
[Parameter(ParameterSetName="Detail")]
[Switch]$Detail,
[ValidateScript({$_ -ge 0})]
[int]$Timeout = 30
)

Begin {
    Write-Verbose -Message "Starting $($MyInvocation.Mycommand)" 
    Write-Verbose -message "Using parameter set $($PSCmdlet.ParameterSetName)" 
} #close begin block

Process {

    Write-Verbose -Message "Testing $uri"
    Try {
     #hash table of parameter values for Invoke-Webrequest
     $paramHash = @{
     UseBasicParsing = $True
     DisableKeepAlive = $True
     Uri = $uri
     Method = 'Head'
     ErrorAction = 'stop'
     TimeoutSec = $Timeout
    }

    $test = Invoke-WebRequest @paramHash

     if ($Detail) {
        $test.BaseResponse | 
        Select ResponseURI,ContentLength,ContentType,LastModified,
        @{Name="Status";Expression={$Test.StatusCode}}
     } #if $detail
     else {
       if ($test.statuscode -ne 200) {
            #it is unlikely this code will ever run but just in case
            Write-Verbose -Message "Failed to request $uri"
            write-Verbose -message ($test | out-string)
            $False
         }
         else {
            $True
         }
     } #else quiet
     
    }
    Catch {
      #there was an exception getting the URI
      write-verbose -message $_.exception
      if ($Detail) {
        #most likely the resource is 404
        $objProp = [ordered]@{
        ResponseURI = $uri
        ContentLength = $null
        ContentType = $null
        LastModified = $null
        Status = 404
        }
        #write a matching custom object to the pipeline
        New-Object -TypeName psobject -Property $objProp

        } #if $detail
      else {
        $False
      }
    } #close Catch block
} #close Process block

End {
    Write-Verbose -Message "Ending $($MyInvocation.Mycommand)"
} #close end block

} #close Test-URI Function

By default the function will give you a true/false answer if the URI exists. Please be careful as some sites might still return a status code of 200 indicating OK. For example, the following returns true even though I fudged the original URI to make it invalid in terms of downloading:

test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip

I added a –Detail parameter to write a custom object to the pipeline to make it easier to identify files from my CSV that can’t be downloaded,.

test-uri http://files.snapfiles.com/localdl936/CrystalDiskInfo.zip -Detail
Using test-url with the -Detail parameter. (Image Credit: Jeff Hicks)
Using test-url with the -Detail parameter. (Image Credit: Jeff Hicks)

I know that the ContentType should be “application” something. Most likely this is pointing to a 404 page. My Test-URI gives me the information I need. I can then use PowerShell to do something with it.

get-content D:\temp\uris.txt | test-uri -Detail | where { $_.status -ne 200 -OR $_.contentType -notmatch "application"}

I created a test file with some intentionally broken links so that you could see the results

I would then know to research these links and update my source file. I want to stress that I didn’t include this functionality in Test-URI. If I had, then Test-URI would be limited, and I’d be unable to use it for other URL or URI testing. When creating PowerShell tools and scripts, you don’t want to code yourself into a corner. Write objects to the pipeline and use existing PowerShell commands to work with the results as I did above.

Related Topics:

BECOME A PETRI MEMBER:

Don't have a login but want to join the conversation? Sign up for a Petri Account

Register
Comments (0)

Leave a Reply

External Sharing and Guest User Access in Microsoft 365 and Teams

This eBook will dive into policy considerations you need to make when creating and managing guest user access to your Teams network, as well as the different layers of guest access and the common challenges that accompany a more complicated Microsoft 365 infrastructure.

You will learn:

  • Who should be allowed to be invited as a guest?
  • What type of guests should be able to access files in SharePoint and OneDrive?
  • How should guests be offboarded?
  • How should you determine who has access to sensitive information in your environment?

Sponsored by:

 
Live Webinar: Active Directory Security: What Needs Immediate Priority!Live on Tuesday, October 12th at 1 PM ET

Attacks on Active Directory are at an all-time high. Companies that are not taking heed are being punished, both monetarily and with loss of production.

In this webinar, you will learn:

  • How to prioritize vulnerability management
  • What attackers are leveraging to breach organizations
  • Where Active Directory security needs immediate attention
  • Overall strategy to secure your environment and keep it secured

Sponsored by: