PowerShell Problem Solver: How to Pull Data from Log Files using PowerShell

Jeff Hicks

Feb 25, 2015

In the world of Windows, a great deal of information is stored away in log files, which can easily be used with PowerShell. If the log files are structured or predictable, it doesn’t take much to create a PowerShell tool for extracting useful information. Many times you can get what you need with a simple one-line command. If this is a process that’s worth repeating or something you have to entrust to someone else, then you’ll want create a re-usable tool. In this article, I’ll show you how to easily pull data from log files using PowerShell, with some additional help from the Get-Content and Import-CSV cmdlets.
Before we even begin, please allow me to stress that even though we will be extracting data from text files, don’t think in terms of textual output. Always be thinking about objects in the pipeline. If you’re thinking in this way, then it’s much easier to get any specific text values that you need.

Let’s use the Windows Update log as an example, since it is something everyone should have. First, what does the data look like?

get-content C:\windows\WindowsUpdate.log -TotalCount 5

Using the get-content cmdlet in Windows PowerShell to get a glance of the data. (Image Credit: Jeff Hicks)

I grabbed the first few lines of the file. Although there is no header, the file appears to have some structure in what is a probably a tab delimited format. Clearly there are some date and time values, as well as message strings. Rule number one is to know your data. In this particular case, Microsoft has provided some documentation on the file format.
Although there’s no header, the data is structured, and I should be able to use Import-CSV to import it. The following lines lets me define and provide my own header.

advertisment

$header = "Date","Time","PID","TID","Component","Message"
$log = import-csv -Delimiter `t -Header $header -Path C:\windows\WindowsUpdate.log

Look at what I get in $log:

The resulting header that we have created. (Image Credit: Jeff Hicks)

I now have objects that I can use like anything else in PowerShell.

We now have objects that we can use in PowerShell. (Image Credit: Jeff Hicks)

For example, the following line of PowerShell let’s us find all fatal problems.

$log | where {$_.message -match "fatal"} | out-gridview -title "Fatal Errors"

A list of fatal errors generated in Windows PowerShell. (Image Credit: Jeff Hicks)

There’s potential pitfall, however, especially if you plan on filtering or sorting. Let’s look again at the resulting object in $log:

With Import-CSV, everything becomes a string. This means you have to take an extra step or two to get the data into a format that’s easier to use. Here’s what I can do with the Windows update log.

$header = "Date","Time","PID","TID","Component","Message"
$log = import-csv -Delimiter `t -Header $header -Path C:\windows\WindowsUpdate.log |
Select-Object @{Name="DateTime";Expression={
"$($_.date) $($_.time.Substring(0,8))" -as [datetime]}},
@{Name="PID";Expression={$_.PID -as [int]}},
@{Name="TID";Expression={$_.TID -as [int]}},Component,Message

The import is very similar to what I did earlier, except that I’m creating another property called DateTime that will be constructed from the incoming Date and Time properties.

In this scenario, I found I needed to strip off the milliseconds from the time stamp in order for PowerShell to format the result as a DateTime object. But now I have something much more useful.

Now I can more easily find the information I need.

$log | where { $_.Datetime -gt "1/5/2015" -AND $_.component -eq "Agent"} | Out-gridview

The resulting out-gridview. (Image Credit: Jeff Hicks)

This works fine locally, but what if I want to able to search update logs on remote computers? This is where scripting comes in handy. Using PowerShell remoting I can easily run the same command in a remote session.

advertisment

#requires -version 3.0
Function Get-WindowsUpdateLog {
[cmdletbinding()]
Param(
[Parameter(Position=0,ValueFromPipeline)]
[ValidateNotNullorEmpty()]
[string[]]$Computername = $env:COMPUTERNAME
)
Begin {
    Write-Verbose "Starting $($MyInvocation.Mycommand)"
    $header = "Date","Time","PID","TID","Component","Message"
} #begin
Process {
    Write-Verbose "Processing Windows Update Log on $($($computername.toUpper()) -join ",")"
    #define a scriptblock to run remotely
    $sb = {
    Import-Csv -Delimiter `t -Header $using:header -Path C:\windows\WindowsUpdate.log |
    Select-Object @{Name="DateTime";Expression={
    "$($_.date) $($_.time.Substring(0,8))" -as [datetime]}},
    @{Name="PID";Expression={$_.PID -as [int]}},
    @{Name="TID";Expression={$_.TID -as [int]}},Component,Message
    }
    Try {
       Invoke-Command -ScriptBlock $sb -ComputerName $Computername -errorAction Stop |
       Select * -ExcludeProperty RunspaceID
    }
    Catch {
        Throw $_
    }
} #process
End {
    Write-Verbose "Ending $($MyInvocation.Mycommand)"
} #end
} #end function

All I’ve really done is take the code that I know already works and wrap it in a function that runs it remotely using Invoke-Command. The remote execution automatically incorporates a computername property.

$data = Get-WindowsUpdateLog -Computername chi-dc01,chi-dc02,chi-dc04 -verbose

The remote execution leverages a computer name property in Windows PowerShell. (Image Credit: Jeff Hicks)

Now I can slice and dice update data across multiple sources.

$data.where({$_.DateTime -ge "12/1/2014" -AND $_.Message -match "Fatal"})| Sort DateTime,PSComputername |
Select PSComputername,DateTime,Component,Message |
format-table –AutoSize

You should be able to use these techniques for any structured log file, with or without a header. The data is there, waiting for you to come and get it.