Tuesday, February 2, 2016

Tegile Zebi Replication Reports

Those of you who are working with Tegile storage systems are very aware that reporting is one of the things that is still lacking.  There are a number of canned reports you can get via the GUI, but as far as custom reports or getting data out programatically,  good luck.

I'm told by folks in Tegiles' management that they are developing PowerShell commandlets for use with their storage appliances.  That's great news.  When we get to see them I don't know.

In the interim I needed to get data out of the arrays.  I came up with a PowerShell script that will connect to a list of controllers and extract replication data into an Excel spreadsheet.  I only care about completed verses failed so the output is limited to that.  Actually as it sits it only reports good runs, I'm working on failed runs.

This script will create a spreadsheet with the last 2 months of successful replication events.  It can be modified for other logs but this is specifically designed to parse the replication logs.  Each target is a new worksheet, and each tab is labeled as such.   It's still a work in progress so some variables are incomplete (I like to have them all at the "script:" context...).  I'll update this as I get farther along, but it does work as is.

UPDATE: 09-02-16
It's been noted that this script doesn't catch failed jobs.  It's very targeted towards completed jobs.  I have retooled it to (hopefully) catch the failed jobs.  Seems to be working on my end.  The parsing of the ingested logs is tricky since the formats change depending on what's being reported.

I've overwritten the code below with the v2 version.  I shrunk the font down to try and avoid wrapping.  Tegile has released their PowerShell commandlets.  They do a lot but still don't allow for this type of reporting.

Updated to correct a few typos... sorry :-)

UPDATE: 11-07-16
Tegile recently released the 3.5.x firmware update.  The initial release had a bug and a I sincerely hope none of you experienced the same issues that we did with it.

Another issue is that they have changed the formatting of the replication log messages.  I had to completely rewrite the script below to allow the results to be captured properly.  In addition I added detection of aborted jobs.

Note the config file has changed.  We now specify DAYS instead of WEEKS to report.

UPDATE: 08-21-18
Note that this script is now located on the MS PowerShell Gallery.  It can be found on my library main page at https://www.powershellgallery.com/profiles/Kcmjr/.  This way I don't need to constantly update this post when versions change.

The script noted below is an OLD VERSION.  See the link for updates...

<#==============================================================================
         File Name : ReplicationReport.ps1
   Original Author : Kenneth C. Mazie (kcmjr @ kcmjr.com)
                   :
       Description : This script will query multiple Tegile Zebi Controllers,
                   : parse the replication logs, and populate an Excel spreadsheet.
                   : The results are pulled from back as far as a preset number of
                   : days.  It then emails the resulting spreadsheet.  X number of
                   : prior runs are retained to avoid filling the disk.
                   :
         Arguments : Named commandline parameters:  (all are optional)
                   :   "-console" - Displays console output during run.
                   :   "-debug" - Switches email recipient and some criteria.
                   :
             Notes : Settings are loaded from an XML file located in the script folder.
                   : See the end of the script for config file example.
                   :
      Requirements : Requires PS v5.  Requires Posh-SSH module.              
                   : 
          Warnings : None
                   :
             Legal : Public Domain. Modify and redistribute freely. No rights reserved.
                   : SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF
                   : ANY KIND. USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.
                   :
           Credits : Code snippets and/or ideas came from many sources around the web.
                   :
    Last Update by : Kenneth C. Mazie (email kcmjr AT kcmjr.com for comments or to report bugs)
   Version History : v1.0 - 02-03-16 - Original
    Change History : v1.1 - 03-09-16 - Added XML config file
                   : v2.0 - 09-02-16 - Added detection of failed jobs. Retooled parser.
                   : v3.0 - 09-12-16 - Retooled for Pos-SSH v1.7.6.  Eliminated PSEXEC
                   : requirement.  Eliminated work file requirement.  Added save only
                   : the last X copies.  Added debug name change.  Added note on email
                   : that refers to failures found.
                   : v4.0 - 11-04-16 - Complete redesign of time calculation.  Now
                   : uses epoch starting at 01-01-1970 as reference.  This avoids issues
                   : with using the week of the year.  Also corrected for up to 4 different
                   : variations is log formats seen in v3.5 Zebi firmware.
#===============================================================================#>
#requires -version 5.0

Param (
       [bool]$Debug = $false,
       [bool]$Console = $false
)

clear-host

#--[ Store all the start up variables so you can clean up when the script finishes. ]--
if ($startupvariables) { try {Remove-Variable -Name startupvariables  -Scope Global -ErrorAction SilentlyContinue } catch { } }
New-Variable -force -name startupVariables -value ( Get-Variable | ForEach-Object { $_.Name } )

If (!(Get-Module Posh-SSH)){Import-Module "Posh-SSH" -ErrorAction SilentlyContinue}
If ($Debug){$Script:Debug = $true}
If ($Console){$Script:Console = $true}
$ErrorActionPreference = "silentlycontinue"
$Script:Attach = $true
$Script:Datetime = Get-Date -Format "MM-dd-yyyy_HH:mm"              #--[ Current date $ time ]--
$Script:Today = Get-Date -Format "MM-dd-yyyy"                       #--[ Current date ]--
$Script:ThisYear = Get-Date -Format "yyyy"                          #--[ Current year ]--
$EpochDiff = New-TimeSpan '01 January 1970 00:00:00' $(Get-Date)    #--[ Seconds since 01-01-1970 ]--
$EpochSecs = [INT] $EpochDiff.TotalSeconds                          #--[ Rounded ]--
$EpochDays = [INT] (($EpochDiff.TotalSeconds)/86400)                #--[ Converted to days ]--
$Script:StatusFail = $false

#--[ Excel Non-Interactive Fix ]------------------------------------------------
If (!(Test-path -Path "C:\Windows\System32\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\System32\config\systemprofile\Desktop"}
If (!(Test-path -Path "C:\Windows\SysWOW64\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\SysWOW64\config\systemprofile\Desktop"}
#--[ Excel will crash when run non-interactively via a scheduled task if these folders don't exist ]--

#--[ Functions ]----------------------------------------------------------------
Function ResetVariables {
Get-Variable | Where-Object { $startupVariables -notcontains $_.Name } | ForEach-Object {
  try { Remove-Variable -Name "$($_.Name)" -Force -Scope "global" -ErrorAction SilentlyContinue -WarningAction SilentlyContinue}
  catch { }
 }
}

Function SendEmail {
       If ($Script:Debug){$ErrorActionPreference = "stop"}
       $email = New-Object System.Net.Mail.MailMessage
       $email.From = $Script:EmailFrom
       $email.IsBodyHtml = $Script:EmailHTML
       $email.To.Add($Script:EmailTo)
       $email.Subject = $Script:EmailSubject
       $email.Body = $Script:ReportBody
       If ($Script:Attach -eq $true){
              $email.Attachments.Add("$PSScriptRoot\$Script:SaveFile.xlsx")
       }     
       $smtp = new-object Net.Mail.SmtpClient($Script:SmtpServer)
       $smtp.Send($email)
       If ($Script:Log){Add-content -Path "$PSScriptRoot\debug.txt" -Value "-------------[ Sending Email ]--------------"}
       If ($Script:Console){Write-Host "`nEmail sent...`n"}
}

#--[ Read and load configuration file ]-----------------------------------------
If (!(Test-Path "$PSScriptRoot\Configuration.xml")){                            #--[ Error out if configuration file doesn't exist ]--
       Write-host "MISSING CONFIG FILE.  Script aborted." -forgroundcolor red
       break
}Else{
       [xml]$Script:Configuration = Get-Content "$PSScriptRoot\Configuration.xml"  #--[ Load configuration ]--
       $Script:DebugEmail = $Script:Configuration.Settings.Email.Debug
       $Script:EmailTo = $Script:Configuration.Settings.Email.To
       $Script:EmailHTML = $Script:Configuration.Settings.Email.HTML
       $Script:EmailSubject = $Script:Configuration.Settings.Email.Subject
       $Script:EmailFrom = $Script:Configuration.Settings.Email.From
       $Script:SmtpServer = $Script:Configuration.Settings.Email.SmtpServer
       $Script:UserName = $Script:Configuration.Settings.Credentials.Username
       $Script:Password = $Script:Configuration.Settings.Credentials.Password
       [array]$Script:Targets = $Script:Configuration.Settings.General.Targets
       $Script:DaysBack = $Script:Configuration.Settings.General.Days
       $Script:ReportsToKeep = $Script:Configuration.Settings.General.Reports        #--[ How many reports to retain ]--
       $Script:FileName = $Script:Configuration.Settings.General.FileName
}     
#-------------------------------------------------------------------------------
#$Credential = New-Object PSCredential($UserName, (ConvertTo-SecureString $Password.SubString(64) -k ($Password.SubString(0,64) -split "(?<=\G[0-9a-f]{2})(?=.)" | % { [Convert]::ToByte($_,16) })))
#$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $UserName, ($Password | ConvertTo-SecureString)

Get-ChildItem -Path "$PSScriptRoot\" | Where-Object {-not $_.PsIsContainer -and $_.Name -like "*.xlsx"} | Sort-Object -Descending -Property LastTimeWrite | Select-Object -Skip $Script:ReportsToKeep | Remove-Item     
If ($Script:Debug){
       [string]$Script:TodayDate = Get-Date -Format MM-dd-yyyy_hhmm    #--[ Adds hour and min to filename for debugging ]--
}Else{
       [string]$Script:TodayDate = Get-Date -Format MM-dd-yyyy
}
[string]$Script:SaveFile = $Script:FileName+"-"+$Script:TodayDate

If (Test-Path "$PSScriptRoot\$Script:SaveFile.xlsx"){     #--[ If not in debug mode will rename exisiting XLS to allow it to overwritten ]--
       Rename-Item -Path "$PSScriptRoot\$Script:SaveFile.xlsx" -NewName "$PSScriptRoot\$Script:SaveFile-old.xlsx"
}

$Script:ReportBody = "Attached are the replication results from the prior "+$Script:DaysBack+" days."
$Script:ReportBody += "<br>- All Zebi controllers are included regardless of whether they actively participate in replication or not."
$Script:ReportBody += "<br>- Each controller has it's own tab on the spreadsheet."
$Script:ReportBody += "<br>- Only outgoing replication jobs are tracked."

$Color = @{
       "1" = "green"
       "2" = "red"  
       "3" = "yellow"
       "4" = "cyan"
       "5" = "magenta"
}

$MissingType = [System.Type]::Missing
$Excel = ""
$Excel = New-Object -ComObject Excel.Application
If ($Script:Console -or $Script:Debug){
       $Excel.visible = $True
       $Excel.DisplayAlerts = $true
       $Excel.ScreenUpdating = $true
       $Excel.UserControl = $true
       $Excel.Interactive = $true
}Else{
       $Excel.visible = $False
       $Excel.DisplayAlerts = $false
       $Excel.ScreenUpdating = $false
       $Excel.UserControl = $false
       $Excel.Interactive = $false
}

$TargetCount = $Script:Targets.Target.count
$Workbook = $Excel.Workbooks.Add()
$Workbook.Worksheets.Add($MissingType, $Workbook.Worksheets.Item($Workbook.Worksheets.Count), ($TargetCount) - $Workbook.Worksheets.Count, $Workbook.Worksheets.Item(1).Type) | Out-Null
#--[ Note that new sheets are always adding in reverse ]--

$SheetCount = 1
$Target = @()
Foreach ($Target in $Script:Targets.Target){
       if ($Console){Write-Host "`n--[ Processing Target: $Target ]-----------------------------------" -ForegroundColor Yellow }                   
       $Count = 1   
       $Worksheet = $workbook.Sheets.Item($SheetCount)
       $Worksheet.Activate()     
       $Worksheet.name = $Target
       $Row = 1
       $Col = 1
       $WorkSheet.Cells.Item($Row,$Col) = "Date:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Source IP:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Source Pool:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Target IP:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Target Pool:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Status:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Parsed Data:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.Cells.Item($Row,$Col) = "Raw Data:"
       $WorkSheet.Cells.Item($Row,$Col).font.bold = $true
       $WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
       $Col++
       $WorkSheet.application.activewindow.splitcolumn = 0
       $WorkSheet.application.activewindow.splitrow = 1
       $WorkSheet.application.activewindow.freezepanes = $true
       $Resize = $WorkSheet.UsedRange
       [void]$Resize.EntireColumn.AutoFit()

       $Cmd = 'cat /var/log/tegile/zebi/zebirep.log | egrep "has completed|has failed"'     #--[ The command sent to the Target controller ]--
      
       #--[ Some optional other commands that could be used ]--
       #      'uname -a '
       #      '/usr/sbin/zfs list -o name'
       #      'pwd'
       #      'df -h'
       #      '/usr/sbin/dladm show-link –S'
       #      '/usr/sbin/fmadm faulty'
       #      '/usr/sbin/zpool list'
       #      '/usr/sbin/zfs list -o name'
       #      'zfs list -t snapshot'
       #      'zfs list -o space -r hostname$/Local'
       #      'zpool iostat 2'                                                                                        #--[ Display ZFS I/O statistics every 2 seconds ]--
       #      'zpool iostat -v 2'                                                                                     #--[ Display detailed ZFS I/O statistics every 2 seconds ]--
      
       Remove-SSHSession -SessionId 0 -ErrorAction SilentlyContinue | Out-Null     #--[ Clear out previous session if it still exists ]--

       $SecPassword = ConvertTo-SecureString $Script:Password -AsPlainText -Force
       $Creds = New-Object System.Management.Automation.PSCredential ($Script:UserName, $SecPassword)
       $SSH = New-SshSession -ComputerName $Target -Credential $Creds -AcceptKey:$true #| Out-Null  #--[ Open new SSH session ]--
       $Script:Return = $(Invoke-SSHCommand -SSHSession $SSH -Command $Cmd).Output   #--[ Invoke SSH command and capture the output as a string ]--

       $Row = 2
       $Col = 1
       $SourcePool = ""
       $DestPool = ""
       $TargetPool = ""
       $TargetIP = ""
       $SourceIP = ""
       $Line = ""
       $RepJob = ""
       $Status = ""
      
       #--[ Process each line of the input file ]--
       ForEach ($Line in $Script:Return){
              $JobDiffDays = "99"
              $JobDate = ""
              $JobDay = ""
              $JobMonth = ""
              $JobYear = ""

              $RepJob = $Line.Split("]")[2]
              If ($RepJob.Contains('null')){
                     $JobDate = (($RepJob.Split('@')[2]).split('-')[1]).split(' ')[0]
              }Else{
                     $JobDate = ($RepJob.Split('@')[1]).split('-')[1]
              }
              If ($JobDate -ne ""){
                     $JobYear = $JobDate.substring(0,4)
                     $JobMonth = $JobDate.substring(4,2)
                     $JobDay = $JobDate.substring(6,2)
                     $JobDate = $JobMonth+'-'+$JobDay+'-'+$JobYear
                     If (!([string]$JobDate -as [DateTime])){
                           $JobDate = (($RepJob.Split('@')[2]).split('-')[1]).split('=')[0]
                           $JobYear = $JobDate.substring(0,4)
                           $JobMonth = $JobDate.substring(4,2)
                           $JobDay = $JobDate.substring(6,2)
                           $JobDate = $JobMonth+'-'+$JobDay+'-'+$JobYear
                     }
                     $JobDiff = New-TimeSpan $JobDate $(Get-Date)                               #--[ Days since date of job ]--
                     $JobDiffDays = [int] (($JobDiff.TotalSeconds)/86400)                #--[ Rounded & Converted to days ]--
                          
                     If ($Line -like "*Abort*"){
                           $RepStatus = "ABORTED"
                           $SourcePool = ($RepJob.Split('@')[0]).Split(' ')[2]
                           $TargetPool = ($RepJob.Split(':')[1]).Split(' ')[0]
                           $TargetIP = ($RepJob.Split(':')[0]).Split('>')[1]
                     }Elseif($Line -like "*Abandon*"){
                           $RepStatus = "ABANDONED"
                     }Else{
                           $RepStatus = $Line.Split("]")[3]
                           If (($RepJob.Split('@')[0]).Split(' ')[2] -eq "from"){
                                  $SourcePool = ($RepJob.Split('@')[0]).Split(' ')[3]
                                  $TargetPool = ($RepJob.Split(':')[1])#.Split('/')[0]
                                  $TargetIP = ($RepJob.Split(':')[0]).Split('>')[1]
                           }Else{
                                  $SourcePool = ($RepJob.Split('@')[0]).Split(' ')[2]
                                  $TargetPool = ($RepJob.Split(':')[1])#.Split('/')[0]
                                  $TargetIP = ($RepJob.Split(':')[0]).Split('>')[1]
                           }
                     }
                                               
                     If($RepJob -like "*@Auto*"){
                           #$RepStatus = $Line.Split("]")[3]
                           $SourcePool = ($RepJob.Split('@')[0]).Split(' ')[2]
                           $TargetPool = (($RepJob.Split('>')[1]).Split(':')[1]).split(" ")[0]
                           $TargetIP = ($RepJob.Split('>')[1]).Split(':')[0]
                     }
              }
             
              #--[ Dump collected data into spreadsheet ]--
              if ([int]$JobDiffDays -le [int]$Script:DaysBack){
                     if ($Console){
                           Write-Host "`n`n-----------------------------------------------"
                           Write-Host "Today               ="$Script:Today -ForegroundColor Yellow
                           Write-Host "Date of Job         ="$JobDate -ForegroundColor Yellow
                           Write-Host "Days since job ran  ="$JobDiffDays -ForegroundColor Yellow
                           #Write-Host "Current Week        ="$Script:CurrentWeek -ForegroundColor Yellow
                           #Write-Host "Result              ="$DateMathResult -ForegroundColor Yellow
                           Write-Host "Match Criteria      ="$Script:DaysBack -ForegroundColor Yellow
                           Write-Host "Parsed Data         ="$RepJob -ForegroundColor Yellow
                     }
      
                     if ($Console){Write-Host "-- Matched criteria --" -ForegroundColor magenta }
                     if ($Console){Write-Host "Writing to tab ="$Target -ForegroundColor magenta }
                     if ($Console){Write-Host "Writing to row ="$row -ForegroundColor magenta }
                     #--[ Job run date ]-----------------------------------------------------                
                     $WorkSheet.Cells.Item($Row,$Col) = "$JobDate"
                     Clear-Variable $JobDate
                     $Col++
                     #--[ Source IP ]--------------------------------------------------------         
                     $WorkSheet.Cells.Item($Row,$Col) = $Target
                     $Col++       
                     #--[ Source Pool ]------------------------------------------------------  
                     if ($Console){Write-Host "Source Pool    ="$SourcePool -ForegroundColor Yellow}
                     $WorkSheet.Cells.Item($Row,$Col) = $SourcePool
                     $Col++
                     #--[ Target IP ]--------------------------------------------------------
                     #$dest = ($a.Split("]")[0]).split(" ")[3]
                     $destpool = (($a.Split("]")[0]).split(" ")[3]).Split(":")[1]
                     if ($Console){Write-Host "Target IP      ="$TargetIP}
                     $WorkSheet.Cells.Item($Row,$Col) = $TargetIP
                     $Col++
                     #--[ Destination Pool ]-------------------------------------------------
                     if ($Console){Write-Host "Target Pool    ="$TargetPool -ForegroundColor Yellow}
                     $WorkSheet.Cells.Item($Row,$Col) = $TargetPool
                     $Col++
                     #--[ Job Status ]-------------------------------------------------------  
                     If ($RepStatus.Contains("has failed")){
                           $Status = "FAILED"
                           $Script:StatusFail = $true
                           if ($Console){Write-Host "Job Status     ="$Status -ForegroundColor Red}
                           $WorkSheet.Cells.Item($Row,$Col).Interior.ColorIndex = 3
                     }
                                         If ($RepStatus -eq "aborted"){
                           $Status = "ABORTED"
                           #$Script:StatusFail = $true
                           if ($Console){Write-Host "Job Status     ="$Status -ForegroundColor Red}
                           $WorkSheet.Cells.Item($Row,$Col).Interior.ColorIndex = 6
                     }
                                         If ($RepStatus -eq "abandoned"){
                           $Status = "ABANDONED"
                           #$Script:StatusFail = $true
                           if ($Console){Write-Host "Job Status     ="$Status -ForegroundColor Red}
                           $WorkSheet.Cells.Item($Row,$Col).Interior.ColorIndex = 6
                     }
                     If ($RepStatus.Contains("has completed")){
                           $Status = "GOOD"
                           if ($Console){Write-Host "Job Status     ="$Status -ForegroundColor Green}
                     }     
                     $WorkSheet.Cells.Item($Row,$Col) = $Status
                     $Col++
                     #--[ Parsed Data ]------------------------------------------------------         
                     $Col++
                     $WorkSheet.Cells.Item($Row,$Col) = $RepJob
                     $Col++
                     #--[ Raw Data ]---------------------------------------------------------         
                     if ($Console){Write-Host "Raw data       = "$Line -ForegroundColor Cyan }
                     $WorkSheet.Cells.Item($Row,$Col) = $Line
      
                     $Line = ""
                     $RepJob = ""
                     $Status = ""
                     $Col = 1
                     $Row++
                     $Resize = $WorkSheet.UsedRange
                     [void]$Resize.EntireColumn.AutoFit()
              }
       }
       $SheetCount++
}

If (Test-Path "$PSScriptRoot\$Script:SaveFile-old.xlsx"){Remove-Item -Path "$PSScriptRoot\$Script:SaveFile-old.xlsx" -Confirm:$false -Force:$true}
$Workbook.SaveAs("$PSScriptRoot\$Script:SaveFile.xlsx")
$Excel.Quit()
$Excel = $Null

If($Script:StatusFail){
       $Script:ReportBody += "<br><br><font color=red>A replication failure was detected and is noted on the report</font><br>"
}Else{
       $Script:ReportBody += "<br><br><font color=green>No replication failures were detected.</font><br>"
}

$Script:ReportBody += "<br>Script executed on "+$Datetime+"<br><br>"
Sleep -Seconds 2
SendEmail

[gc]::Collect()
[gc]::WaitForPendingFinalizers()
ResetVariables

if ($Console){Write-Host "--- Completed ---" -ForegroundColor Red }

<#-----------------------------[ Config File ]---------------------------------

The configuration file must be named "Configuration.xml" and must reside in
the same folder as the script.  Below is the format and element list:

<!-- Settings & Configuration File -->
<Settings>
       <General>
              <ScriptName>ReplicationReport</ScriptName>
              <FileName>ZebiReplLogs</FileName>
              <DebugTarget>test-server</DebugTarget>
              <Targets>
                     <Target>10.100.1.1</Target>
                     <Target>10.100.1.2</Target>
                     <Target>10.100.1.3</Target>
                     <Target>10.100.1.4</Target>
                     <Target>10.100.1.5</Target>
                     <Target>10.100.1.6</Target>
                     <Target>10.100.1.7</Target>
                     <Target>10.100.1.8</Target>
              </Targets>
              <Days>14</Days>
              <Reports>20</Reports>
       </General>
       <Email>
              <From>WeeklyReports@mydomain.com</From>
              <To>me@mydomain.com</To>
              <Subject>Weekly Zebi Replication Status Report</Subject>
              <HTML>$true</HTML>
              <SmtpServer>10.100.1.10</SmtpServer>
       </Email>
       <Credentials>
              <UserName>zebiadminuser</UserName>
              <Password>zebiadminpwd</Password>
       </Credentials>
</Settings>  


#>








Friday, January 29, 2016

Zerohorizon.net Browser Redirect Trojan

As I sit here I am frequently being annoyed by a nasty little redirect Trojan that "someone" who used my computer picked up.  I am extremely careful, and very cautious about who uses my PC.  Each of my kids has their own PC, as does my wife.  No one outside of my immediate family uses my PC.  So that leaves either myself or my wife.

It doesn't really matter and I'm not blaming anyone.  In fact my every-day operating system is Centos Linux so this is little more than an occasional annoyance for me.  It doesn't help that I'm so irritated with Microsoft these days that I only use Windows when and IF I have to.  That boils down to about three programs.  Everything else is on Linux.

The Linux side works fine, it's just this Swiss-Cheese OS named Windows that's any issue.  Anyway, I could go on for hours about the various issues I have with Microsoft, but I digress.

So I am documenting what I have found so that if anyone else with this issue can benefit from it, so much the better.

This Trojan is sneaky.  You can search Google all day long and mostly what you find are very randomly named web sites that appear to all be slight variations on the same destination.  Where you wind up is some site that wants to sell you "their" virus removal tool, something called "SpyHunter".
I have no idea if this product is legitimate or not but I recommend that anyone reading this stay away from it.  The fact that 40 or 50 oddly named web sites, each with a slightly varied content, all suggest you use this product makes me very suspicious.

Almost all of these supposed "removal" sites run you through removing installed "bad" programs, "bad" browser extension, and editing the system registry.  Trouble is these steps don't help and don't accomplish anything.  The final step is "oh, none of this worked, try buying our tool...".  Yeah, right...

Also, why have none of the major anti-virus program authors jumped on this?  The fact that they are saying nothing again makes me very suspicious.  This seems to be a prety common infection, there are loads of Google hits when you look up zerohorizon.net.

I've scanned my system with Comodo, Avast, Kaspersky, and also scanned the Windows OS drive while Linux was running (the only REAL way to scan Windows for viruses), and nothing has been found.  Very odd.

This seems to primarily affect the Google Chrome browser (note that as far as I can tell Google has said nothing about this either), but Firefox, and IE have both been reported to be affected.

OK, enough soap box, time for details.

This Trojan is a browser redirect.  At random times it pops up new tabs or windows that load one of a small list of sites.  These sites are all adware sites or sites with fake pages that try to fool you into clicking on links which will likely install other nasty things.  It may also be doing any number of other things, there is really no way to tell without reverse engineering it.  Before that can be done it needs to be located.  The list of sites it uses (so far) is below.  They all seem to eventually bounce through www.buy-targeted-traffic.com and if that doesn't sound like a shady site I don't know what does:

  • buy-targeted-traffic.com
  • orion.zerohorizon.net
  • oziris.zerohorizon.net
  • zerohorizon.net
  • onclicktop.com
  • fugdownload164.com
  • fugdownload173.com
  • ptp24.com
  • cdn.shorte.st
  • shorte.st
  • bundleworldbits.com
  • putono5.com
  • cdn.putono5.com
  • d.putono5.com
  • a.putono5.com
(Damn! Within 15 minutes of posting this I found 2 more sites to add...)
  • adspserving.com
  • xl415.com

  • truequotes.org       (added 1-31-16)

I will add to that list as I find more entries.  These sites were found either via a direct pop-up, or by  reviewing the source code of the pop-up pages.  By the way, to view the HTML code to find these site just right mouse click the blank web page in the browser and select "view source".  Most entries will be bad but some are ok such as "http://www.w3.org" which is the group that sets web code standards.  Look for entries such as "http://<whateversite.com>", these are the "bad" sites it's trying to direct you to.

I have so far not been able to determine the method this thing uses to load.  It appears to be pretty stealthy.  I have some experience with PC forensics and to date I have yet to locate the root cause.  Many other posts seem to indicate a bogus browser plugin is the culprit.  I do occasionally use some but nothing I would be wary of.

In any case the best thing so far is placing entries for each site in the local PC hosts file.  This file is the first location the computer uses when it tries to identify the TCP/IP address of a site.  For those unfamiliar with this, the computer connects to the site IP address, not the site name, so it must translate between the two first.  Normally this is done automatically via DNS, but if a hosts file exists those entries take precedence.   In fact many viruses and Trojans use this file to redirect you to bad sites.

The hosts file is located here:  C:\Windows\System32\Drivers\Etc\hosts and no, there is no extension such as ".txt" on it.  In fact you may not be able to see the file without making a few adjustments in the Windows file explorer (look for info at Google on viewing hidden files).

By adding these entries the browser pop-up still occurs, but it cannot find the sites it wants and so does nothing but load a blank page.  This "should" have the effect of neutering the Trojan.  This won't stop it, or remove it, but it seems to slow it down.  Below is what the file will look like after editing.  This is a copy-&-paste of my own file:

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
# 127.0.0.1       localhost
# ::1             localhost
127.0.0.1  buy-targeted-traffic.com
127.0.0.1  orion.zerohorizon.net
127.0.0.1  oziris.zerohorizon.net
127.0.0.1  zerohorizon.net
127.0.0.1  onclicktop.com
127.0.0.1  fugdownload164.com
127.0.0.1  fugdownload173.com
127.0.0.1  ptp24.com
127.0.0.1  cdn.shorte.st
127.0.0.1  shorte.st
127.0.0.1  bundleworldbits.com
127.0.0.1  putono5.com
127.0.0.1  cdn.putono5.com
127.0.0.1  d.putono5.com
127.0.0.1  a.putono5.com
127.0.0.1 adspserving.com
127.0.0.1 xl415.com
127.0.0.1 truequotes.org

The entries prefixed with "127.0.0.1" cause a lookup of any of those sites to be redirected to "loopback" which is a local test address that goes no where.

Again, this is a "band-aid" and will not stop the Trojan.

My suspicion is that when this thing was installed it added a hook into the operating system at some non-common area.  It would appear to be a scheduled task of some sort but so far none of the tasks seem out of order.  Windows 7, 8, and 10 include a plethora of schedules tasks that do a myriad of things.  So many that it's hard to figure out what belongs.  There also may be settings added to the browser or system registry but so far I've identified nothing.  More than like this is running as a scheduled task somewhere due to the regular occurrence of it.

As I find new info I'll list it here.  If anyone reading this has comments or additions please leave a comment.  I would very much like to kill this thing without reinstalling Windows.  It's annoying more than anything and since I usually work in Linus it's even more annoying that I have to deal with it the few times I run Windows.

More later....


Dang.... Sorry I can't seem to post replies to comments.  Not sure why.   The comment about auditing is a good one.  I set it up on the "...\Local\Temp" folder and found Chrome.exe to be the culprit.  Trouble is that doesn't tell me what caused Chrome to execute a new instance.  I think I'll audit the exe itself and see what I get...


Update 02-21-16
Since I've mostly switch to Linux this issue hasn't been on the top of my priority list.  Either way I did make some headroom.  What I did was adjust the Windows startup to see what could be turned off to stop this.  I'm down to three items still shut off.  First is the Windows sidebar at "c:\program files\windows sidebar\sidebar.exe".  Next is the Windows error reporting server wer.exe at "c:\programdata\microsoft\windows\wer\wer.exe".  Lastly the Comodo "GeekBuddy" service, which I just read is a huge security risk anyway.  I use Comodo for Antivirus.  So far there have been NO new occurrences.  I would be curious to see if anyone else sees the same effect.

Update 03-11-16
Still cant reply to comments for some reason.  In response to the third comment...  Very interesting.  What files were involved and what did you do to identify and fix it?  The more detail we can post the more it might help someone else.  I'm planning to reinstall Windows (something I've done way too many times in my life) since It will only be left on my system for the things I absolutely can't run under Linux.  After that I expect the issue will be a moot point.

Update 05-28-16
In response to the May 19th post...  I would love to be able to test this but I have been exclusively running Linux for some time now. I never noticed any bogus accounts on my system and being a professional sysadmin I tend to have a pretty intimate knowledge of my systems.  Still this could be a valuable check should it find something.  Best of luck.

Update 07-27-16
I haven't suffered from this beasties sting for a number of months now.  Since switching over exclusively to Linux I've seen nothing.  I'm pretty sure I now know why.  I'm surprised I hadn't hit on this before due to the obvious symptoms.  I feel like an audience member watching a magic act and never noticing the assistant in the audience feeding the magician clues.

Like I said before... scheduled tasks...  the trick here is "trigger start tasks" which are a new feature with Windows 7 and later versions of the OS.  Basically Microsoft has once again given the bad guys a wonderful tool to pick our pockets.  True, it's a great feature, if you use your powers for good.  Trouble is when powerful things are placed in the hands of idiots it doesn't take long for them to become corrupted.  Heck I've used them at work for things myself.

Trigger start tasks are those that only fire off when a specific action is detected on the PC.  This could be logon, logoff, system idle, or... a mouse click in a certain place, like say a browser window....  See where I'm going here.  This is a well know tactic of adware and malware authors.  See this article for details: https://blog.malwarebytes.com/cybercrime/2015/03/scheduled-tasks/

The hard part is that Microsoft pretty much runs Windows on these tasks and they include a LOT of them out of the box.  Did you think the OS detected you inserting a DVD or memory key by magic?  Nope, trigger start tasks.

So, why you may ask, am I still blabbering about these PUPs (Potentially Unwanted Programs) again?  Well I initially got nailed by some damn Chrome extension, god only knows which.  I, like many of you sync my settings between browsers, and now my work browser running on Windows has dutifully synced whatever extension came with this crap and so once again I'm trying to rid my life of it.

I'm working on a few things to clean this up like PowerShell scripts to list and purge the trigger tasks.  The new task scheduler store task definitions as XML files with a ".JOB" extension so you can search for them and scan their contents.  Once I get something concrete I'll post it here.



Tuesday, January 19, 2016

Tegile Zebi Storage and PowerShell

For any of you that use SAN products from Tegile, they have had a REST API available since 2014 to access the SAN controllers, but it's never been really useful.  They tell me that this will be changing and they will actually have some commandlets available soon.

The user guide for the API is written from a PERL viewpoint and all the examples are in PERL.  That's great except that I use very little PERL in my day-to-day life.

I decided to try and convert to PowerShell and see what I could see.  There are a number of examples of using REST API via PowerShell around the net.  After trying variations of a number of them I would up with the code below.  It will pull out the Zebi version information from the array.

It's still very limited as to what you can do, but it's a start.  The full list of commands available are in the REST API user guide over at the Tegile support site.

This script is functional but not pretty.  It was basically just proving that it worked.



Clear-Host

$Username = Read-Host "Enter the username"
$Password = Read-Host "Enter the password"
$EncodedAuthorization = [System.Text.Encoding]::UTF8.GetBytes($Username + ':' + $Password)
$EncodedPassword = [System.Convert]::ToBase64String($EncodedAuthorization)
$IP = Read-Host "Enter the IP Address"
$BaseURL = 'https://' + $IP
$Headers = @{"Authorization"="Basic $($EncodedPassword)"}

$ResourceURL = "/zebi/api/v1/listShares"                      #--[ Remove the body option if using this ]--
$Body = '[["ZEBI_API_VERSION","ZEBI_APPLIANCE_VERSION","ZEBI_API_MINOR_VERSION","ZEBI_APPLIANCE_MODEL","ZEBI_GUI_VERSION"]]'     #--[ Edit this as required ]--

$ResourceURL = "/zebi/api/v1/listProjects"                      #--[ Remove the body option if using this ]--

$ResourceURL = "/zebi/api/v1/listSystemProperties"            #--[ Edit this as required ]--
$Body = '[["ZEBI_API_VERSION","ZEBI_APPLIANCE_VERSION","ZEBI_API_MINOR_VERSION","ZEBI_APPLIANCE_MODEL","ZEBI_GUI_VERSION"]]'     #--[ Edit this as required ]--

$URI = $BaseURL + $ResourceURL


Try
     {
     $Result = Invoke-RestMethod -Uri $URI -Method post -Headers $Headers -ContentType "application/json; charset=utf-8" -ErrorAction Stop -Body $Body
     }
Catch
     {
    $ErrorMessage = $_.Exception.Message
    $FailedItem = $_.Exception.ItemName
    write-host "Error Message : "$ErrorMessage -ForegroundColor yellow
    write-host "Failed Item : "$FailedItem -ForegroundColor yellow
     }


write-host `n"     URL: "$URI -ForegroundColor Yellow
write-host " API Ver: "$Result[0] -ForegroundColor Yellow
write-host "Zebi Ver: "$Result[1] -ForegroundColor Yellow


$Result

Friday, September 18, 2015

Great balls o' fire...

For anyone reading this that's looking for updated information on our situation, everything is good.

For those curious, I live in the Sierra foothills in California.  The Butte fire which started a week ago just ravaged my area.  70,000 acres burned, 360 homes destroyed, 2 fatalities.

The fire stopped about 500 yards from my home.  We were EXTREMELY lucky.  Seems to be a freak effect of the area we live in that caused the fire to shift and detour around our street.  All the houses on our street are fine, no damage, and just across the creek are foundations with no houses.  Damn scary.

The roads are still closed due to downed power poles and lines.  I understand the need to make things safe, and I am amazed at the herculean effect PG&E is taking to get power back in place,  but I am really getting tired of people telling that "the roads are still closed" just to watch the "special" people drive right down them.  Those would include anyone with any sort of "official" looking vehicle, regardless of who they may be.  And no, that doesn't include us.

Many thanks to the hundreds of fire fighters who focused on saving homes.  They did a phenomenal job and many homes were saved, mine among them.  Tragically two hold-outs lost their lives when they refused to leave when evacuations were called.  I do promise to not complain for the next year about the "special fire tax" being imposed on us due to gross mismanagement at CalFire, or state government, or whoever screwed that up, but after that I reserve the right to complain again.

Here some pics I took to give you an idea.  I left them full size so they are pretty big.

These are views from the house we stayed at while evacuated.  We got evacuated from there twice as well.






Yesterday I was able to get back home for the first time via some nasty logging roads to get the fridge cleaned out and disinfected.  Smelled pretty nasty.  The main road is still closed to it will be hit and miss on getting normal access.  For now looks like the 1.5 hour trip down the logging road.  That road is actually pretty scary.  There are some compounds back there miles from anything with high chain link fences with razor wire.  Every time I drive there I hear the banjos from Deliverance playing in my head...

I took this one from my porch.  You can see how close the fire got through the trees.  The hillside is all burned as well as the trees across the creek.