Tuesday, September 1, 2015

VMware Datastore Utilization Tracker

I like to keep an arsenal of automated scripts that run as scheduled tasks to report on the status of various things I manage.  Here is one of those.

Our virtualization environment uses a SAN backend as most do.  We run exclusively in NFS (NAS) mode.  This has numerous advantages (an explanation of which is outside the scope of this post).  I like to keep a rough idea of the gains and losses of each NFS share on a daily basis.

I run the following script automatically each day and it emails me the details of the gains or losses due to systems being added or removed.  I included a "noupdate" option that allows testing without processing the files.  That way you can rerun using the current file again and again for testing.

Results are very basic and color coded for gain/loss.  The raw dump files are stored in the script folder.  I have another process I use to import them into Excel and graph them over time.  If I get a chance I'll post that as well.

NOTE: This NOT the current version.  I am keeping current versions on the MS PowerShell Gallery at: https://www.powershellgallery.com/profiles/Kcmjr/.

<#======================================================================================
         File Name : Datastore-Tracker.ps1
   Original Author : Kenneth C. Mazie (kcmjr AT kcmjr DOT com)
                   :
       Description : Tracks SAN datastores over time.  Emails a daily report on changes.
                   :
             Notes : Normal operation is with no command line options.  Basic logs are written to C:\Scripts
                   : Optional argument: -Console $true (defaults to false)
                   :                    -NoUpdate $true (runs with current files and doesnt replace them for debugging)
                   :
          Warnings : The script is coded to switch colors between 2 vCenters.  Adjust as
                   : needed below where commented if you use more or less.
                   :  
             Legal : Public Domain. Modify and redistribute freely. No rights reserved.
                   : SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF
                   : ANY KIND. USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.
                   :
           Credits : Code snippets and/or ideas came from many sources including but
                   :   not limited to the following:
                   : Based on "Track Datastore Space script" Created by Hugo Peeters of www.peetersonline.nl
                   :
    Last Update by : Kenneth C. Mazie
   Version History : v1.0 - 09-16-14 - Original
    Change History : v1.1 - 08-28-15 - Edited to allow color coding of HTML output    
                   :
#=======================================================================================#>
#requires -version 3.0

Param(
     [bool]$Console = $False,
     [bool]$NoUpdate = $False
     )
If ($Console){$Script:Console = $true}

#--[ Uncomment to programatically hide the console ]--
#Add-Type -Name Window -Namespace Console -MemberDefinition '
#[DllImport("Kernel32.dll")]
#public static extern IntPtr GetConsoleWindow();
#
#[DllImport("user32.dll")]
#public static extern bool ShowWindow(IntPtr hWnd, Int32 nCmdShow);
#'
#$Script:consolePtr = [Console.Window]::GetConsoleWindow()
#[Console.Window]::ShowWindow($Script:consolePtr, 0)

Clear-host
$Script:Body = ""  
$ErrorActionPreference = "silentlycontinue"

#--[ Excel Non-Interactive Fix ]------------------------------------------------
If (!(Test-path -Path "C:\Windows\System32\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\System32\config\systemprofile\Desktop"}
If (!(Test-path -Path "C:\Windows\SysWOW64\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\SysWOW64\config\systemprofile\Desktop"}
#--[ Excel will crash when run non-interactively via a scheduled task if these folders don't exist ]--

#-------------------------------------------------------------------------------
#--[ NOTE: The script is set up for two vCenters.  Adjust the coloring below if you have more or less ]--
$Script:vCenters = @("vCenter1","vCenter2")  
#-------------------------------------------------------------------------------
If ($Script:NoUpdate){
       $Script:CurrentFile = $Script:PSScriptRoot+'\Temp_Current.xml'
       $Script:PreviousFile = $Script:PSScriptRoot+'\Datastores_Previous.xml'
       $Script:DifferenceFile = $Script:PSScriptRoot+'\Temp_Difference.txt'
}Else{
       $Script:CurrentFile = $Script:PSScriptRoot+'\Datastores_Current.xml'
       $Script:PreviousFile = $Script:PSScriptRoot+'\Datastores_Previous.xml'
       $Script:DifferenceFile = $Script:PSScriptRoot+'\Datastores_Difference.txt'
}
If (Test-Path $Script:DifferenceFile) {rm $Script:DifferenceFile -Confirm:$false -Force}
If (!(Get-PSSnapin | ? {$_.name -like “vmware.vimautomation.core”})) {Add-PSSnapin vmware.vimautomation.core}
$Script:Digits = 2
$Script:From = "DailyRepoprts@mydomain.com"                                     #--[ Who the email is being sent by ]--
$Script:UserName = "mydomain\username"                                          #--[ The user to connect to vCenter with ]--
$Script:Password = "userpassword"                                               #--[ That users password ]--
$Script:EmailTargets = "user1.domain.com, user2.domain.com"                     #--[ a comma seperated list of email recipients ]--
$Script:SMTPServer = "HostName.domain.int"                                      #--[ what SMTP server to use ]--

Function SendEmail ($Script:OutBody){                                           #--[ Email settings ]--
       $Script:HTML = $true
       $Script:Email = new-object System.Net.Mail.MailMessage
       $Script:Email.From = $Script:From
       $Script:Email.To.Add($Script:EmailTargets)
       $Script:Email.Subject = "SAN Daily Utilization Status"
       $Script:Email.IsBodyHtml = $true
       $Script:Email.Body = $Script:ReportBody
       $Script:smtp = new-object System.Net.Mail.SmtpClient($Script:SMTPServer)
       $Script:smtp.Send($Script:Email)
}

#--[ Add header to html log file ]----------------------------------------------
$Script:ReportBody = @()
$Script:ReportBody += '
<style type="text/css">
       table.myTable { border:5px solid black;border-collapse:collapse; }
    table.myTable td { border:2px solid black;padding:5px}
       table.myTable th { border:2px solid black;padding:5px;background: #949494 }
       table.bottomBorder { border-collapse:collapse; }
       table.bottomBorder td, table.bottomBorder th { border-bottom:1px dotted black;padding:5px; }
</style>
<left><h1>&nbsp;&nbsp;&nbsp;- SAN Volume Utilization Report -</h1></left>
The following report displays the current SAN volume usage and the percent of<br>
change from yesterdays usage.  The raw data files are retained with the script<br>
for use in long term utilization tracking purposes.<br><br>'
$Script:ReportBody += '<table class="myTable"><tr><th>vCenter</th><th>Volume Name</th><th>Percent Free</th><th>Difference</th></tr>'

If (!(Test-Path "$PSScriptRoot\creds.txt")){
  $Script:Credential = Get-Credential
  $Script:Credential.Password | ConvertFrom-SecureString | Set-Content "$PSScriptRoot\creds.txt"
}

#--[ Use if you need to use a credential file, otherwise current creds are used ]--
#$Encrypted = Get-Content "$Folder\creds.txt" | ConvertTo-SecureString
#$Credential = New-Object System.Management.Automation.PsCredential($Username, $Encrypted)
#$LogFileName = "SAN-Vol_{0:MM-dd-yyyy_mm}_Stats.log" -f (Get-Date)
$Script:myColCurrent = @()                                                       #--[ Create an array to hold the output ]--

ForEach ($Script:VIServer in $Script:vCenters){
       If ($Script:Console){Write-Host "`r`nProcessing: $Script:VIServer" -ForegroundColor Cyan}
       $Script:VC = Connect-VIServer -Server $Script:VIServer -user $Script:UserName -pass $Script:Password #-Credential $Script:Credential #--[ Connect to Virtual Center ]--
       $Script:datastores = Get-Datastore | Sort-Object Name | Select -Unique            #--[ Get all datastores and put them in alphabetical order & remove accidental duplicates ]--
       ForEach ($Script:store in $Script:datastores)                                     #--[ Loop through datastores ]--
              if ($Script:store -notlike "*Local*"){
                     $Script:myObj = "" | Select-Object vCenter, Name, CapacityGB, UsedGB, PercFree              #--[ Create a custom object and define its properties ]--
                     #--[ Set the values of each property ]--
                     $Script:myObj.vCenter = $Script:VIServer
                     $Script:myObj.Name = $Script:store.name
                     $Script:myObj.CapacityGB = [math]::Round($Script:store.capacityMB/1024,$Script:digits)
                     $Script:myObj.UsedGB = [math]::Round(($Script:store.CapacityMB - $Script:store.FreeSpaceMB)/1024,$Script:digits)
                     $Script:myObj.PercFree = [math]::Round(100*$Script:store.FreeSpaceMB/$Script:store.CapacityMB,$Script:digits)
                     $Script:myColCurrent += $Script:myObj   
                     #--[ Add the object to the output array ]--    
              }
       }
       Disconnect-VIServer -Confirm:$False                                      #--[ Disconnect from Virtual Center ]--
}

$Script:myColCurrent | Export-Clixml -Path $Script:CurrentFile                  #--[ Export the output to an xml file; the new Current file ]--

$Script:CurrentDate = (Get-Item $Script:CurrentFile).LastWriteTime              #--[ Get file dates for new file names ]--
$Script:PreviousDate = (Get-Item $Script:PreviousFile).LastWriteTime

#--[Compare the Current information to that in the Previous file ]--------------
If (Test-Path $Script:PreviousFile){                                            #--[ Check if a Previous file exists ]--
       $Script:myColPrevious = Import-Clixml $Script:PreviousFile               #--[ Import the Previous file ]--
       $Script:myColCurrent= Import-Clixml $Script:CurrentFile                  #--[ Import the Previous file ]--

       $Script:myColDiff = @()                                                                                   #--[ Create an array to hold the differences ]--
       ForEach ($Script:myObjCurrent in $Script:myColCurrent){                  #--[ Loop through the current datastores ]--
           $Script:VCCurrent = $Script:myObjCurrent.vCenter
              $Script:RowData = ""
              $Script:diff = Compare-Object ($Script:myColPrevious | Where { $_.Name -eq $Script:myObjCurrent.Name }) $Script:myObjCurrent -Property PercFree  # The actual compare command
              $Script:myObjDiff = "" | Select-Object vCenter, VolName, PercentFree, Diff #--[ Create a custom object and properties for outputting results ]--
              $Script:myObjDiff.vCenter = $Script:myObjCurrent.vCenter                          #--[ Setting the values of each property ]--
              $Script:myObjDiff.VolName = $Script:myObjCurrent.Name                            
              $Script:myObjDiff.PercentFree = $Script:myObjCurrent.PercFree
                    
              #--[ The most important property is the calculated difference between the current and previous values of PercFree. You can substitute it for UsedGB if you like. ]--
              $Script:myObjDiff.Diff = ($Script:diff | Where { $_.SideIndicator -eq '=>' }).PercFree - ($Script:diff | Where { $_.SideIndicator -eq '<=' }).PercFree
                    
              If (($Script:myObjDiff.Diff -eq "") -or ($Script:myObjDiff.Diff -eq $null)){$Script:myObjDiff.Diff = "0.00"}
              $Script:myColDiff += $Script:myObjDiff                                                   #--[ Adding it to the output array ]--

              $Script:BGColor = "#dfdfdf"                                         #--[ Grey cell background ]--
              $Script:RowData += '<tr>'
                          
              if($Script:VCCurrent -eq "vcenter1"){                               #--[ NOTE: Tweak this name to keep colors rotating ]--
                     $Script:FGColor = "#408080"                                  #--[ Color to distinguish 1st vCenter ]--
              }ElseIf($Script:VCCurrent -eq "vcenter2"){                          #--[ NOTE: Tweak this name to keep colors rotating ]--
                     $Script:FGColor = "#808000"                                  #--[ Color to distinguish 2nd vCenter ]--
              }Else{
                     $Script:FGColor = "#000000"                                  #--[ Default color, black ]--
                     }
             
              $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:myObjDiff.vCenter + '</td>'           

              $Script:FGColor = "#000000"
              $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:myObjDiff.VolName + '</td>'           
              $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:myObjDiff.PercentFree + '</td>'       
              If ($Script:myObjDiff.Diff -eq "0.00"){
                     $Script:FGColor = "#000000"                                   #--[ Black foreground for no change ]--
                     $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:myObjDiff.Diff
              }ElseIf ($Script:myObjDiff.Diff -lt 0){
                     $Script:FGColor = "#700000"                                   #--[ Red foreground for storage loss]--
                     $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '><strong>' + $Script:myObjDiff.Diff + '</strong>'
              }Else
                     $Script:FGColor = "#007000"                                   #--[ Green foreground for storage gain ]--
                     $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '><strong>' + $Script:myObjDiff.Diff + '</strong>'
              }
                          
              $Script:RowData += '</td></tr>'
              $Script:ReportBody += $Script:RowData
              Clear-Variable diff -ErrorAction "SilentlyContinue"                                      #--[ Clearing the variable used inside the loop to prevent incorrect output in case of problems setting the variable! ]--
       }
       If ($Script:myColDiff.Length -eq 0){$Script:myColDiff = "No changes since last check."#--[ If nothing changed, we don't want an empty file. ]--
       $Script:myColDiff | Format-Table -AutoSize | Out-File $Script:DifferenceFile -Force -Append         #--[ And we conclude by outputting the results to a text file, which can be emailed or printed. ]--
}

If ($Script:myColDiff.Length -ne 0){
       (Get-Content $Script:DifferenceFile) | Where { $_ } | Set-Content $Script:DifferenceFile
       (Get-Content $Script:DifferenceFile) | Where {$_ -notmatch '----'} | Set-Content $Script:DifferenceFile
       Add-Content $Script:DifferenceFile –value "`nCurrent-Report $Script:CurrentDate &nbsp; &nbsp;`nPrevious-Report $Script:PreviousDate &nbsp; &nbsp;"
       $Script:FGColor = "#000000"
       $Script:BGColor = "#bbbbbb"
       $Script:RowData += '<tr>'
       $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>Current Report</td>'    
       $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:CurrentDate + '</td>'  
       $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>Previous Report</td>'   
       $Script:RowData += '<td bgcolor=' + $Script:BGColor + '><font color=' + $Script:FGColor + '>' + $Script:PreviousDate + '</td>' 
       $Script:RowData += '</tr>'
       $Script:ReportBody += $Script:RowData
}

$Script:ReportBody += '</table><br><br>'
<#--[ Enable this section to add notes to the bottom of the report. ]--
$Script:ReportBody += '
<ul>
    <li>Only powered-on Vms are shown since the script relies on WMI to get the block size.</li>
       <li>"Vol", or "Volume" refers to a Windows disk partition.</li>
       <li>The hidden system partition, which is typically Vol 0 on Disk 0, is not shown.</li>
       <li>Disk numbers are assigned beginning with 0 (zero).</li>
       <li>Partition numbers are assigned beginning with 1 (not 0).</li>
</ul>'
#>
$Script:OutBody = Get-Content $Script:DifferenceFile #| Out-String
SendEmail $Script:OutBody

If ($Script:Console){Get-Content $Script:DifferenceFile | Out-String}

#--[ Cleanup ]--
$Script:attachment.Dispose()
$Script:msg.Attachments.Dispose()
$Script:msg.Dispose()
$Script:smtp.Dispose()
If ($Script:NoUpdate){
       If (Test-Path $Script:DifferenceFile){Remove-Item -Path $Script:DifferenceFile -force}
       If (Test-Path $Script:CurrentFile){Remove-Item -Path $Script:CurrentFile -force}
}Else{
       If (Test-Path $Script:DifferenceFile){rename-Item -Path $Script:DifferenceFile -newname ("$PSScriptRoot\SAN-Vol_{0:MM-dd-yyyy}_Diff.log" -f (Get-Date))}
       If (Test-Path $Script:PreviousFile){Remove-Item -Path $Script:PreviousFile -Force} #If a Previous file exists remove it
       If (Test-Path $Script:CurrentFile){Copy-Item -Path $Script:CurrentFile -Destination ("$PSScriptRoot\SAN-Vol_{0:MM-dd-yyyy}_Stats.xml" -f (Get-Date))}
       If (Test-Path $Script:CurrentFile){Rename-Item -Path $Script:CurrentFile -NewName $Script:PreviousFile}                    # If a Current file exists, rename this Current file to Previous
}
$Script:Credential = ""
$Script:CurrentFile = ""
$Script:PreviousFile = ""
$Script:DifferenceFile = ""
$Script:OutBody = ""

If ($Script:Console){Write-Host '--- Completed ---'}

#iex '$PSScriptRoot\reload.bat'  #--[ Used for input file refresh while debugging ]--

Time variance across a domain.

In my never ending quest to rid myself of time sync issues I created the following PowerShell script.  I wrote it some time ago and can't recall if I borrowed bits so if you see what looks like your code snippets in there forgive me.

The script scans the current domain using the current users credentials.  It generates an Excel spreadsheet and populates it with the names of all Windows systems it finds.  I have an exclusion to bypass anything with ESX in the name so my VMware hosts are ignored.  It then does a time comparison between the PDC and the target PC and logs the results to the spreadsheet.

<#==============================================================================
         File Name : Domain-Time-Validator.ps1
   Original Author : Kenneth C. Mazie (kcmjr AT kcmjr DOT com)
                   :
       Description : Scans a domain and reports the time variance between the PDC and the target
                   :
             Notes : Normal operation is with no command line options.  Spreadsheet
                   : is written to C:\Scripts.  Console display counts down to zero
                   : so you know where it's at.
                   :
        Arguements : -console $true (defaults to false) use this to display to local console.
                   :
                   :
          Warnings : None
                   :  
             Legal : Public Domain. Modify and redistribute freely. No rights reserved.
                   : SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF
                   : ANY KIND. USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.
                   :
           Credits : Code snippets and/or ideas came from many sources including but
                   : not limited to the following:
                   :
    Last Update by : Kenneth C. Mazie
   Version History : v1.0 - 06-10-14 - Original
    Change History : v1.1 - 00-00-00 - Tweaked inline documentation
                   :
#=============================================================================#>
#requires -version 3.0

Param(
     [bool]$Console = $False
     )
If ($Console){$Script:Console = $true}   #--[ Just to be sure... ]--
Clear-Host

#--[ Excel Non-Interactive Fix ]------------------------------------------------
If (!(Test-path -Path "C:\Windows\System32\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\System32\config\systemprofile\Desktop"}
If (!(Test-path -Path "C:\Windows\SysWOW64\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\SysWOW64\config\systemprofile\Desktop"}
#--[ Excel will crash when run non-interactively via a scheduled task if these folders don't exist ]--

$Datafile = "C:\Scripts\TimeSyncData.xlsx"
$Row = 0
$Col = 1
#--[ Create new Excel object ]--------------------------------------------------
$Excel = New-Object -Com Excel.Application
$Excel.visible = $true
$Excel.DisplayAlerts = $false
$Excel.ScreenUpdating = $true
$Excel.UserControl = $true
$Excel.Interactive = $true
$Workbook = $Excel.Workbooks.Add()
$WorkSheet = $Workbook.WorkSheets.Item(1)
# Write Worksheet title
#$WorkSheet.Cells.Item(1,1) = "Local Administrator Group Membership Report - $DateTime"
#$WorkSheet.Cells.Item(1,1).font.bold = $true
#$WorkSheet.Cells.Item(1,1).font.underline = $true
#$WorkSheet.Cells.Item(1,1).font.size = 18
# Write worksheet column headers
$Row ++
$WorkSheet.Cells.Item($Row,$Col) = "TARGET:"
$WorkSheet.Cells.Item($Row,$Col).font.bold = $true
$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4
$Col++
$WorkSheet.Cells.Item($Row,$Col) = "PDC Time:"
$WorkSheet.Cells.Item($Row,$Col).font.bold = $true
$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4
$Col++
$WorkSheet.Cells.Item($Row,$Col) = "Target Time:"
$WorkSheet.Cells.Item($Row,$Col).font.bold = $true
$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4
$Col++
$WorkSheet.Cells.Item($Row,$Col) = "Variance:"
$WorkSheet.Cells.Item($Row,$Col).font.bold = $true
$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1
#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4
$Resize = $WorkSheet.UsedRange
[void]$Resize.EntireColumn.AutoFit()
$Row++
$Col = 1

#--[ Identify the PDC ]---------------------------------------------------------
$Script:PDC = (Get-ADDomain).pdcemulator
$Script:Source = $Script:PDC.split(".")[0]

#--[ Get the list of targets ]--------------------------------------------------
$Computers = Get-ADComputer -Filter {operatingsystem -like "*windows*" -and name -notlike "*esx*"} | sort name
$Total = $Computers.Count

#--[ Cycle through targets ]----------------------------------------------------
ForEach($Script:Target in $Computers ){
       $Script:Target = $Script:Target.name
      
       $Col = 1
#      $WorkSheet.Cells.Item(1,1).font.colorindex = 1
       $WorkSheet.Cells.Item($row,$col) = $Script:Target
       $Col++
       If ($Script:Console){Write-Host "--[ $Script:Target ]--[ $Total ]--------------------------------------------" -ForegroundColor Cyan }
       If(Test-Connection -ComputerName $Script:Target -count 1 -BufferSize 16 -ErrorAction SilentlyContinue ) {
              $Script:SourceDate = invoke-command -ComputerName $Script:Source -ScriptBlock {get-date}
              $Script:TargetDate = invoke-command -ComputerName $Script:Target -ScriptBlock {get-date}
              $Script:TimeSpan = [DateTime]$Script:SourceDate - [DateTime]$Script:TargetDate
             
              If($Script:TimeSpan.seconds -lt 0){$Script:Offset = "Plus"}Else{$Script:Offset = "Minus"}
             
              $Script:Inspect = new-timespan -Start (icm $Source {get-date}) -end (icm $Script:Target {get-date})
      
              If ($Script:Console){Write-Host "PDC Time    :" $Script:SourceDate}
              $WorkSheet.Cells.Item($row,$col) = $Script:SourceDate
              $Col++
              If ($Script:Console){Write-Host "Target Time :" $Script:TargetDate}
              $WorkSheet.Cells.Item($row,$col) = $Script:TargetDate
              $Col++
              If ($Script:Console){Write-Host "Variance    : " -NoNewline }
              If($Script:TimeSpan.minutes -ge 5 -or $Script:TimeSpan.minutes -lt -5){
                     $Color = "red"
                     $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 9
              }Else{
                     $Color = "green"
                     $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 10
              }

              $WorkSheet.Cells.Item($row,$col) = $Script:TimeSpan.Hours.tostring()+" Hours "+$Script:TimeSpan.Minutes.tostring()+" Min "+$Script:TimeSpan.seconds.tostring()+" Sec"
              If ($Script:Console){Write-Host $Script:TimeSpan.Hours.tostring() "Hours" $Script:TimeSpan.minutes.tostring() "Min" $Script:TimeSpan.seconds.tostring() "Sec" -ForegroundColor $Color }
       }Else{
              $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 9
              $WorkSheet.Cells.Item($Row,$Col) = "Unable to connect to target..."
              If ($Script:Console){Write-Host "Unable to connect to target..." -ForegroundColor Red }
       }
       $Total --
       $Row++
       $Resize = $WorkSheet.UsedRange
       [void]$Resize.EntireColumn.AutoFit()
}


If ($Script:Console){Write-Host "--- Completed ---" -ForegroundColor Red  }

Monday, August 31, 2015

There's no time like Windows time...

I haven't been able to post in a while due to the workload at work.  I though this was important enough to post in the event anyone else ran into it.  From my research dealing with the Windows time service it is something that will bite EVERYONE at one time or another.   For years I've been battling with time sync in the various domains I've managed, and it's always a nightmare. Considering that Bill Gates is the only person I've ever found that has successfully altered the flow of time, (Windows file copy dialog status bar sitting at 98% for hours...) this issue should never have been an issue.

But, I think I just found the answer.

Microsoft has plenty of documentation about W32TM and I cannot recall seeing this noted anywhere. I find it really hard to believe that Microsoft has never hammered all the nails in the coffin of this service.  I've read KB article after KB article, blog after blog, and never found the definitive answer to what the heck is going on with Windows time.  This time I feel like I may have finally slayed the domain time beast.  Like I said, I looked all over for information on how to make things sync properly in a domain.

Turns out the fix was simple, it only took around 8 years and hundreds of blogs and support sites to finally stumble over the answer.  Why in heaven's name this is pasted all over in big red crayon letters I don't know (but there they are, big red letters :-P ).   And to make matters worse I found this bombshell:  https://support.microsoft.com/en-us/kb/939322.  I am quoting from the article at that link:
We do not guarantee and we do not support the accuracy of the W32Time service between nodes on a network. The W32Time service is not a full-featured NTP solution that meets time-sensitive application needs. The W32Time service is primarily designed to do the following: 
  • Make the Kerberos version 5 authentication protocol work. 
  • Provide loose sync time for client computers.The W32Time service cannot reliably maintain sync time to the range of 1 to 2 seconds. Such tolerances are outside the design specification of the W32Time service.
WHAT???  They don't support it?  No wonder.

Anyway, after wiping the blood from my forehead from banging it against the desk I returned to my research on answers for the Windows error "A good time server could not be located" (and no there are no night clubs or bars involved here...) I found a blog entry over at Pete Long's blog in the UK discussing this exact issue.  The first thing he discusses is running DCDIAG and noting a failure in the DC advertising within AD.  I was seeing the same thing.  He recommended removing any and all GPO settings for the time service on policies for the DCs, so what the heck, I gave it a try.

Turns out when you use a Windows domain controller as a time source you CANNOT (or should not) use any GPOs to apply settings to them or it screws them up.  Huh?  I always  use GPOs to set the time settings on DC's, I always have, it's the way I was taught to do it. Unfortunately, when you do, it stops the server from properly advertising as a reliable NTP time source in AD. 

By design when any Windows computer joins a domain it adopts the domain PDC as its master time source.  As a fallback it will try various sources including time.windows.com, and then default to the local hardware clock for reference.  We block NTP at our firewall so no system can use time.windows.com to sync.  And since there didn't appear to be a valid time server on the domain our systems were using their local hardware clock as a fallback. Ugh.

The culprit in our case was that the PDC server wasn’t advertising itself as a reliable time source in AD.  It was getting valid time, and would return that if queried, but if it doesn’t advertise in AD none of the domain members can use it for a time source.    Once all domain policies were disabled on the PDC domain controller and the local settings manually reset on it, everything started working.

So there it is.  After all that ranting and researching just removing the GPO settings fixed it.

To test things is simple.  From an Admin CMD prompt run these commands:

net time  
This should return the time at the domain controller you authenticated from.  Ex:
Current time at \\DC02.mydomain.int is 8/31/2015 2:29:05 PM
If there is no time source advertised in AD you see this:
Could not locate a time-server.

w32tm /query /source
This will list the domain controller you are getting time sync from. Should be the PDC.

w32tm /resync /nowait
This will force your local NTP client to immediately sync from the domain time source.

w32tm /resync /rediscover
This forces the local NTP client query for a valid domain source and then resync.

w32tm /query /status
This reports the current status of your local NTP client.  Note the items in red.
Leap Indicator: 0(no warning)
Stratum: 2 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0312500s
Root Dispersion: 16.0100000s
ReferenceId: 0xC0A85514 (source IP:  192.168.0.20)
Last Successful Sync Time: 8/31/2015 2:32:45 PM
Source: DC01.mydomain.int
Poll Interval: 10 (1024s)

w32tm /query /configuration

This reports your current configuration.  (long output not listed here)

Below is a short list of the many references I've used to isolate this issue:
NTP time source lists:
If you are looking for a freeware local time sync daemon I recommend the one from here: https://www.meinbergglobal.com/english/sw/ntp.htm.  It is based on the open source client from ntp.org and is a very professional package.  It installs as a configurable Windows service.

I hope all this helps someone else.  I spent plenty of time locating it, perhaps it might save you some.