Time variance across a domain.

In my never ending quest to rid myself of time sync issues I created the following PowerShell script.  I wrote it some time ago and can't recall if I borrowed bits so if you see what looks like your code snippets in there forgive me.

The script scans the current domain using the current users credentials.  It generates an Excel spreadsheet and populates it with the names of all Windows systems it finds.  I have an exclusion to bypass anything with ESX in the name so my VMware hosts are ignored.  It then does a time comparison between the PDC and the target PC and logs the results to the spreadsheet.

<#==============================================================================

         File Name : Domain-Time-Validator.ps1

   Original Author : Kenneth C. Mazie (kcmjr AT kcmjr DOT com)

                   :

       Description : Scans a domain and reports the time variance between the PDC and the target

                   :

             Notes : Normal operation is with no command line options.  Spreadsheet

                   : is written to C:\Scripts.  Console display counts down to zero

                   : so you know where it's at.

                   :

        Arguements : -console $true (defaults to false) use this to display to local console.

                   :

                   :

          Warnings : None

                   :  

             Legal : Public Domain. Modify and redistribute freely. No rights reserved.

                   : SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF

                   : ANY KIND. USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.

                   :

           Credits : Code snippets and/or ideas came from many sources including but

                   : not limited to the following:

                   :

    Last Update by : Kenneth C. Mazie

   Version History : v1.0 - 06-10-14 - Original

    Change History : v1.1 - 00-00-00 - Tweaked inline documentation

                   :

#=============================================================================#>

#requires -version 3.0

Param(

     [bool]$Console = $False

     )

If ($Console){$Script:Console = $true}   #--[ Just to be sure... ]--

Clear-Host

#--[ Excel Non-Interactive Fix ]------------------------------------------------

If (!(Test-path -Path "C:\Windows\System32\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\System32\config\systemprofile\Desktop"}

If (!(Test-path -Path "C:\Windows\SysWOW64\config\systemprofile\Desktop")){New-Item -Type Directory -Name "C:\Windows\SysWOW64\config\systemprofile\Desktop"}

#--[ Excel will crash when run non-interactively via a scheduled task if these folders don't exist ]--

$Datafile = "C:\Scripts\TimeSyncData.xlsx"

$Row = 0

$Col = 1

#--[ Create new Excel object ]--------------------------------------------------

$Excel = New-Object -Com Excel.Application

$Excel.visible = $true

$Excel.DisplayAlerts = $false

$Excel.ScreenUpdating = $true

$Excel.UserControl = $true

$Excel.Interactive = $true

$Workbook = $Excel.Workbooks.Add()

$WorkSheet = $Workbook.WorkSheets.Item(1)

# Write Worksheet title

#$WorkSheet.Cells.Item(1,1) = "Local Administrator Group Membership Report - $DateTime"

#$WorkSheet.Cells.Item(1,1).font.bold = $true

#$WorkSheet.Cells.Item(1,1).font.underline = $true

#$WorkSheet.Cells.Item(1,1).font.size = 18

# Write worksheet column headers

$Row ++

$WorkSheet.Cells.Item($Row,$Col) = "TARGET:"

$WorkSheet.Cells.Item($Row,$Col).font.bold = $true

$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4

$Col++

$WorkSheet.Cells.Item($Row,$Col) = "PDC Time:"

$WorkSheet.Cells.Item($Row,$Col).font.bold = $true

$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4

$Col++

$WorkSheet.Cells.Item($Row,$Col) = "Target Time:"

$WorkSheet.Cells.Item($Row,$Col).font.bold = $true

$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4

$Col++

$WorkSheet.Cells.Item($Row,$Col) = "Variance:"

$WorkSheet.Cells.Item($Row,$Col).font.bold = $true

$WorkSheet.Cells.Item($Row,$Col).HorizontalAlignment = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).LineStyle = 1

#$WorkSheet.Cells.Item($Row,$Col).Borders.Item(10).Weight = 4

$Resize = $WorkSheet.UsedRange

[void]$Resize.EntireColumn.AutoFit()

$Row++

$Col = 1

#--[ Identify the PDC ]---------------------------------------------------------

$Script:PDC = (Get-ADDomain).pdcemulator

$Script:Source = $Script:PDC.split(".")[0]

#--[ Get the list of targets ]--------------------------------------------------

$Computers = Get-ADComputer -Filter {operatingsystem -like "*windows*" -and name -notlike "*esx*"} | sort name

$Total = $Computers.Count

#--[ Cycle through targets ]----------------------------------------------------

ForEach($Script:Target in $Computers ){

       $Script:Target = $Script:Target.name

      

       $Col = 1

#      $WorkSheet.Cells.Item(1,1).font.colorindex = 1

       $WorkSheet.Cells.Item($row,$col) = $Script:Target

       $Col++

       If ($Script:Console){Write-Host "--[ $Script:Target ]--[ $Total ]--------------------------------------------" -ForegroundColor Cyan }

       If(Test-Connection -ComputerName $Script:Target -count 1 -BufferSize 16 -ErrorAction SilentlyContinue ) {

              $Script:SourceDate = invoke-command -ComputerName $Script:Source -ScriptBlock {get-date}

              $Script:TargetDate = invoke-command -ComputerName $Script:Target -ScriptBlock {get-date}

              $Script:TimeSpan = [DateTime]$Script:SourceDate - [DateTime]$Script:TargetDate

             

              If($Script:TimeSpan.seconds -lt 0){$Script:Offset = "Plus"}Else{$Script:Offset = "Minus"}

             

              $Script:Inspect = new-timespan -Start (icm $Source {get-date}) -end (icm $Script:Target {get-date})

      

              If ($Script:Console){Write-Host "PDC Time    :" $Script:SourceDate}

              $WorkSheet.Cells.Item($row,$col) = $Script:SourceDate

              $Col++

              If ($Script:Console){Write-Host "Target Time :" $Script:TargetDate}

              $WorkSheet.Cells.Item($row,$col) = $Script:TargetDate

              $Col++

              If ($Script:Console){Write-Host "Variance    : " -NoNewline }

              If($Script:TimeSpan.minutes -ge 5 -or $Script:TimeSpan.minutes -lt -5){

                     $Color = "red"

                     $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 9

              }Else{

                     $Color = "green"

                     $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 10

              }

              $WorkSheet.Cells.Item($row,$col) = $Script:TimeSpan.Hours.tostring()+" Hours "+$Script:TimeSpan.Minutes.tostring()+" Min "+$Script:TimeSpan.seconds.tostring()+" Sec"

              If ($Script:Console){Write-Host $Script:TimeSpan.Hours.tostring() "Hours" $Script:TimeSpan.minutes.tostring() "Min" $Script:TimeSpan.seconds.tostring() "Sec" -ForegroundColor $Color }

       }Else{

              $WorkSheet.Cells.Item($Row,$Col).font.colorindex = 9

              $WorkSheet.Cells.Item($Row,$Col) = "Unable to connect to target..."

              If ($Script:Console){Write-Host "Unable to connect to target..." -ForegroundColor Red }

       }

       $Total --

       $Row++

       $Resize = $WorkSheet.UsedRange

       [void]$Resize.EntireColumn.AutoFit()

}

If ($Script:Console){Write-Host "--- Completed ---" -ForegroundColor Red  }

There's no time like Windows time...

I haven't been able to post in a while due to the workload at work.  I though this was important enough to post in the event anyone else ran into it.  From my research dealing with the Windows time service it is something that will bite EVERYONE at one time or another.   For years I've been battling with time sync in the various domains I've managed, and it's always a nightmare. Considering that Bill Gates is the only person I've ever found that has successfully altered the flow of time, (Windows file copy dialog status bar sitting at 98% for hours...) this issue should never have been an issue.

But, I think I just found the answer.

Microsoft has plenty of documentation about W32TM and I cannot recall seeing this noted anywhere. I find it really hard to believe that Microsoft has never hammered all the nails in the coffin of this service.  I've read KB article after KB article, blog after blog, and never found the definitive answer to what the heck is going on with Windows time.  This time I feel like I may have finally slayed the domain time beast.  Like I said, I looked all over for information on how to make things sync properly in a domain.

Turns out the fix was simple, it only took around 8 years and hundreds of blogs and support sites to finally stumble over the answer.  Why in heaven's name this is pasted all over in big red crayon letters I don't know (but there they are, big red letters :-P ).   And to make matters worse I found this bombshell:  https://support.microsoft.com/en-us/kb/939322.  I am quoting from the article at that link:
We do not guarantee and we do not support the accuracy of the W32Time service between nodes on a network. The W32Time service is not a full-featured NTP solution that meets time-sensitive application needs. The W32Time service is primarily designed to do the following: 

• Make the Kerberos version 5 authentication protocol work. 
• Provide loose sync time for client computers.The W32Time service cannot reliably maintain sync time to the range of 1 to 2 seconds. Such tolerances are outside the design specification of the W32Time service.

WHAT???  They don't support it?  No wonder.

Anyway, after wiping the blood from my forehead from banging it against the desk I returned to my research on answers for the Windows error "A good time server could not be located" (and no there are no night clubs or bars involved here...) I found a blog entry over at Pete Long's blog in the UK discussing this exact issue.  The first thing he discusses is running DCDIAG and noting a failure in the DC advertising within AD.  I was seeing the same thing.  He recommended removing any and all GPO settings for the time service on policies for the DCs, so what the heck, I gave it a try.

Turns out when you use a Windows domain controller as a time source you CANNOT (or should not) use any GPOs to apply settings to them or it screws them up.  Huh?  I always  use GPOs to set the time settings on DC's, I always have, it's the way I was taught to do it. Unfortunately, when you do, it stops the server from properly advertising as a reliable NTP time source in AD. 

By design when any Windows computer joins a domain it adopts the domain PDC as its master time source.  As a fallback it will try various sources including time.windows.com, and then default to the local hardware clock for reference.  We block NTP at our firewall so no system can use time.windows.com to sync.  And since there didn't appear to be a valid time server on the domain our systems were using their local hardware clock as a fallback. Ugh.

The culprit in our case was that the PDC server wasn’t advertising itself as a reliable time source in AD.  It was getting valid time, and would return that if queried, but if it doesn’t advertise in AD none of the domain members can use it for a time source.    Once all domain policies were disabled on the PDC domain controller and the local settings manually reset on it, everything started working.

So there it is.  After all that ranting and researching just removing the GPO settings fixed it.

To test things is simple.  From an Admin CMD prompt run these commands:

net time  

This should return the time at the domain controller you authenticated from.  Ex:

Current time at \\DC02.mydomain.int is 8/31/2015 2:29:05 PM

If there is no time source advertised in AD you see this:

Could not locate a time-server.

w32tm /query /source

This will list the domain controller you are getting time sync from. Should be the PDC.

w32tm /resync /nowait

This will force your local NTP client to immediately sync from the domain time source.

w32tm /resync /rediscover

This forces the local NTP client query for a valid domain source and then resync.

w32tm /query /status

This reports the current status of your local NTP client.  Note the items in red.

Leap Indicator: 0(no warning)

Stratum: 2 (secondary reference - syncd by (S)NTP)

Precision: -6 (15.625ms per tick)

Root Delay: 0.0312500s

Root Dispersion: 16.0100000s

ReferenceId: 0xC0A85514 (source IP:  192.168.0.20)

Last Successful Sync Time: 8/31/2015 2:32:45 PM

Source: DC01.mydomain.int

Poll Interval: 10 (1024s)

w32tm /query /configuration

This reports your current configuration.  (long output not listed here)

Below is a short list of the many references I've used to isolate this issue:

• https://technet.microsoft.com/en-us/library/w32tm(v=WS.10).aspx
• https://support.microsoft.com/en-us/kb/816042 
• https://support.microsoft.com/en-us/kb/135646
• https://technet.microsoft.com/en-us/library/Cc773013(v=WS.10).aspx
• https://technet.microsoft.com/en-us/library/a0fcd250-e5f7-41b3-b0e8-240f8236e210(v=ws.10)#w2k3tr_times_intro
• https://technet.microsoft.com/en-us/library/cc773263.aspx
• https://technet.microsoft.com/en-us/library/Cc779145(v=WS.10).aspx
• http://blogs.msdn.com/b/w32time/archive/2009/08/07/net-time-and-w32time.aspx
• http://serverfault.com/questions/98416/time-not-propagating-to-machines-on-windows-domain
• http://www.petenetlive.com/KB/Article/0000112.htm
• https://community.spiceworks.com/how_to/65413-configure-dc-to-synchronize-time-with-external-ntp-server
• http://www.techrepublic.com/article/synchronize-time-throughout-your-entire-windows-network/
• http://www.eecis.udel.edu/~mills/ntp/html/warp.html
• https://www.meinbergglobal.com/english/sw/ntp.htm
• http://ntp.org/
• http://support.ntp.org/bin/view/Main/SoftwareDownloads
• http://www.ntpclient.com/
• http://www.analogx.com/contents/download/Network/ats/Freeware.htm

NTP time source lists:

• http://tf.nist.gov/tf-cgi/servers.cgi
• http://www.pool.ntp.org/en/
• http://tycho.usno.navy.mil/NTP/
• https://support.microsoft.com/en-us/kb/262680

If you are looking for a freeware local time sync daemon I recommend the one from here: https://www.meinbergglobal.com/english/sw/ntp.htm.  It is based on the open source client from ntp.org and is a very professional package.  It installs as a configurable Windows service.

I hope all this helps someone else.  I spent plenty of time locating it, perhaps it might save you some.

Server hang monitor

I recently had an issue where a Windows 2012 R2 VM would hand in the middle of the night.  No clue why.  Event logs were of no help.  Our best guess is that our internal vulnerability scan made it upset.  Either way this server was a major departmental file store and if it isn't up in the morning we get hammered with tickets.

I decided to create a script to manage this issue.  It uses 2 text files for input located in the samne folder as the script..  It requires the VMware PowerShell extensions.  It only works on virtual servers as written.  If the server is running but not reponding vSphere will reset it.  I have this running on a recurring 15 minute scheduled task.

Param(

$Debug = $False

)

#======================================================================================

#         File Name : Server-Alive.ps1

#   Original Author : Kenneth C. Mazie (kcmjr AT kcmjr.com)

#                   :

#    PowerShell Ver : 3

#                   :

#       Description : Used to reboot hung virtual servers automatically.

#                   :

#             Notes : Normal operation is with no command line options.  Use -debug $true

#                   : to enable debugging messages on console.  Requires VMware

#                   : PowerShell extensions.  Needs 2 config files located with the script

#                   : Servers.txt contains list of server(s) one per line to target,

#                   : Config.txt contains all other needed settings-

#                   :   user = vsphere username

#                   :   pass = vsphere password

#                   :   vCenter = vsphere server IP

#                   :   smtpServer = snmp server

#                   :   emailDomain = obvious

#                   :   emailfrom = what this process is called (minus email domain)

#                   :   emailTo = who should get the emails (minus email domain)

#                   :

#          Warnings : Yes Virginia, this WILL reboot your server...

#                   : 

#             Legal : Public Domain. Modify and redistribute freely. No rights reserved.

#                   : SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF

#                   : ANY KIND. USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.

#                   :

#           Credits : Code snippets and/or ideas came from many sources including but

#                   :   not limited to the following:

#                   :

#    Last Update by : Kenneth C. Mazie

#   Version History : v1.0 - 01-27-15 - Original

#    Change History : v1.1 - 00-00-00 -    

#                   :

#=======================================================================================

clear-host

$ErrorActionPreference = "stop" #silentlycontinue"

$Out = Get-PSSnapin | Where-Object {$_.Name -like "vmware.vimautomation.core"};if ($Out -eq $null) {Add-PSSnapin vmware.vimautomation.core}

$Services = ""

If ($Debug){$Global:Debug = $True}

If (Test-Path "$PSScriptRoot\Config.txt"){

  Get-Content "$PSScriptRoot\Config.txt" | foreach-object -begin {$Config=@{}} -process { $x = [regex]::split($_,'='); if(($x[0].CompareTo("") -ne 0) -and ($x[0].StartsWith("[") -ne $True)) { $Config.Add($x[0], $x[1]) } }

  $user = $Config.user

  $pass = $Config.pass

  $vCenter = $Config.vCenter

  $smtpServer = $Config.smtpServer

  $EmailDomain = $Config.EmailDomain

  $emailFrom = $Config.emailFrom

  $emailTo = $Config.emailTo

}Else{ 

  $user = Read-Host -Prompt "Enter the vSphere username."

  $pass = Read-Host -Prompt "Enter the vSphere password."

  $vCenter = Read-Host -Prompt "Enter the vSphere server."

  $smtpServer = Read-Host -Prompt "Enter the SMTP host."

  $EmailDomain = Read-Host -Prompt "Enter the Email domain."

  $emailFrom = 'ServerHealth@$EmailDomain'

  $emailTo = Read-Host -Prompt "Enter the email recipient without domain."

}

$subject = 'Server Health Status Check'

$EmailBody = ""

If (Test-Path "$PSScriptRoot\Servers.txt"){

  $Servers = Get-Content "$PSScriptRoot\Servers.txt"

}Else{

  $Server = Read-Host -Prompt "Enter the Server to test."

}

$Connect = Connect-VIServer -Server $vCenter -User $user -Password $pass

#----------------------------[ Functions ]--------------------------------------

Function SendEmail ($EmailBody){

If ($Global:Debug){Write-Host "`n   Sending email..."}

$email = New-Object System.Net.Mail.MailMessage

$email.From = $emailFrom

$email.IsBodyHtml =$True

$email.To.Add($emailTo)

$email.Subject = $subject

$email.Body = $EmailBody

$smtp = new-object Net.Mail.SmtpClient($smtpServer)

$smtp.Send($email)

}

Function Connection ($Server) {

try

   {$Connection = Test-Connection -ComputerName $Server -Count 1 -ErrorAction stop}

catch [System.Management.Automation.ActionPreferenceStopException]{

   try

      {throw $_.exception}

   catch [System.Net.NetworkInformation.PingException]

      {

      $ConnException = "Caught Ping Exception"

         Return $ServerExists = $False

      }

   catch

      {

      $ConnException = "General catch"

         Return $ServerExists = $False

      }

  }

  Return $ServerExists = $True

}

#--------------------------[ End of Functions ]---------------------------------

ForEach($Server in $Servers){

  $Server = $Server.ToUpper()

  If ($Global:Debug){Write-Host "Beginning connectivity test to ""$Server""..." -ForegroundColor Cyan}

  $EmailBody = $EmailBody + "--[ Heath check for server $Server ]--<br>"

  If($Global:Debug){Write-Host "`n--[ Check #1 - Generic Ping Test ]---------------------------------------------" -ForegroundColor Cyan}

  $Check1 = $True  

  $Result = ""

  if (Connection $Server){

    If ($Global:Debug){Write-Host "   ""$Server"" has responded to ping..." -ForegroundColor Green }

       $EmailBody = $EmailBody + "<font color=green><br>- $Server has successfully responded to the 1st ping check..."

  }Else{

    If ($Global:Debug){Write-Host "   ""$Server"" has failed 1st ping check..." -ForegroundColor Red }

       $EmailBody = $EmailBody + "<font color=red><br>- $Server has failed to respond to the 1st ping check..."

       $Check1 = $False

       If ($Global:Debug){sleep -Seconds 3}Else{sleep -Seconds 30}

    if (Connection $Server){

      If ($Global:Debug){Write-Host "   ""$Server"" has responded to ping..." -ForegroundColor Green }

         $EmailBody = $EmailBody + "<font color=green><br>- $Server has successfully responded to the 2nd ping check..."

    }Else{

      If ($Global:Debug){Write-Host "   ""$Server"" has failed 2nd ping check..." -ForegroundColor Red }

         $EmailBody = $EmailBody + "<font color=red><br>- $Server has failed to respond to the 2nd ping check..."

         $Check1 = $False

         If ($Global:Debug){sleep -Seconds 3}Else{sleep -Seconds 30}

      if (Connection $Server){

        If ($Global:Debug){Write-Host "   ""$Server"" has responded to ping..." -ForegroundColor Green }

              $EmailBody = $EmailBody + "<font color=green><br>- $Server has successfully responded to the 3rd ping check..."

      }Else{

        If ($Global:Debug){Write-Host "   ""$Server"" has failed 3rd ping check..." -ForegroundColor Red }

              $EmailBody = $EmailBody + "<font color=red><br>- $Server has failed to respond to the 3rd ping check..."

              $Check1 = $False

        If ($Global:Debug){sleep -Seconds 3}Else{sleep -Seconds 30}

      }

    }

  } 

  If($Global:Debug){Write-Host "`n--[ Check #2 - Service Inspection ]--------------------------------------------" -ForegroundColor Cyan}

  $Check2 = $True  

  $Result = ""

  $ErrorActionPreference = "silentlycontinue"

  $Result = Get-Service -ComputerName $Server | Where-Object {$_.DisplayName -eq "Server"}

  $ErrorActionPreference = "stop" #silentlycontinue"

  If ($Result.Status -eq "Running"){

    If ($Global:Debug){write-host "   Server service on $Server is running..." -ForegroundColor Green }

    $EmailBody = $EmailBody + "<font color=green><br>- Powershell detected that the SERVER service is running on $Server..."

  }Else{

    If ($Global:Debug){write-host "   Server service on $Server is unable to be detected..." -ForegroundColor Red }

    $EmailBody = $EmailBody + "<font color=red><br>- Powershell detected that the SERVER service is NOT running on $Server..."

    $Check2 = $False

  }

  If($Global:Debug){Write-Host "`n--[ Check #3 - VMWare Status ]-------------------------------------------------" -ForegroundColor Cyan}

  $Check3 = $True  

  $Result = ""

  $Result = Get-VMGuest -VM $Server 

  If ($Result.State -eq "Running"){

    If ($Global:Debug){write-host "   $Server is running..." -ForegroundColor Green }

    $EmailBody = $EmailBody + "<font color=green><br>- VMware detected that $Server is running..."

  }Else{

    If ($Global:Debug){write-host "   $Server is shut down..." -ForegroundColor Red }

    $EmailBody = $EmailBody + "<font color=red><br>- VMware detected that $Server is not running..."

    $Check3 = $False

  }

  If($Global:Debug){Write-Host "`n--[ Final Determination ]-----------------------------------------------------" -ForegroundColor Cyan}

  $Now = "{0:HH:mm}" -f ([DateTime]::Now)

  If (($Check1 -eq $False) -and ($Check2 -eq $False) -and ($Check3 -eq $True)){

    #--[ Server is hung... restart if within the window ]--

    if (((get-date).hour -le 6) -or ((get-date).hour -ge 18)){

      $Restart = ReStart-VM -VM $Server -Confirm:$false

      If ($Global:Debug){Write-Host "   Server $Server has failed some health checks and is being forceably restarted. " -ForegroundColor Red }

      $EmailBody = $EmailBody + "<font color=red><br>- Server $Server has failed some health checks and is being forceably restarted. "

      SendEmail $EmailBody

    }else{

      If ($Global:Debug){Write-Host "   Server $Server has failed some health checks and should be forceably restarted but cannot due to being outside the time window. " -ForegroundColor Red }

      $EmailBody = $EmailBody + "<font color=darkcyan><br>- Server $Server has failed some health checks and should be forceably restarted but cannot due to being outside the safe reboot time window. "

      SendEmail $EmailBody

    }

  }ElseIf (($Check1 -eq $False) -and ($Check2 -eq $False) -and ($Check3 -eq $False)){

    #--[ Server may be hung but could be rebooting or intentially shut down.  No auto restart, email only ]--

    If ($Global:Debug){Write-Host "   Server $Server has failed some health checks and should be checked.  It is NOT being forceably restarted. " -ForegroundColor Yellow  }

    $EmailBody = $EmailBody + "<font color=red><br>- Server $Server has failed some health checks and should be checked.  It is NOT being forceably restarted. "

    SendEmail $EmailBody

  }Else{

    If ($Global:Debug){Write-Host "   Server $Server has passed enough connection validation to assume it is onlne.  No action taken." -ForegroundColor Green}

    #$EmailBody = $EmailBody + "<font color=green><br>- Server $Server has passed enough connection validation to assume it is onlne.  No action taken. "

    #SendEmail $EmailBody

  }

}

DisConnect-VIServer -Server $vCenter -Confirm:$false -Force:$true