Hello,
I am developing a script for an upgrade of NSX VIBs across our environment. The problem is that after an upgrade of NSX manager, it tries to update the VIBs on the hosts (once they are put in maintenance mode) but most of the time it just fails and the VIBs have to be force uninstalled and the host rebooted. After the host reboot, NSX manager is able to install the new VIB successfully.
So the end-goal I'm trying to achieve is this:
1. Put a host in the cluster into maintenance mode.
2. Wait for NSX manager to try to uninstall the VIBs (otherwise race condition occurs) and install the new VIBs
3. Check if the old VIBs still exist after x mins. If it doesn't, skip to 8.
4. If it does, force-uninstall the VIBs and reboot the host.
5. Allow time for host to come back online and check that it's back up.
7. Wait for NSX manager to install the new VIBs.
8. Check if the new VIBs are installed.
9. Repeat for the next host in the list.
Nice-to-have feature: Perform this on a chunk of hosts simultaneously.
So far, I have the below script. When I run it, I get the catch statement "The vib has already been removed by NSX manager" even though the VIB is found. The script doesn't remove the VIB but goes directly to rebooting the host.
Can anyone suggest what I'm doing wrong and any improvements to the script?
Thanks in advance.
Script:
$server = Read-Host "Enter VC IP"
$creds = Get-Credential
Connect-VIServer -Server $server -Credential $creds
$esxiName = Read-Host ("enter esxi host name")
$esxi = Get-VMHost -Name $esxiName
#$ErrorActionPreference = "Stop"
$Endloop=$false
######################################################## checking if esxi is in mainternance mode, if not setting in maintenance mode
Write-Host "Checking if ESXi in maintenance mode" -ForegroundColor Yellow
Write-Host ("ESxi is in " + $esxi.ConnectionState + " mode") -ForegroundColor Red
if ($esxi.ConnectionState -ne "Maintenance"){
Write-Host ("Setting " +$esxiName +" to maintenance mode") -foregroundcolor Yellow
get-VMHost $esxiName | set-VMHost -state maintenance -confirm:$false -runasync |out-null
sleep 10
$count = 0
while ($true) {
if ( (get-VMHost $esxiName).ConnectionState -eq "Maintenance") {
Write-Host (get-date) ": $esxiName in maintenance mode" -foregroundcolor Green
break;
}
else {
Write-Host (get-date) + ": waiting for $esxiName to go into maintenance..." -foregroundcolor Yellow
sleep 120
$count++
}
if ($count -gt 15) {
Write-Host (get-date) ": Waited too long for maintenance.. quiting!" -foregroundcolor Red
$Endloop=$true
break;
}
}
}
################################################## if esxi is in maintenance mode, remove Vibs
if (((get-VMHost $esxiName).ConnectionState -eq "Maintenance") -or (!$Endloop)) {
Write-Host ("esxi is in " + (get-VMHost $esxiName).ConnectionState + " mode") -ForegroundColor Green
$esxcli = Get-EsxCli -VMHost $esxi -V2
$vibs = $esxcli.software.vib.list.Invoke() | where{$_.Name -match "esx-vxlan" -or $_.Name -match "esx-vsip" }
$esxcliRemoveVibArgs = $esxcli.software.vib.remove.CreateArgs()
try {
if ($vibs -ne $null){
foreach ($vib in $vibs){
Write-Host ("Found Vib: " + $vib.Name)
$esxcliRemoveVibArgs.vibname = $vib.Name
$esxcliRemoveVibArgs.noliveinstall = $true
Write-Host ("Removing vib:" + $vib.Name) -ForegroundColor Green
esxcli.software.vib.remove.Invoke($esxcliRemoveVibArgs)
}
}
else {
Write-Host ("No vibs found with name esx-vxlan or esx-vsip")
}
}
catch {
Write-Host ("The vib has already been removed by NSX manager")
}
}
##################################################### if esxi is in maintenance mode, check if esx-nsxv vib installed, if not installed, Reboot ESXi . After reboot again check for installed Vib and display messgae accordingly
if (((get-VMHost $esxiName).ConnectionState -eq "Maintenance") -or (!$Endloop)) {
$esxcli = Get-EsxCli -VMHost $esxi -V2
$installedvib = $esxcli.software.vib.list.Invoke() | where{$_.Name -eq "esx-nsxv"}
$count = 0
while ($true) {
if (((Get-EsxCli -VMHost $esxi -V2).software.vib.list.Invoke() | where{$_.Name -eq "esx-nsxv"}) -ne $null)
{
Write-Host ("esx-nsxv vib found on esxi, no need for reboot " +$esxiName) -ForegroundColor Green
break;
}
else {
Write-Host (get-date) ":Rebooting $esxiName " -foregroundcolor Yellow
if ( (get-VMHost $esxiName).ConnectionState -eq "Maintenance") {
get-VMHost $esxiName | restart-VMHost -confirm:$false -force | out-null
sleep 600
$count = 0
while ($true) {
if ((get-VMHost $esxiName).ConnectionState -eq "Maintenance") {
Write-Host (get-date) ": $esxiName is up and in maintenance mode" -foregroundcolor Green
Write-Host (Get-Date) ": checking again if the required esx-nsxv vib is installed" -ForegroundColor Yellow
if (((Get-EsxCli -VMHost $esxi -V2).software.vib.list.Invoke() | where{$_.Name -eq "esx-nsxv"}) -ne $null)
{
Write-Host ("esx-nsxv vib found on esxi" +$esxiName) -ForegroundColor Green
break;
}
else {
Write-Host ("esx-nsxv vib not found even after reboot" +$esxiName) -ForegroundColor Green
$Endloop=$true
break;
}
break ;
}
else {
Write-Host (get-date) ": waiting for $esxiName to be online..." -foregroundcolor Yellow
sleep 300
$count++
}
if ($count -gt 6) {
Write-Host (get-date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red
$Endloop=$true
break;
}
}
}
else {
Write-Host (get-date) ": $esxiName is not in maintenance, cannot reboot!" -foregroundcolor Red
$Endloop=$true
break;
}
}
}
}
else {
Write-Host (get-date) ": $esxiName is not in maintenance, cannot check if esx-nsxv is installed!" -foregroundcolor Red
$Endloop=$true
break;
}
################################################## if esxi is in maintenance mode, exit out of maintenance mode
if (((get-VMHost $esxiName).ConnectionState -eq "Maintenance") -or (!$Endloop)) {
Write-Host "Setting $esxiName back online" -foregroundcolor Yellow
get-VMHost $esxiName | set-VMHost -state connected -confirm:$false | out-null
sleep 20
$count = 0
while ($true) {
if ((get-VMHost $esxiName).ConnectionState -eq "Connected") {
Write-Host (get-date) ": $esxiName is up and connected" -foregroundcolor Green
break;
}
else {
Write-Host (get-date) ": waiting for $esxiName to be online..." -foregroundcolor Yellow
sleep 300
$count++
}
if ($count -gt 6) {
Write-Host (get-date) ": Waited too long for host to be up. quiting!" -foregroundcolor Red
$Endloop=$true
break;
}
}
}
#################################################
if ($Endloop) {break;}
Script output:
enter esxi host name: $hostname
Checking if ESXi in maintenance mode
ESxi is in Connected mode
Setting $hostname to maintenance mode
esxi is in Maintenance mode
Found Vib: esx-vsip
Removing vib:esx-vsip
The vib has already been removed by NSX manager
4/30/20 2:11:25 PM :Rebooting $hostname
4/30/20 2:31:31 PM : checking again if the required esx-nsxv vib is installed
esx-nsxv vib found on esxi $hostname
esx-nsxv vib found on esxi, no need for reboot $hostname
Setting $hostname back online