Hi Everyone!
I have a script which is used to collect usage and capacity stats from vCenters and calculate N+1 capacity, availability and consolidation ratios etc.
This works perfectly across all except 2 of the VCs which it collects from. The issue seems to be with the data returned by the Get-Stat cmdlet, specifically for the cpu.usagemhz.average stat. mem.consumed.average appears to be accurate.
The VCs which are at fault are running VCSA 6.0 U3b (build 5318203), however, interestingly, 2 of the VCs which report correctly are also running this exact build as well. The only differences are that:
- The incorrect ones were deployed on this build, whereas the working ones were deployed at U1 and later upgraded
- The incorrect ones have statistics level 3 configured for the top 3 options (5min/1day, 30mins/1week, 2hours/1month) and level one for the last (1day/1year) whereas the working ones have level 1 configured on all options.
It seems as though the values returned are about double what is shown on the performance charts.
I have tried running this from various versions of Powershell / PowerCLI and get the same result so I suspect the data returned by the vCenter is simply wrong.
This is a snippet of the script which shows the code I am using to collect the stats and then take the Max and Avg from the values:
$Clusters = @( Get-Cluster | Where {$_.Name -like "$ClusterFilter"} | Sort Name )
$ResPools = Get-View -ViewType ResourcePool
$CpuStats1d = Get-Stat -Entity $Clusters -Stat cpu.usagemhz.average -Start $StartTime.AddDays(-1) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
$MemStats1d = Get-Stat -Entity $Clusters -Stat mem.consumed.average -Start $StartTime.AddDays(-1) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
$CpuStats1w = Get-Stat -Entity $Clusters -Stat cpu.usagemhz.average -Start $StartTime.AddDays(-7) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
$MemStats1w = Get-Stat -Entity $Clusters -Stat mem.consumed.average -Start $StartTime.AddDays(-7) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
$CpuStats1m = Get-Stat -Entity $Clusters -Stat cpu.usagemhz.average -Start $StartTime.AddDays(-30) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
$MemStats1m = Get-Stat -Entity $Clusters -Stat mem.consumed.average -Start $StartTime.AddDays(-30) -Finish $StartTime | Select Entity, MetricId, Value, Unit, Timestamp, Description, IntervalSecs
New-Variable -Name ClusterAudit -Value @() -Scope Script -Force
foreach ($Cluster in $Clusters) {
$ResPool = $ResPools | Where {$_.MoRef -eq $Cluster.ExtensionData.ResourcePool}
$ClstCpuStats1d = (($CpuStats1d | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClstMemStats1d = (($MemStats1d | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClstCpuStats1w = (($CpuStats1w | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClstMemStats1w = (($MemStats1w | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClstCpuStats1m = (($CpuStats1m | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClstMemStats1m = (($MemStats1m | Where {$_.Entity.Name -eq $Cluster.Name}).Value | Measure-Object -Average -Maximum )
$ClusterDetails = New-Object -TypeName PSObject
$ClusterDetails | Add-Member -MemberType NoteProperty -Name Cluster -Value $Cluster.Name
$ClusterDetails | Add-Member -MemberType NoteProperty -Name NumberOfHosts -Value $Cluster.ExtensionData.Summary.NumHosts
$ClusterDetails | Add-Member -MemberType NoteProperty -Name EffeciveNumberOfHosts -Value $Cluster.ExtensionData.Summary.NumEffectiveHosts
$ClusterDetails | Add-Member -MemberType NoteProperty -Name TotalCpuCapacityGHz -Value ($Cluster.ExtensionData.Summary.TotalCpu / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name EffectiveCpuCapacityGHz -Value ($Cluster.ExtensionData.Summary.EffectiveCpu / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name TotalMemoryCapacityGB -Value ($Cluster.ExtensionData.Summary.TotalMemory / 1GB)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name EffectiveMemoryCapacityGB -Value ($Cluster.ExtensionData.Summary.EffectiveMemory / 1KB)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuReservationCapacityGHz -Value ($ResPool.Runtime.Cpu.MaxUsage / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuReservationUsedGHz -Value ($ResPool.Runtime.Cpu.ReservationUsed / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuResvPwrdOffVmsGHz -Value ($Cluster.ExtensionData.Summary.UsageSummary.PoweredOffCpuReservationMhz / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemoryReservationCapacityGB -Value ($ResPool.Runtime.Memory.MaxUsage / 1GB)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemoryReservationUsedGB -Value ($ResPool.Runtime.Memory.ReservationUsed / 1GB)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemoryResvPwrdOffVmsGB -Value ($Cluster.ExtensionData.Summary.UsageSummary.PoweredOffMemReservationMB / 1GB)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageAvg1dGHz -Value ($ClstCpuStats1d.Average / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageMax1dGHz -Value ($ClstCpuStats1d.Maximum / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageAvg1wGHz -Value ($ClstCpuStats1w.Average / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageMax1wGHz -Value ($ClstCpuStats1w.Maximum / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageAvg1mGHz -Value ($ClstCpuStats1m.Average / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name CpuUsageMax1mGHz -Value ($ClstCpuStats1m.Maximum / 1000)
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageAvg1dGB -Value ($ClstMemStats1d.Average / 1MB )
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageMax1dGB -Value ($ClstMemStats1d.Maximum / 1MB )
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageAvg1wGB -Value ($ClstMemStats1w.Average / 1MB )
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageMax1wGB -Value ($ClstMemStats1w.Maximum / 1MB )
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageAvg1mGB -Value ($ClstMemStats1m.Average / 1MB )
$ClusterDetails | Add-Member -MemberType NoteProperty -Name MemUsageMax1mGB -Value ($ClstMemStats1m.Maximum / 1MB )
$Script:ClusterAudit += $ClusterDetails
}
Usually, the only retained info is the Avg and Max value from the data, however, the below is a sample of what is returned:
Entity MetricId Value Unit Timestamp Description IntervalSecs
------ -------- ----- ---- --------- ----------- ------------
Cluster1 cpu.usagemhz.average 493505 MHz 08/04/2020 13:30:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 490822 MHz 08/04/2020 13:25:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 503932 MHz 08/04/2020 13:20:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 502446 MHz 08/04/2020 13:15:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 500512 MHz 08/04/2020 13:10:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 517569 MHz 08/04/2020 13:05:00 CPU usage in megahertz during the interval 300
Cluster1 cpu.usagemhz.average 479613 MHz 08/04/2020 13:00:00 CPU usage in megahertz during the interval 300
etc.
The above is just the first first few results. In total 288 results are returned which accounts for the 5 min interval for 24 hours as expected.
The problem is, these do not match the performance chart stats. See below image for the same cluster at the same time period, 13:20 reports 503,932 MHz from Get-Stat and performance charts shows it at 253,483 MHz, which I would say is accurate.
Performance charts also shows the Max for the last day to be 277,762 MHz whereas Get-Stat Max is 552,252 MHz.
I have tried to attach an image below but it looks very small so I'm not sure if it will be helpful.
Has anybody else come across a similar issue to this before or have any suggestions as to what might be the issue?
I'm not likely to be available for a few days but I will try to provide any additional info you might need as soon as possible.
Thanks in advance for any help or advice.
Rob.
Message was edited by: Rob Hayward Edited due to code snippet formatting changing on posting.