Solved

Programatically retrieve VM Table information


Badge +1
I'm trying to figure out a way via PowerShell, or even the main Prism interface to export all of the table information from the VM > Table space within Prism. Like this.



Doing an entire .CSV export of this would be just fine, but it appears to limit to 1000 rows. I would expect something along the lines of grabbing all the VM's with Get-NTNXVM and piping that into Get-NTNXVMStat, but this cmdlet appears to need several metrics specified that I just can't dig into properly. I receive 500 errors any time I try to retrieve information, and there's no Get-Help / example entries for the cmdlet.

Is there an easy way I'm overlooking to retrieve exactly that formatted table from prism? If not, any canned PS scripts that can do it? The individual I/O statistics are very helpful here, as a whole this is an excellent, granular snapshot of the environment.

Thanks!
icon

Best answer by Chandru 10 October 2017, 13:32

If you are looking for a powershell script you can use the following

Connect-NTNXCluster -Server -UserName admin -AcceptInvalidSSLCerts$static_stats=@("vmname","numVCpus","memoryCapacityInBytes","diskCapacityInBytes")$dyn_stats=@("hypervisor_cpu_usage_ppm", "hypervisor_memory_usage_ppm","controller_avg_io_latency_usecs", `"controller_num_read_iops", "controller_num_write_iops", "controller_io_bandwidth_kBps")$vm_col=@()Get-NtnxVM | %{ $vmstat=New-Object System.Management.Automation.PSObject $static_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $dyn_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $vmstat.("vmname")=$_.vmname $vmstat.("numvcpus")=$_.numvcpus $vmstat.("memorycapacityinbytes")=$_.memorycapacityinbytes $vmstat.("DiskCapacityInbytes")=$_.diskCapacityInBytes Get-NtnxVMstat -VmId $_.vmid -Metrics $($dyn_stats -join ',') | %{ $vmstat.($_.metric)=$_.values[0] } $vm_col+=$vmstat}$vm_col | select vmname,numvcpus,@{Name="Memory (GB)";Expression={$_.memoryCapacityInBytes/1GB}}, `@{Name="Disk Size (GB)";Expression={[math]::Round($_.diskCapacityInBytes/1GB)}},@{Name="CPU Usg %";Expression={$_.hypervisor_cpu_usage_ppm/10000}},`@{Name="Memory Usg %";Expression={$_.hypervisor_memory_usage_ppm/10000}},@{Name="read iops";Expression={$_.controller_num_read_iops}},`@{Name="Write IOPS";Expression={$_.controller_num_write_iops}},@{Name="Bandwidth (kbps)";Expression={$_.controller_io_bandwidth_kBps}},`@{Name="Avg Latency (ms)";Expression={$_.controller_avg_io_latency_usecs/1000}} | Export-csv -Path result.csv -NoTypeInformationPlease replace the with CVM ip address. This script will output a csv file "result.csv" with the required stats

View original

10 replies

Userlevel 2
Badge +12

I cobbled together a python script to list all the VM's and convert the json output to csv

It may help...

edit; it removed all the indentation 😞

#!/usr/bin/env python
import csvimport pprintimport jsonimport osimport randomimport sysimport tracebackimport getpassimport timeimport requestsfrom requests.packages.urllib3.exceptions import InsecureRequestWarningrequests.packages.urllib3.disable_warnings(InsecureRequestWarning)
t = (time.strftime("%H%M%S%d%m%Y"))clus = raw_input("Enter Cluster Name: ")user = raw_input("Enter your Prism username: ")p = getpass.getpass("Enter you Prism Password: ")rv = 'v2.0'

#This block initializes the parameters for the request.class TestRestApi():def __init__(self):
#Initializes the options and the logfile from GFLAGS.self.serverIpAddress = clusself.username = userself.password = p# Base URL at which REST services are hosted in Prism Gateway.BASE_URL = 'https://%s:9440/PrismGateway/services/rest/'+rvself.base_url = BASE_URL % self.serverIpAddressself.session = self.get_server_session(self.username, self.password)
def get_server_session(self, username, password):
#Creating REST client session for server connection, after globally setting#Authorization, content type, and character set for the session.session = requests.Session()session.auth = (username, password)session.verify = Falsesession.headers.update({'Content-Type': 'application/json; charset=utf-8'})return session

def getVMs(self):
vmsVMURL = self.base_url + "/vms/?include_vm_disk_config=false&include_vm_nic_config=false"print "List of VMs in cluster %s" % self.serverIpAddressserverResponse = self.session.get(vmsVMURL)print "Response code: %s" % serverResponse.status_codereturn json.loads(serverResponse.text)

def to_string(s):try:return str(s)except:#Change the encoding type if neededreturn s.encode('utf-8')

def reduce_item(key, value):global reduced_item
#Reduction Condition 1if type(value) is list:i=0for sub_item in value:reduce_item(key+'_'+to_string(i), sub_item)i=i+1
#Reduction Condition 2elif type(value) is dict:sub_keys = value.keys()for sub_key in sub_keys:reduce_item(key+'_'+to_string(sub_key), value[sub_key])
#Base Conditionelse:reduced_item[to_string(key)] = to_string(value)
if __name__ == "__main__":
csv_file_path = t+'_'+clus+'_vms.csv'
try:testRestApi = TestRestApi()vms = testRestApi.getVMs()raw_data = vmsdata_to_be_processed = vms['entities']except Exception as ex:print exsys.exit(1)data_to_be_processed = vms
processed_data = []header = []for item in data_to_be_processed:reduced_item = {}reduce_item('entities', item)
header += reduced_item.keys()
processed_data.append(reduced_item)
header = list(set(header))header.sort()
with open(csv_file_path, 'wb+') as f:writer = csv.DictWriter(f, header, quoting=csv.QUOTE_ALL)writer.writeheader()for row in processed_data:writer.writerow(row)
print "Just completed writing csv file with %d columns" % len(header)
Userlevel 2
Badge +12
#!/usr/bin/env pythonimport csvimport pprintimport jsonimport osimport randomimport sysimport tracebackimport getpassimport timeimport requestsfrom requests.packages.urllib3.exceptions import InsecureRequestWarningrequests.packages.urllib3.disable_warnings(InsecureRequestWarning)t = (time.strftime("%H%M%S%d%m%Y"))clus = raw_input("Enter Cluster Name: ")user = raw_input("Enter your Prism username: ")p = getpass.getpass("Enter you Prism Password: ")rv = 'v2.0'#This block initializes the parameters for the request.class TestRestApi(): def __init__(self): #Initializes the options and the logfile from GFLAGS. self.serverIpAddress = clus self.username = user self.password = p # Base URL at which REST services are hosted in Prism Gateway. BASE_URL = 'https://%s:9440/PrismGateway/services/rest/'+rv self.base_url = BASE_URL % self.serverIpAddress self.session = self.get_server_session(self.username, self.password) def get_server_session(self, username, password): #Creating REST client session for server connection, after globally setting #Authorization, content type, and character set for the session. session = requests.Session() session.auth = (username, password) session.verify = False session.headers.update( {'Content-Type': 'application/json; charset=utf-8'}) return session def getVMs(self): vmsVMURL = self.base_url + "/vms/?include_vm_disk_config=false&include_vm_nic_config=false" print "List of VMs in cluster %s" % self.serverIpAddress serverResponse = self.session.get(vmsVMURL) print "Response code: %s" % serverResponse.status_code return json.loads(serverResponse.text)def to_string(s): try: return str(s) except: #Change the encoding type if needed return s.encode('utf-8')def reduce_item(key, value): global reduced_item #Reduction Condition 1 if type(value) is list: i=0 for sub_item in value: reduce_item(key+'_'+to_string(i), sub_item) i=i+1 #Reduction Condition 2 elif type(value) is dict: sub_keys = value.keys() for sub_key in sub_keys: reduce_item(key+'_'+to_string(sub_key), value[sub_key]) #Base Condition else: reduced_item[to_string(key)] = to_string(value)if __name__ == "__main__": csv_file_path = t+'_'+clus+'_vms.csv' try: testRestApi = TestRestApi() vms = testRestApi.getVMs() raw_data = vms data_to_be_processed = vms['entities'] except Exception as ex: print ex sys.exit(1) data_to_be_processed = vms processed_data = [] header = [] for item in data_to_be_processed: reduced_item = {} reduce_item('entities', item) header += reduced_item.keys() processed_data.append(reduced_item) header = list(set(header)) header.sort() with open(csv_file_path, 'wb+') as f: writer = csv.DictWriter(f, header, quoting=csv.QUOTE_ALL) writer.writeheader() for row in processed_data: writer.writerow(row) print "Just completed writing csv file with %d columns" % len(header)I cobbled together this python script, it may help

Needs changing to get IO info though
Userlevel 2
Badge +12
If you are looking for a powershell script you can use the following

Connect-NTNXCluster -Server -UserName admin -AcceptInvalidSSLCerts$static_stats=@("vmname","numVCpus","memoryCapacityInBytes","diskCapacityInBytes")$dyn_stats=@("hypervisor_cpu_usage_ppm", "hypervisor_memory_usage_ppm","controller_avg_io_latency_usecs", `"controller_num_read_iops", "controller_num_write_iops", "controller_io_bandwidth_kBps")$vm_col=@()Get-NtnxVM | %{ $vmstat=New-Object System.Management.Automation.PSObject $static_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $dyn_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $vmstat.("vmname")=$_.vmname $vmstat.("numvcpus")=$_.numvcpus $vmstat.("memorycapacityinbytes")=$_.memorycapacityinbytes $vmstat.("DiskCapacityInbytes")=$_.diskCapacityInBytes Get-NtnxVMstat -VmId $_.vmid -Metrics $($dyn_stats -join ',') | %{ $vmstat.($_.metric)=$_.values[0] } $vm_col+=$vmstat}$vm_col | select vmname,numvcpus,@{Name="Memory (GB)";Expression={$_.memoryCapacityInBytes/1GB}}, `@{Name="Disk Size (GB)";Expression={[math]::Round($_.diskCapacityInBytes/1GB)}},@{Name="CPU Usg %";Expression={$_.hypervisor_cpu_usage_ppm/10000}},`@{Name="Memory Usg %";Expression={$_.hypervisor_memory_usage_ppm/10000}},@{Name="read iops";Expression={$_.controller_num_read_iops}},`@{Name="Write IOPS";Expression={$_.controller_num_write_iops}},@{Name="Bandwidth (kbps)";Expression={$_.controller_io_bandwidth_kBps}},`@{Name="Avg Latency (ms)";Expression={$_.controller_avg_io_latency_usecs/1000}} | Export-csv -Path result.csv -NoTypeInformationPlease replace the with CVM ip address. This script will output a csv file "result.csv" with the required stats
Badge +1
Thank you! Pretty sure this is what I need, I appreciate it. I'm currently trying to find a way to filter it down right now. This cluster has about 2500 VM's in it, with a 4.5:1 CPU ratio and heavily utilization, I think this is causing the Get-NTNXVM cmdlet to time out. I believe I can accomplish this by importing a .csv with chunks of names of VM's, and then combining the .CSV files later.

Thanks again : )




Userlevel 2
Badge +12
mhutchison89 Since you have a large number of vm's i have changed the code to use Object collections instead of normal arrays which will be more memory efficient. The changes are in only two lines, please use the below script,

Connect-NTNXCluster -Server 10.63.4.113 -UserName admin -AcceptInvalidSSLCerts$static_stats=@("vmname","numVCpus","memoryCapacityInBytes","diskCapacityInBytes")$dyn_stats=@("hypervisor_cpu_usage_ppm", "hypervisor_memory_usage_ppm","controller_avg_io_latency_usecs", `"controller_num_read_iops", "controller_num_write_iops", "controller_io_bandwidth_kBps")$vm_col= New-Object System.Collections.ArraylistGet-NtnxVM | %{ $vmstat=New-Object System.Management.Automation.PSObject $static_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $dyn_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $vmstat.("vmname")=$_.vmname $vmstat.("numvcpus")=$_.numvcpus $vmstat.("memorycapacityinbytes")=$_.memorycapacityinbytes $vmstat.("DiskCapacityInbytes")=$_.diskCapacityInBytes Get-NtnxVMstat -VmId $_.vmid -Metrics $($dyn_stats -join ',') | %{ $vmstat.($_.metric)=$_.values[0] } $vm_col.add($vmstat) | Out-Null}$vm_col | select vmname,numvcpus,@{Name="Memory (GB)";Expression={$_.memoryCapacityInBytes/1GB}}, `@{Name="Disk Size (GB)";Expression={[math]::Round($_.diskCapacityInBytes/1GB)}},@{Name="CPU Usg %";Expression={$_.hypervisor_cpu_usage_ppm/10000}},`@{Name="Memory Usg %";Expression={$_.hypervisor_memory_usage_ppm/10000}},@{Name="read iops";Expression={$_.controller_num_read_iops}},`@{Name="Write IOPS";Expression={$_.controller_num_write_iops}},@{Name="Bandwidth (kbps)";Expression={$_.controller_io_bandwidth_kBps}},`@{Name="Avg Latency (ms)";Expression={$_.controller_avg_io_latency_usecs/1000}} | Export-csv -Path test.csv -NoTypeInformation
Badge +1
Thanks Chandru, I have a question..

The updated PS code using objects works great if I specify a single VM, like this, where I'm using -searchString to specify a single VM. The .CSV will export perfectly for that VM. As soon as I remove that search criteria the GET-NTNXVM cmdlet will time out within about 15-30 seconds.
Get-NtnxVM -SearchString 'vmname' | forEach{ $vmstat=New-Object System.Management.Automation.PSObject $static_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $dyn_stats | %{Add-Member -InputObject $vmstat -MemberType NoteProperty -Name $_ -Value ""} $vmstat.("vmname")=$_.vmname

I have tried substituting a variable with about 25 VM names in it, in the form of..

Get-NTNXVM -SearchString $vm.Name | % (etc...), but it appears to only retrieve the first name in the array, I assume because -searchString limits it to one item.

Is there another way to specify a smaller collection of VM's? I imagine querying the entire inventory is taxing that cmdlet. We do have a lot of VM's in a single cluster.


Thanks! I'm thinking I may need to open a case to figure out why code is timing out though?


Userlevel 2
Badge +12
mhutchison89 Does it work when you use regular arrays instead of the object colletions? You can try to limit the count of vm returned by Get-NTNXvm using the -Count parameter and check if it is getting output.


I tested it on a 120 vm test cluster didn't receive any timeouts , you can limit to the lesser number and see if it works.

Badge +1
Yep, the script runs if I limit the count to < 300. If I could feed the script a list of VM names I could probably get around that limitation. I've tried Get-NTNXVM -count 300 | Where {$_.VMName -like $arrayofnames} but the command times out with that as well, I assume because it's having to crawl the entire inventory again.
Userlevel 2
Badge +12
mhutchison89 what happens when you just run Get-NTNXVM | select vmname does it timeout too? If that times out too then we may need to use rest api to query the vms and get a list of them.
Badge +1
Yeah, even if I just filter down to the single VMname property, the cmdlet still times out. I've also tried Getting the list of VM names via the API, using the Invoke-RestMethod cmdlet. Like this, but I receive a time out there as well (With or without the timeout flag)

$vms = Invoke-RestMethod -Uri https://CVMVIP:9440/api/nutanix/v2.0/vms/? -Headers $Header -TimeoutSec 600


Thanks!

Reply