How To enable nested virtualization on AHV?


Badge +1
How To enable nested virtualization on AHV?

13 replies

Userlevel 1
Badge +9
Is this AHV on CE or commercial Nutanix?

Badge +1
Its AHV
Badge
I'm curious about this as well. Is there any way to enable nested virtualization (make hardware assisted virtualization available to the guest os?)
Userlevel 7
Badge +30
Yes, you can enable cpu_passthrough flag in ACLI vm.update

https://portal.nutanix.com/#/page/docs/details?targetId=AMF-Guide-AOS-v50:acl-acli-vm-auto-r.html

That will expose typical nested virtualization support.

Note: Enabling nested virtualization precludes that specific vm from many features, such as ADS and live migration. It also precludes the cluster from doing any sort of rolling maintenance that would require live migration.

i.e if you had to upgrade the hypervisor or BIOS, you'd have to shut down the VM with this flag enabled.

This is because KVM/QEMU based systems, just at general technology level, do not support live migration for nested virtualization enabled VMs. Google apparently just patched this for GCP but that code hasnt made it upstream yet.
Badge +3
I am trying to enable an nested ESXi 6.5 instance running on AHV.
I have neabled the setting which allows the install to process however am now presented with a Network Driver Issue. "No Networks Adapaters were deteced"
Any advice?
Badge +3
Incase it helps I got this working by doing the below:-

1. Created New VM meeting min requirements for ESXi 6.5Using ACLI changed the following2. Add compatible Nic to stop NIC error:-vm.nic_create VMNAME network="NETWORKNAME" model=e10003. Enable CPU Passthrough stop CPU error:-vm.update VMNAME cpu_passthrough=true4. Add and ATA / IDE disk so a boot disk is discoverable at installvm.disk_create VMNAME container=default-container-XXXX create_size=100G bus=ide index=1
Badge +2
Similarly I got it to install however, after sometime, it purple screened. On subsequent reboots, it vails while loading the Balloon VMCI piece. It ran successfuly for quite awhile though. How is yours working?

Thanks
Badge
Hi

I enabled cpu_passtrough flag for my VM, but when trying to configure the Hyper-V on my Windows 2016 Guest VM, I am now getting a different error stating that "virtualization support is not enabled in the BIOS".

I saw there is the argument "nested_hv", so I gave it a try, but I receives the error "Unknow keyword argument: nested_hv"

Any idea of what I'm doing wrong?
Userlevel 7
Badge +30
Hyper-V on KVM based platforms (like AHV) is still maturing upstream. To be clear, nested virtualization in AHV currently is only targeted at KVM based guests. Hyper-V and ESXi are "wild wild west" at best.

As that support matures upstream, and we consume those upstream updates, we'll get better and better here, but for now, I'd suspect this won't work well for now.

Out of posterity, the universal recommendation here is to make sure you're on the absolute latest AOS, with the absolute latest AHV. This will be true when AOS 5.5 comes out (shortly) as we've done quite a massive update on the AHV side, so you may find support here is a bit better. Can't promise as I haven't tested it myself, but its worth checking out.

If you're still having an issue there, feel free to submit a support ticket so we can make sure we're tracking this properly.
Badge +1
Hi Jon,

Do you know whether or not:
a) The API exposes cpu_passthrough
b) The cpu_passthrough setting will be configurable on the VM configuration page via the Prism UI anytime soon
c) The cpu_passthrough setting (if enabled on a VM) will also be applied to a clone of the given VM
d) Support for nested virtualization for Hyper-V is any closer towards coming out of the "wild wild west"

We are using the APIs extensively in our organization but can't find reference to the cpu_passthrough setting in the API.
Userlevel 7
Badge +30

testworksau wrote:Hi Jon,

Do you know whether or not:
a) The API exposes cpu_passthrough
b) The cpu_passthrough setting will be configurable on the VM configuration page via the Prism UI anytime soon
c) The cpu_passthrough setting (if enabled on a VM) will also be applied to a clone of the given VM
d) Support for nested virtualization for Hyper-V is any closer towards coming out of the "wild wild west"

We are using the APIs extensively in our organization but can't find reference to the cpu_passthrough setting in the API.


Hey thanks for reaching out. I'm curious, whats your use case, specifically?

Answers as of today:
A) no
😎 no
C) good question, I don't recall offhand. Should be an easy test, but I'm on a plane right now, don't have good connectivity back to lab.
D) I can't say without saying something forward looking in a public forum. We're working on making overall nested support better. Even then, there is still one key patch missing upstream, more below.

Basically, we won't "GA" full nested support until live migration, which is not committed upstream yet, though that work is in progress. Otherwise, nested VM's become "special" VMs, where you can't do live migration, hypervisor patching, and other lifecycle operations that require host reboots. That means nested VM's would require manual and mandatory downtime during this operations. We dont think thats a good customer experience.
Trying to enable Hypervisor support on a VM in Nutanix 5.5 AHV.

Version 5.0 had an ACLI command that I assume can be applied to a VM as follows:
acli vm.update my_vm_name nested_hv="true"

This is documented here for 5.0:
https://portal.nutanix.com/#/page/docs/details?targetId=AMF-Guide-AOS-v50:acl-acli-vm-auto-r.html

In any version after 5.0 (5.1, 5.2, 5.5) the command is gone. On 5.5 you get an error:
Unknown keyword argument: nested_hv

The nested_hv is gone from the ACLI documentation:
https://portal.nutanix.com/#/page/docs/details?targetId=Command-Ref-AOS-v55:acl-acli-vm-auto-r.html


Use case is for a simulator VM that runs KVM underneath.
Is there another way to do this in 5.5 or was the feature just removed?
I opened a case and got this reply - It answers the question. I used this line:

code:

admin@BLAH~$ acli vm.update VIRL cpu_passthrough=true
VIRL: pending
VIRL: complete
admin@BLAH~$



======EMAIL FROM NUTANIX TECH SUPPORT========

Severity:
====
P3 - Normal


Action plan:
====
+ Although the nested_vm switch was there in AOS 5.0 it actually didn't work and nested VM wasn't supported until AOS 5.5.0.4.
+ Please find the release notes below for AOS 5.5.0.4

https://portal.nutanix.com/#/page/docs/details?targetId=Release-Notes-Acr-v5504:Release-Notes-Acr-v5504

+ Please find the abstract below.

New Features

Nested VMs
PM-615
  • Nutanix now provides limited support for nested virtualization, specifically nested KVM VMs in an AHV cluster as of AOS 5.5.0.4 with AHV-20170830.58. Enabling nested virtualization will disable live migration and high availability features for the nested VM. You must power off nested VMs during maintenance events that require live migration.
+ I'd recommend to upgrade to the latest AOS 5.5.0.5 and then AHV to AHV-20170830.58 as per documenation.
+ Once upgraded, you can use the following to enable nested VM. This passes through all the required CPU features to allow nested virtualization

acli vm.update vmname cpu_passthrough=true

Reply