Question

First install of CE fails - Network issue unable to route to 192.168.5.2

  • 8 September 2019
  • 13 replies
  • 3996 views

Badge
Hi Everyone,

I've got 3 old Lenovo Tiny computers that sort of meet the bare minimum for running a little home-lab nutanix setup. I've been using them for months at home for my esxi setup but with a new AHV based rollout being planned for a few of our clients, I thought I'd get ahead by running up my home lab to get me familiar with Prism and migration from a few hypervisors (esxi and hyper-v) to AHV.

The lenovo tiny's are physically hardware configured like this:
  • 256GB M.2 SSD
  • 1TB 2.5" HDD
  • 64GB USB
  • 16GB RAM (though if proof of concept works, i'll go replace those with 16GB modules making 32GB per node)
I've built this as a bare metal. No nested virtualisation here.

Network assignment is a new virtual network, it doesn't really matter but i've got a cisco switch and a Barracuda CGF router. VLAN16 is my lan. Router is on 10.135.16.253.
I've got a Ubuntu machine on this network, it's able to route out to google browse sites, and even able to ping the host

My IP Assignment goes like this:
Host:
  • Host IP: 10.135.16.1
  • Host Subnet: 255.255.255.0 (24bit)
  • Host Gateway: 10.135.16.253
CVM:
  • CVM IP: 10.135.16.11
  • CVM Subnet: 255.255.255.0 (24bit)
  • Host Gateway: 10.135.16.253




The install gets to the point where it wants to autostart the NTNX-28028e87-A-CVM virtual machine. Then it eventually times out.



I looked at log file 'firstboot.out' and i can see the following lines:
  • INFO: Running cmd ['virsh autostart "NTNX-28028e87-A-VCM"]
  • INFO: Run ssh cmd on SVM
  • INFO: Running cmd ['/usr/bin/ssh -i /root/firstboot/ssh_keys/nutanix -o StrictHostKeyChecking=no -o NumberOfPasswordPrompt...
  • ssh: connect to host 192.168.5.2 port 22: No route to host]. Will retry in 5 seconds
  • INFO: Running cmd ['/usr/bin/ssh -i /root/firstboot/ssh_keys/nutanix -o StrictHostKeyChecking=no -o NumberOfPasswordPrompt...
  • ssh: connect to host 192.168.5.2 port 22: No route to host]. Will retry in 5 seconds
  • ...
  • ...



At first i had no idea this network was there and I wondered if i made a mistake, but it seems like no. That must be an internal network for CVM autopathing based on the googleing i did, i ended up here: https://vcdx133.com/2015/07/19/nutanix-xcp-deep-dive-part-17-cvm-autopathing-with-esxi/ and if i had to guess without knowing - i'd guess it's used for NOS to talk to it's VMs.

Anyway - clearly a network issue so I jump on to the machine with a keyboard and mouse (which i needed to install the machine anyway). First thing I did was try to ping 192.168.5.2. Destination host unreachable.



I check ip a and have a look at the network interfaces. Surprisingly theres 7 network interfaces, not including loopback. There's really only 2 that look interesting

br 0: IP: 10.135.16.1/24 brd 10.135.16.255
virbr0: IP 192.168.5.1/24 brd 192.168.5.255



I ping 192.168.5.1, pings back. Its myself, so i'm not surprised.
I ping 192.168.5.2, nothing


So i google again. I read that virsh is kvm's commands to check on guest vm's and in this case i want to see if the CVM guest is running so i go:

virsh list --all
code:
Id              Name                                      State
1 NTNX-28028e87-A-CVM Running



I dont' know anything about KVM but i use virsh -help and see there's a console command to connect to a VM's console so i make an educated guess

virsh console NTNX*
error: Failed to get domain 'NTNX-CVM.xml'
error: Domain not found: no domain with matching name NTNX-CVM.xml

Well.. not knowing kvm or nutanix yet, i can only guess that this is the configuration file used to setup the VM parameters or something since google tells me to edit that file. But i can run this command:
virsh dumpxml 1 | more
and that gives me an XML configuration for the CVM machine.

Now I've reinstalled a few times. So my Nutanix CVM machine name has been changing.

When i browse to /root i can see a file NTNX-CVM.xml and i can see there what looks to be a template.

But this is where I'm stuck.

I need some advice since i've got some surrounding information but no clear path forward. Even just advice for log files i haven't checked would be nice. I don't mind doing the legwork but this is my first time installing CE.

13 replies

Badge

Tell me more ChrispyChops? Can you post a screenshot of the two bits you changed? Do you mean KVM config file for the CVM? How’d you change the boot drive as SAS? Bios? 

By the by, it’s working fine on a set of 3 lenovo M4’s and intel NUC but these little lenovo Tiny machines i never got further and they’re 2x the CPU grunt of the NUC CPU’s which is a bit of a shame. It’d be really good if you can help :) 

 

 

Did you ever work this out? Had the same issue.

I ended up changing the OS to Other/Other (64-bit) and setting the boot drive as SAS 0:0

It now starts up the CVM properly.

Badge

OK.. more reading:

I’ve found that logs for the CVM vm are located in: 

[root@NTNX-53a761df-A qemu]# cd /var/log/libvirt/qemu/

in there i can check the related CVM log (there’s a couple of logs from reinstalls). Since all three have the same issue, all three logs are the same for me. 

 

This is the end of the log file for the NON working Lenovo Tiny:

Domain id=1 is tainted: host-cpu
20191116 00:21:39.942769 qemu-kvm: System reset requested from keyboard
20191116 00:21:40.294742 qemu-kvm: System reset requested from pci
20191116 00:22:05.996956 qemu-kvm: System reset requested from keyboard
20191116 00:22:06.351122 qemu-kvm: System reset requested from pci
20191116 00:22:32.082198 qemu-kvm: System reset requested from keyboard
20191116 00:22:32.437290 qemu-kvm: System reset requested from pci
20191116 00:22:58.171660 qemu-kvm: System reset requested from keyboard
20191116 00:22:58.526616 qemu-kvm: System reset requested from pci
20191116 00:23:24.268388 qemu-kvm: System reset requested from keyboard
20191116 00:23:24.623731 qemu-kvm: System reset requested from pci
20191116 00:23:50.366456 qemu-kvm: System reset requested from keyboard
20191116 00:23:50.719657 qemu-kvm: System reset requested from pci
20191116 00:24:16.463295 qemu-kvm: System reset requested from keyboard
20191116 00:24:16.818137 qemu-kvm: System reset requested from pci
20191116 00:24:42.560020 qemu-kvm: System reset requested from keyboard
20191116 00:24:42.911126 qemu-kvm: System reset requested from pci
20191116 00:25:08.665544 qemu-kvm: System reset requested from keyboard
20191116 00:25:09.022637 qemu-kvm: System reset requested from pci
20191116 00:25:30.679752 qemu-kvm: System powerdown requested from qmp
20191116 00:25:34.732111 qemu-kvm: System reset requested from keyboard
20191116 00:25:35.085014 qemu-kvm: System reset requested from pci
20191116 00:26:00.771466 qemu-kvm: System reset requested from keyboard
20191116 00:26:01.123654 qemu-kvm: System reset requested from pci
20191116 00:26:24.859363 qemu-kvm: System reset requested from qmp
20191116 00:26:25.211233 qemu-kvm: System reset requested from pci
20191116 00:26:50.957578 qemu-kvm: System reset requested from keyboard
20191116 00:26:51.309265 qemu-kvm: System reset requested from pci
20191116 00:27:17.044205 qemu-kvm: System reset requested from keyboard
20191116 00:27:17.394981 qemu-kvm: System reset requested from pci
20191116 00:27:43.093387 qemu-kvm: System reset requested from keyboard
20191116 00:27:43.450720 qemu-kvm: System reset requested from pci
20191116 00:27:59.969694 qemu-kvm: System powerdown requested from qmp
20191116 00:28:02.296633 qemu-kvm: System reset requested from qmp
20191116 00:28:02.648191 qemu-kvm: System reset requested from pci
20191116 00:28:28.380915 qemu-kvm: System reset requested from keyboard

But that all sounds bad - however let me also point out that my working-first-time after installation nutanix on the Intel NUCs all show something similar to this (This is a working log, with a perfectly healthy CVM):

Domain id=4 is tainted: host-cpu
2019-10-07T02:28:53.465606Z qemu-kvm: Warning: Large machine and max_ram_below_4g (536870912) not a multiple of 1G; possible bad performance.
20191021 12:47:51.905545 qemu-kvm: System powerdown requested from qmp
20191021 12:48:18.024682 qemu-kvm: System shutdown requested from acpi
20191021 12:48:18.026208 qemu-kvm: System reset requested from qmp
20191021 12:48:18.042817 qemu-kvm: System reset requested from pci
20191021 13:04:17.499471 qemu-kvm: System powerdown requested from qmp
20191021 13:04:23.032725 qemu-kvm: System shutdown requested from acpi
2019-10-21T13:04:23.034321Z qemu-kvm: terminating on signal 15 from pid 1178
2019-10-21 13:04:23.289+0000: shutting down
2019-10-21 13:05:29.830+0000: starting up libvirt version: 2.0.0, package: 12.20190211.1.23763.el7 (Unknown, 2019-02-12-01:50:31, f7f4189726e1), qemu version: 2.6.0 (qemu-kvm-2.6.0-30.20190211.1.45983.el7), hostname: NTNX-8e534b03-A
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=NTNX-8e534b03-A-CVM,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-687efdab-661b-41eb-b22b-c1ec079df140/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,max-ram-below-4g=536870912 -cpu host,+kvm_pv_eoi -m 12288 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 687efdab-661b-41eb-b22b-c1ec079df140 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-687efdab-661b-41eb-b22b-c1ec079df140/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=off,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/NTNX-CVM/svmboot.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/dev/disk/by-id/ata-WDC_WD10SPZX-21Z10T0_WD-WXK1A293XT3X,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native -device 'scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,wwn=0x50014ee2bb9fc5b5,vendor=ATA,product=WDC WD10SPZX-21Z' -drive file=/dev/disk/by-id/ata-CT250MX500SSD4_1916E1FC98C9,format=raw,if=none,id=drive-scsi0-0-0-1,serial=1916E1FC98C9,cache=none,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1,wwn=0x500a0751e1fc98c9,vendor=ATA,product=CT250MX500SSD4 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=50:6b:8d:d2:ed:7e,bus=pci.0,addr=0x3 -netdev tap,fd=28,id=hostnet1,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=50:6b:8d:0d:9c:18,bus=pci.0,addr=0x4 -chardev file,id=charserial0,path=/tmp/NTNX.serial.out.0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Domain id=16 is tainted: host-cpu
2019-10-21T13:05:31.391209Z qemu-kvm: Warning: Large machine and max_ram_below_4g (536870912) not a multiple of 1G; possible bad performance.

Searching for CVM “System reset requested from Keyboard” isn’t showing much on the forums or on google. I think the message “Domain id=X is tainted: host-cpu” is innocuous since that exists on both working and non-working. 

 

The standout note here seems to be that my working CVM log doesn’t have this “System reset requested from keyboard” but the non-working CVM log does. 

I don’t know how to stop this but i might play with BIOS and enable legacy ps2 or something to see what happens.

Badge

Hi I didn’t update this for a while because initially i was asked to SSH to the CVM which i explained in the first post wasn’t possible so I gave up.

 

I decided to bite the very expensive bullet and bought a part list of Intel NUCs which were part identical to a guide linked in the download CE post. With the Intel NUCs i’ve got that working using 256GB NVMe and 500GB HDD and 32GB of ram in each. I had no issues at all.

 

However, after migrating all my servers from the two Lenovo Tiny VMWare ESXi ‘servers’ i had running, i’ve noticed that CPU is a bit of a constraint. The 7th gen intel NUCs are only 2 cores really. The lenovo tiny i5’s run 6th gen intels but the part is 4core. Keen to potentially push past the issue of the CVM not booting, i tried again today.

 

So I started up the Lenovo tiny and started reading.  After initially being unable to get console access to the CVM via virsh i edited the XML file and found that all i needed to do was really change the console type from file to pty. https://geekysnippets.com/2018/01/29/how-to-enable-kvm-virsh-console-access-for-linux-vm/ 

 

Now that I can connect to the console I can see the CVM is stuck in a perpetual boot loop. Though I’ve got no reason for why. Please see this screenshot: 

 

Nothing in that list seems to be a problem, so I’m confused to why it’s rebooting. Does anyone have some advice?

I have used 3 Samsung EVO 850 SSD today. (instead of SK Hynix)

Good news, I’ll confirm that it is working now. Thx for the guidance.

not sure i’m using NVMe, i’m going to do some extra tests with other disks.

The one i’m using now is 2.5” SKhynix SC308 SATA 512GB.

 

thx for the feedback.

Badge +1

I was told the issue is because CE doesn’t yet support CVM booting from NVMe devices.

Hello,

 

I tried to install CE on a dell R710 today with a stable release from November 2th.

Unfortunately, I’m running into the same issue as above.

 

Does anyone have a solution yet?

 

Many thx for helping,

kind regards,

Carl

Badge +1

I was able to connect to the console of the cvm….it’s in a continuous boot loop.  I grabbed a screen shot:

 

This server does have a pair of Intel DC P3700 NVMe drives.  Could that be the problem.  During install it appears they get set to PCI Pass through:

 

 

Badge +1
I'm having same issue installing CE on some HPE DL380 Gen9's. CVM appears shows as running, however, I get the same "No route to host" in the firstboot log. Were you able to figure out the problem?
I had exact the same issue this week.
Difference is that I was installing NCE as nested AHV on ESXi. And AHV VMs were modified - added 3 more NICs to setup.
Reinstalled with only one NIC per VM, and issue did not appear again.
Maybe this will be a clue, but I can not be sure.
Userlevel 2
Badge +4
Hey AidenClark,

You found the correct documentation about the 192.168.5.1 and 2 addresses. These are internal between the CVM (the .2) and the Host (the .1). The virtsh console command output is normal and that xml file is just that, the config file.

If you dump that XML file and look in the interfaces section, make sure you see an interface entry for br0 (your private network) and NTNX-Local-Network bridge=virbr0.

Try to SSH into the CVM Private IP you gave it ( 10.135.16.11) from the AHV host and see if it is actually running.
Badge
Hate to shamelessly bump this, but all I think I need is a short list of logs that would be valuable to check.

So does anyone know where install log locations are?
CVM VM log locations?

Does this smell like the KVM machine hasn't booted? If so, and it says its running, how can i check?

Reply