Failures attempting to fix consitency across nodes | Nutanix Community
Skip to main content

I’m trying not to need to re-image a node, but the networking does not seem to be cooperating.

Here is the error I’m getting:

Errors in cluster networking
Bond configuration is not consistent across nodes. No bond found on host:192.168.86.70, but present on host:192.168.86.170

Here is .170

sroot@NTNX-895b5886-A ~]# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.86.170/24 brd 192.168.86.255 scope global br0
       valid_lft forever preferred_lft forever
15: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.5.1/24 brd 192.168.5.255 scope global virbr0
       valid_lft forever preferred_lft forever
rroot@NTNX-895b5886-A ~]# 
rroot@NTNX-895b5886-A ~]# ovs-vsctl list bridge |grep name
name                : br2.local
name                : br0.local
name                : br.microseg
name                : br1
name                : br1.local
name                : brSpan
name                : br0
name                : br.mx
name                : br.nf
name                : br.dmx
name                : br2
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br2.local
br2.local.d
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br0.local
br0.local.d
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br.microseg
br.microseg.d
br.microseg.u
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br1
br1-arp
br1-dhcp
br1-up
br1.u
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br1.local
br1.local.d
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports brSpan
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br0
br0-arp
br0-dhcp
br0-up
br0.u
vnet0
vnet2
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br.mx
br.mx.d
br.mx.u.br0
br.mx.u.br1
br.mx.u.br2
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br.nf
br.nf.d
br.nf.u
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br.dmx
br.dmx.d.br0
br.dmx.d.br1
br.dmx.d.br2
br.dmx.u
rroot@NTNX-895b5886-A ~]# ovs-vsctl list-ports br2
br2-arp
br2-dhcp
br2.u
eth0
vnet4
rroot@NTNX-895b5886-A ~]# ovsdb-client dump
<ouput omitted>
676261b1-8a07-4acf-8de6-8d8fd4866c78 "a0:36:9f:1f:c1:18" 0              false           balance-tcp 0             ]     {}           false        50a1474a-f1b5-4af2-a142-66d0c5749b9a, cb13ef58-5cb7-4562-86ad-2de1ed75e39c] active i]  br0-up        {lacp-fallback-ab=True, lacp-time=fast} false      ]  {}              {}          {}         {}      ]  []      ]
7d91390d-8e61-4a88-8c75-0731e8f51535 "a0:36:9f:1f:c1:1a" 0              false           balance-tcp 0             ]     {}           false        17e55f0e-c671-4065-8bc5-cb95f466d564, 53ec5f8d-de7e-4d3a-a316-19d2910bb25d] active i]  br1-up        {lacp-fallback-ab=True, lacp-time=fast} false      ]  {}              {}          {}         {}      ]  []      ]

and here is the .70

oroot@NTNX-244131ad-A ~]# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.86.70/24 brd 192.168.86.255 scope global br0
       valid_lft forever preferred_lft forever
15: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 192.168.5.1/24 brd 192.168.5.255 scope global virbr0
       valid_lft forever preferred_lft forever
rroot@NTNX-244131ad-A ~]# ovs-vsctl list bridge |grep name
name                : brSpan
name                : br1
name                : br.mx
name                : br0
name                : br0.local
name                : br.dmx
name                : br.nf
name                : br.microseg
name                : br1.local
name                : br1-up
name                : br1-up.local
rroot@NTNX-244131ad-A ~]# 
rroot@NTNX-244131ad-A ~]# ovsdb-client dump
<output omitted>
6adb357e-5e3d-44d6-a8e4-4fdc09fc90e7 0]                  0              false            ]          0             ]     {}           false        717665f9-d270-484d-8397-ba11bef7edbd]                                                                              ]      ]  br1-up          {}                                      false      ]  {}              {}          {}         {}      ]  []      ]

52edc5f3-c290-4af6-836d-65240d5bd478 "a0:36:9f:8d:f2:20" 0              false           balance-tcp 0             ]     {}           false        1ec4ec6b-c3d2-4166-96b8-5395dd0babec, c1ee5617-4d42-4c6d-aabc-50bf4bb84bae, d4ca1e3b-cae9-47a3-b7e1-6c8aa4ddf46c] active i]  br0-up          {lacp-fallback-ab=True, lacp-time=fast} false      ]  {}              {}          {}         {}      ]  []      ]


 

 

Is the problem that in .170 that the br1-up is a port in bridge br1 and by contract on .70 that br1-up is itself a bridge?

 

I think I will just need to re-AHV the NTNX-244131ad-A node...

Your networking needs to be IDENTICAL for number and name of bonds on each node.  IPs can be different, but the same bonds need to be on each node.  You have br2 on your 170 node and not on your 70 node.  Also, unless you are changing the bond mode, avoid ovs-vsctl for configuration, do it through prism element in the virtual networking configuration, otherwise the nutanix will get temperamental.   

broot@NTNX-895b5886-A ~]# ovs-vsctl list bridge |grep name name : br2.local name : br0.local name : br.microseg name : br1 name : br1.local name : brSpan name : br0 name : br.mx name : br.nf name : br.dmx name : br2

broot@NTNX-244131ad-A ~]# ovs-vsctl list bridge |grep name name : brSpan name : br1 name : br.mx name : br0 name : br0.local name : br.dmx name : br.nf name : br.microseg name : br1.local name : br1-up name : br1-up.local

Reply


Terms & Conditions