Installation & Configuration

Welcome to the Nutanix NEXT community. To get started please read our short welcome post. Thanks!

cancel
Showing results for 
Search instead for 
Did you mean: 

Need to change cluster network

SOLVED Go to solution
Adventurer

Need to change cluster network

Hi,

 

I need to change the IP addresses of an installed cluster to use a different subnet (move from 10.x.x.x/24 to 192.168.y.y/25).

 

When following the documentation on how to change CVM addresses, the reconfiguration website fails because the hypervisor is not in the new CVM subnet. Thus the IP CVM addresses cannot be changed.

 

How can a complete cluster, including hypervisor and IPMI, be moved from using one set of IP addresses from subnet 1 to using a different set of IP addresses from subnet 2?

 

Thanks,

Erik

1 ACCEPTED SOLUTION

Accepted Solutions
Adventurer

Re: Need to change cluster network

@hienle, if you want to stay in the same IP network (subnet), the normal procedures should work. You would need to use some temporary IP addresses for swapping / shuffling the already used ones.

 

Then there is the general problem that current versions of Chrome and Firefox do not support IPv6 link-local addresses. Therefore the documented way to change CVM IP addresses does not work. It can be made to work by e.g. using SSH port forwarding to forward a local port to port 2100 of one CVM using the CVM's IPv6 link-local for the SSH tunnel. I have used that in the past.

 

The support was not able to give me a correct IP change procedure, but figured out how to do it anyway. The problem with the procedure from support was that they wanted me to restart Genesis at the wrong time (at several wrong times, actually). There is just one manual Genesis restart just before re-starting the cluster.

 

To change the cluster IP network, the cluster needs to be stopped and all CVMs need to be rebooted. Thus there needs to be a maintenance window scheduled with sufficient time for a lengthy manual change procedure, reboots of all CVMs, manual clean-up afterwards, and tests. The maintenance windows needs to include sufficient time for rollback and tests as well.

 

I have used the following procedure:

 

----------------------------------------------------------------------------------

 

A. Pre-change verification / adjustment
1. Verify that the DNS server is reachable using both old and new addresses
[normal documentation asks to remove the DNS server if it will not be
usable after the change]
ncli cluster get-name-servers
2. Verify that the NTP server is reachable using both old and new addresses
[normal documentation asks to remove the DNS server if it will not be
usable after the change]
ncli cluster get-ntp-servers
3. Verify that the "metadata store" is enabled on all nodes:
[on one CVM] ncli host ls | grep "Metadata store status"

 

B. Prepare cluster for reconfiguration
1. Stop cluster
[on one CVM] cluster stop
2. Put cluster into reconfiguration mode
[on one CVM] cluster reconfig
3. Wait for Genesis to await configuration via RPC
[on one CVM] allssh tail -n1 /home/nutanix/data/logs/genesis.out

[all CVMs must show a line like "2017-03-23 00:38:41 INFO node_manager.py:5191 Waiting for node to become configured through RPC." before proceeding]

 

C. Change the IP address configuration of all CVMs
1. For each node in the cluster
a. Log on to hypervisor host
b. Connect to the local CVM via internal network
ssh nutanix@192.168.5.254
c. Change CVM network configuration to new IP address
sudoedit /etc/sysconfig/network-scripts/ifcfg-eth0
[this change will not take effect until restarting the CVM network,
which will be done by rebooting in step F]

 

D. Update the IP addresses of the Zookeeper hosts
1. On one CVM
a. edit-zkmapping check_can_modify_zk_map
b. edit-zkmapping modify_zk_server_config
[this automatically sets the new IP addresses from step C]
c. edit-zkmapping check_can_modify_zk_map

 

E. Change all hypervisor host IP addresses
1. For each node in the cluster
a. Follow the hypervisor documentation to change the IP address
[This includes an "/etc/init.d/network restart" on AHV]
[Use Nutanix internal network to change ESXi IP & gateway via esxcli]
esxcli network ip interface ipv4 set -i vmk0 -I <IPADDRESS> -N <NETMASK> -t static
esxcli network ip route ipv4 add -g <GATEWAY_IP> -n default

 

F. Reboot all CVMs
1. For each node in the cluster
a. Log on to CVM and reboot the CVM
sudo reboot

 

G. Take all CVMs out of reconfiguration mode
1. For each node in the cluster
a. Log on to CVM
b. Take CVM out of reconfiguration mode
[on each CVM] rm -v ~/.node_reconfigure
c. Restart genesis
[on each CVM] genesis restart
d. Wait long enough for genesis to restart
[on one CVM] sleep 5m
e. Check cluster status (should return state 'stop' and 'success' at end)

 

H. Start the cluster
1. [on one CVM] cluster start

 

I. Change virtual cluster address
1. Log on to Prism
2. Open "Cluster Details" from gears menu and change the virtual IP

 

J. Change all IPMI IP addresses
1. For each node in the cluster
a. Log on to hypervisor host
b. use ipmitool to reconfigure IPMI IP address
ipmitool lan print 1
ipmitool lan set 1 ipaddr <IPADDRESS>
ipmitool lan set 1 netmask <NETMASK>
ipmitool lan set 1 defgw ipaddr <GATEWAY_IP>
[use /ipmitool on ESXi, ipmitool on AHV]

c. Restart Genesis
genesis restart

 

K. If necessary, remove old DNS & NTP server

 

L. Verify the cluster works

 

----------------------------------------------------------------------------------

 

From looking at some threads in this forum, the old way to manually reconfigure IP addresses involved editing Zookeeper configuration files. That part has been replaced by a Python script, i.e. edit-zkmapping. This script relies on Genesis being in the correct state. Manually editing configuration files and then restarting everything (e.g. by rebooting) would have been more robust and saved me a painful experience with Nutanix support. What have been at most one hour (including preparation and tests) took two and a half days. :-(

 

A Nutanix employee has told me that they plan to publish a manual IP change procedure soon...

 

Best regards,

Erik

 

Edited to add step J.1.c.: Genesis needs to be restarted after the (regular) IPMI IP address change. The IPMI address change is executed after the cluster has started, thus the Genesis restart that noticed the new CVM and hypervisor IP addresses did still see the old IPMI addresses.

4 REPLIES
Adventurer

Re: Need to change cluster network

I have opened a support case and received instructions on how to do this. This is a manual procedure and quite cumbersome. Additionally, the procedure failed, at least initially.

 

I plan to add another post after successfully completing the cluster IP address change from one IP network (aka subnet) to another.

Outrider

Re: Need to change cluster network

@auerswald hope to see your guide soon

 

currently, Nutanix recommends us to change the CVM IP address based on reconfiguring-mode of Cluster via web browser as you did but this sometimes lead to failed cases (I also gave it a try but failed) . Unluckily, Nutanix has changed the content of the section on how to reconfigure the CVM IP by manually (They actually had a section showed us how to do that but now they deleted it and changed to other content that recommending us to have Nutanix SE consulted)

 

By the way, basically we need to change the IP from left to right (IPMI -> Host -> CVM) to make sure that we can have everything under our control and easier to troubleshoot if any bad things occured.

 

Good luck

Adventurer

Re: Need to change cluster network

@hienle, if you want to stay in the same IP network (subnet), the normal procedures should work. You would need to use some temporary IP addresses for swapping / shuffling the already used ones.

 

Then there is the general problem that current versions of Chrome and Firefox do not support IPv6 link-local addresses. Therefore the documented way to change CVM IP addresses does not work. It can be made to work by e.g. using SSH port forwarding to forward a local port to port 2100 of one CVM using the CVM's IPv6 link-local for the SSH tunnel. I have used that in the past.

 

The support was not able to give me a correct IP change procedure, but figured out how to do it anyway. The problem with the procedure from support was that they wanted me to restart Genesis at the wrong time (at several wrong times, actually). There is just one manual Genesis restart just before re-starting the cluster.

 

To change the cluster IP network, the cluster needs to be stopped and all CVMs need to be rebooted. Thus there needs to be a maintenance window scheduled with sufficient time for a lengthy manual change procedure, reboots of all CVMs, manual clean-up afterwards, and tests. The maintenance windows needs to include sufficient time for rollback and tests as well.

 

I have used the following procedure:

 

----------------------------------------------------------------------------------

 

A. Pre-change verification / adjustment
1. Verify that the DNS server is reachable using both old and new addresses
[normal documentation asks to remove the DNS server if it will not be
usable after the change]
ncli cluster get-name-servers
2. Verify that the NTP server is reachable using both old and new addresses
[normal documentation asks to remove the DNS server if it will not be
usable after the change]
ncli cluster get-ntp-servers
3. Verify that the "metadata store" is enabled on all nodes:
[on one CVM] ncli host ls | grep "Metadata store status"

 

B. Prepare cluster for reconfiguration
1. Stop cluster
[on one CVM] cluster stop
2. Put cluster into reconfiguration mode
[on one CVM] cluster reconfig
3. Wait for Genesis to await configuration via RPC
[on one CVM] allssh tail -n1 /home/nutanix/data/logs/genesis.out

[all CVMs must show a line like "2017-03-23 00:38:41 INFO node_manager.py:5191 Waiting for node to become configured through RPC." before proceeding]

 

C. Change the IP address configuration of all CVMs
1. For each node in the cluster
a. Log on to hypervisor host
b. Connect to the local CVM via internal network
ssh nutanix@192.168.5.254
c. Change CVM network configuration to new IP address
sudoedit /etc/sysconfig/network-scripts/ifcfg-eth0
[this change will not take effect until restarting the CVM network,
which will be done by rebooting in step F]

 

D. Update the IP addresses of the Zookeeper hosts
1. On one CVM
a. edit-zkmapping check_can_modify_zk_map
b. edit-zkmapping modify_zk_server_config
[this automatically sets the new IP addresses from step C]
c. edit-zkmapping check_can_modify_zk_map

 

E. Change all hypervisor host IP addresses
1. For each node in the cluster
a. Follow the hypervisor documentation to change the IP address
[This includes an "/etc/init.d/network restart" on AHV]
[Use Nutanix internal network to change ESXi IP & gateway via esxcli]
esxcli network ip interface ipv4 set -i vmk0 -I <IPADDRESS> -N <NETMASK> -t static
esxcli network ip route ipv4 add -g <GATEWAY_IP> -n default

 

F. Reboot all CVMs
1. For each node in the cluster
a. Log on to CVM and reboot the CVM
sudo reboot

 

G. Take all CVMs out of reconfiguration mode
1. For each node in the cluster
a. Log on to CVM
b. Take CVM out of reconfiguration mode
[on each CVM] rm -v ~/.node_reconfigure
c. Restart genesis
[on each CVM] genesis restart
d. Wait long enough for genesis to restart
[on one CVM] sleep 5m
e. Check cluster status (should return state 'stop' and 'success' at end)

 

H. Start the cluster
1. [on one CVM] cluster start

 

I. Change virtual cluster address
1. Log on to Prism
2. Open "Cluster Details" from gears menu and change the virtual IP

 

J. Change all IPMI IP addresses
1. For each node in the cluster
a. Log on to hypervisor host
b. use ipmitool to reconfigure IPMI IP address
ipmitool lan print 1
ipmitool lan set 1 ipaddr <IPADDRESS>
ipmitool lan set 1 netmask <NETMASK>
ipmitool lan set 1 defgw ipaddr <GATEWAY_IP>
[use /ipmitool on ESXi, ipmitool on AHV]

c. Restart Genesis
genesis restart

 

K. If necessary, remove old DNS & NTP server

 

L. Verify the cluster works

 

----------------------------------------------------------------------------------

 

From looking at some threads in this forum, the old way to manually reconfigure IP addresses involved editing Zookeeper configuration files. That part has been replaced by a Python script, i.e. edit-zkmapping. This script relies on Genesis being in the correct state. Manually editing configuration files and then restarting everything (e.g. by rebooting) would have been more robust and saved me a painful experience with Nutanix support. What have been at most one hour (including preparation and tests) took two and a half days. :-(

 

A Nutanix employee has told me that they plan to publish a manual IP change procedure soon...

 

Best regards,

Erik

 

Edited to add step J.1.c.: Genesis needs to be restarted after the (regular) IPMI IP address change. The IPMI address change is executed after the cluster has started, thus the Genesis restart that noticed the new CVM and hypervisor IP addresses did still see the old IPMI addresses.

Highlighted
Community Manager

Re: Need to change cluster network

Thanks for sharing @auerswald