Do you know you can run a individual log collector just for LCM logs if you are having any troubles running updates from LCM?
LCM (Life Cycle Manager) Log Collection Utility provides an independent script designed to collect all the important log files necessary to debug any LCM issue.
The utility is available in LCM version 2.2 or later by default. It could also be used on earlier versions of LCM.
Functionality:
- Collects LCM log files.
- Detects if the node is stuck in Phoenix and collects kernel logs.
- Collects logs for the services with which LCM interacts (Example: Foundation, Prism, etc.).
- The utility state is independent of such of the LCM hence it is capable of collecting logs even if the framework is down.
- Parses LCM configurations from Zookeeper nodes.
Files/Info being collected:
- Node Level Information
- LCM leader
- Foundation version
- LCM version
- Hypervisor type/version
- LCM configuration
- /etc/nutanix/hardware_config.json
For LCM 2.3.1 and later (Automatic log collection)
- LCM 2.3.1 automatically triggers the LCM log collector utility when any LCM operation fails.
- Log bundle will be created in /home/nutanix/data/log_collector/.
- For LCM 2.3.1.1 and later, LCM prints Log location along with the failure message as below. Log into the CVM IP to fetch the LCM log bundle and upload it to the Support case for review.
Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: ixx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).
Logs have been collected and are available to download on xx.xx.x.xx at /home/nutanix/data/log_collector/lcm_logs__xx.xx.x.xx__2020-05-05_08-48-19.113101.tar.gz
For LCM 2.2.x and later (Manual log collection)
- It can be used to trigger a new log collection.
- Run the following command from a CVM to begin log collection.
nutanix@cvm$ python /home/nutanix/cluster/bin/lcm/lcm_log_collector.py
More Information - https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e0000009CUZCA2