Skip to main content

Hi,

we just upgraded our prism central instance to the most current community edition version of Nutanix (2020.09.16). We also upgraded our cluster successfully. For this to happen I had to unregister the cluster from prism central first (with the help of: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e000000XeZjCAK). After everything was updated I wanted to register the cluster again in prism central, but after some time I get the error message “Error: Failed to create remote connection”. I found this KB article: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e0000009BwxCAE and executed all the steps therein (which are mostly the steps I already had done), but the issue persists.

 

On the prism central VM I can see the following entries in the “prism_gateway.log” file when trying to register the cluster:
 

WARN  2021-05-12 15:17:55,410Z http-nio-0.0.0.0-9081-exec-8 8] commands.auth.PAMAuthenticationProvider.checkFailedAttempts:395 Failed to check the Failed Attempts..                          
WARN 2021-05-12 15:17:55,437Z http-nio-0.0.0.0-9081-exec-8 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
INFO 2021-05-12 15:17:55,446Z http-nio-0.0.0.0-9081-exec-8 8] commands.upgrade.ValidateCompatibility.doExecute:96 Checking PE compatibility, installed PC version: 2020.09.16, target PE vers
ion: 2020.09.16
INFO 2021-05-12 15:17:55,448Z http-nio-0.0.0.0-9081-exec-8 8] upgrade.retreivers.RetrieverImpl.getHttpsConnection:378 Retrieving staging server metadata from https://release-api.nutanix.com
/api/v2/prismcentral/2020.09.16?stage=GA

ERROR 2021-05-12 15:17:55,589Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible:66 Unable to fetch versions from portal
java.lang.NullPointerException
at com.nutanix.prism.commands.upgrade.retreivers.RetrieverImpl.getSoftwareVersions(RetrieverImpl.java:18

<further stacktrace messages>


WARN 2021-05-12 15:17:55,591Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible:72 falling back to zkNode compatibility for pc version 2
020.09.16 and software versions from portal null
WARN 2021-05-12 15:17:55,592Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.ZkNodeCompatibilityMatrix.isNosVersionCompatible:61 falling back to static compatibility for pc version 2
020.09.16 and software versions from zknode at path prism_central
INFO 2021-05-12 15:17:55,597Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.StaticCompatibilityMatrix.getCeVersionMap:239 Got CE version map with 53 entries
INFO 2021-05-12 15:17:55,598Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.StaticCompatibilityMatrix.getNosVersionForCE:227 getNosVersionForCE(2020.09.16) - containsKey: true, ver:
5.18.0.6
INFO 2021-05-12 15:17:55,599Z http-nio-0.0.0.0-9081-exec-8 8] commands.multicluster.StaticCompatibilityMatrix.isNosVersionCompatible:181 PC is compatible with PE of version 2020.09.16

WARN 2021-05-12 15:17:55,856Z http-nio-0.0.0.0-9081-exec-4 8] commands.auth.PAMAuthenticationProvider.checkFailedAttempts:395 Failed to check the Failed Attempts..
INFO 2021-05-12 15:17:55,861Z http-nio-0.0.0.0-9081-exec-4 8] multicluster.registration.AddClusterExternalState.prepareToExecute:114 Prism Element version is lesser than 5.10 : 2020.09.16
INFO 2021-05-12 15:17:55,864Z http-nio-0.0.0.0-9081-exec-4 8] commands.upgrade.ValidateCompatibility.doExecute:96 Checking PE compatibility, installed PC version: 2020.09.16, target PE vers
ion: 2020.09.16
INFO 2021-05-12 15:17:55,866Z http-nio-0.0.0.0-9081-exec-4 8] upgrade.retreivers.RetrieverImpl.getHttpsConnection:378 Retrieving staging server metadata from https://release-api.nutanix.com
/api/v2/prismcentral/2020.09.16?stage=GA
ERROR 2021-05-12 15:17:55,989Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible:66 Unable to fetch versions from portal
java.lang.NullPointerException
at com.nutanix.prism.commands.upgrade.retreivers.RetrieverImpl.getSoftwareVersions(RetrieverImpl.java:188)
at com.nutanix.prism.commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible(PortalCompatibilityMatrix.java:64)

<further stacktrace messages>


WARN 2021-05-12 15:17:55,990Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible:72 falling back to zkNode compatibility for pc version 2
020.09.16 and software versions from portal null
WARN 2021-05-12 15:17:55,991Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.ZkNodeCompatibilityMatrix.isNosVersionCompatible:61 falling back to static compatibility for pc version 2
020.09.16 and software versions from zknode at path prism_central
INFO 2021-05-12 15:17:55,995Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.getCeVersionMap:239 Got CE version map with 53 entries
INFO 2021-05-12 15:17:55,998Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.getNosVersionForCE:227 getNosVersionForCE(2020.09.16) - containsKey: true, ver:
5.18.0.6
INFO 2021-05-12 15:17:55,999Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.isNosVersionCompatible:181 PC is compatible with PE of version 2020.09.16
INFO 2021-05-12 15:17:56,015Z Curator-PathChildrenCache-0 e] commands.multicluster.ZkChildrenListener.childEvent:102 Cluster External State Cache Value added : 0005ad13-49c9-ed61-7f19-a4bf0
167629f
INFO 2021-05-12 15:17:56,066Z http-nio-0.0.0.0-9081-exec-4 8] multicluster.registration.AddClusterExternalState.getClusterDataStateBuilder:312 the specified node does not exist
INFO 2021-05-12 15:17:56,095Z http-nio-0.0.0.0-9081-exec-4 8] multicluster.registration.AddClusterExternalState.initClusterDataState:286 Created the cluster data state entity with entity i
d : 0005ad13-49c9-ed61-7f19-a4bf0167629f in IDf successfully
WARN 2021-05-12 15:17:56,339Z http-nio-0.0.0.0-9081-exec-3 8] commands.auth.PAMAuthenticationProvider.checkFailedAttempts:395 Failed to check the Failed Attempts..
INFO 2021-05-12 15:17:56,342Z http-nio-0.0.0.0-9081-exec-3 8] prism.proxy.LocalGenericProxy.<init>:94 Constructing proxy to Genesis service at 127.0.0.1:2100
WARN 2021-05-12 15:17:56,462Z http-nio-0.0.0.0-9081-exec-6 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
WARN 2021-05-12 15:17:56,464Z http-nio-0.0.0.0-9081-exec-1 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
INFO 2021-05-12 15:18:01,144Z pool-11-thread-1 -] prism.init.UnregistrationFixerTask.run:158 Fixing unregistration for set: ]
WARN 2021-05-12 15:18:03,625Z http-nio-0.0.0.0-9081-exec-2 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
ERROR 2021-05-12 15:18:13,936Z pool-15-thread-1 -] commands.license.LicenseManagingZkImpl.getLicenseFromLicenseFile:141 Exception while reading license file
com.nutanix.prism.base.zk.ProtobufZNodeManagementException$NoNodeException: the specified node does not exist
at com.nutanix.prism.base.zk.ProtobufManagingZookeeperImpl.read(ProtobufManagingZookeeperImpl.java:77)
at com.nutanix.prism.commands.license.LicenseManagingZkImpl.getLicenseFromLicenseFile(LicenseManagingZkImpl.java:139)
at com.nutanix.prism.background.multicluster.ZeusDataSender.buildClusterDataIncrement(ZeusDataSender.java:267)
at com.nutanix.prism.background.multicluster.ZeusDataSender.run(ZeusDataSender.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /appliance/logical/license/license_file
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
at com.nutanix.prism.base.zk.BasicConnectionManagingZkClient.getData(BasicConnectionManagingZkClient.java:157)
at com.nutanix.prism.base.zk.ProtobufManagingZookeeperImpl.read(ProtobufManagingZookeeperImpl.java:65)
... 6 more


WARN 2021-05-12 15:17:55,990Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.PortalCompatibilityMatrix.isNosVersionCompatible:72 falling back to zkNode compatibility for pc version 2
020.09.16 and software versions from portal null
WARN 2021-05-12 15:17:55,991Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.ZkNodeCompatibilityMatrix.isNosVersionCompatible:61 falling back to static compatibility for pc version 2
020.09.16 and software versions from zknode at path prism_central
INFO 2021-05-12 15:17:55,995Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.getCeVersionMap:239 Got CE version map with 53 entries
INFO 2021-05-12 15:17:55,998Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.getNosVersionForCE:227 getNosVersionForCE(2020.09.16) - containsKey: true, ver:
5.18.0.6
INFO 2021-05-12 15:17:55,999Z http-nio-0.0.0.0-9081-exec-4 8] commands.multicluster.StaticCompatibilityMatrix.isNosVersionCompatible:181 PC is compatible with PE of version 2020.09.16
INFO 2021-05-12 15:17:56,015Z Curator-PathChildrenCache-0 e] commands.multicluster.ZkChildrenListener.childEvent:102 Cluster External State Cache Value added : 0005ad13-49c9-ed61-7f19-a4bf0
167629f
INFO 2021-05-12 15:17:56,066Z http-nio-0.0.0.0-9081-exec-4 8] multicluster.registration.AddClusterExternalState.getClusterDataStateBuilder:312 the specified node does not exist
INFO 2021-05-12 15:17:56,095Z http-nio-0.0.0.0-9081-exec-4 8] multicluster.registration.AddClusterExternalState.initClusterDataState:286 Created the cluster data state entity with entity i
d : 0005ad13-49c9-ed61-7f19-a4bf0167629f in IDf successfully
WARN 2021-05-12 15:17:56,339Z http-nio-0.0.0.0-9081-exec-3 8] commands.auth.PAMAuthenticationProvider.checkFailedAttempts:395 Failed to check the Failed Attempts..
INFO 2021-05-12 15:17:56,342Z http-nio-0.0.0.0-9081-exec-3 8] prism.proxy.LocalGenericProxy.<init>:94 Constructing proxy to Genesis service at 127.0.0.1:2100
WARN 2021-05-12 15:17:56,462Z http-nio-0.0.0.0-9081-exec-6 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
WARN 2021-05-12 15:17:56,464Z http-nio-0.0.0.0-9081-exec-1 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
INFO 2021-05-12 15:18:01,144Z pool-11-thread-1 -] prism.init.UnregistrationFixerTask.run:158 Fixing unregistration for set: ]
WARN 2021-05-12 15:18:03,625Z http-nio-0.0.0.0-9081-exec-2 8] util.common.ArithmosIDFServiceUtil.getMappedFilteredCriteria:130 Entity type node
ERROR 2021-05-12 15:18:13,936Z pool-15-thread-1 -] commands.license.LicenseManagingZkImpl.getLicenseFromLicenseFile:141 Exception while reading license file
com.nutanix.prism.base.zk.ProtobufZNodeManagementException$NoNodeException: the specified node does not exist
at com.nutanix.prism.base.zk.ProtobufManagingZookeeperImpl.read(ProtobufManagingZookeeperImpl.java:77)
at com.nutanix.prism.commands.license.LicenseManagingZkImpl.getLicenseFromLicenseFile(LicenseManagingZkImpl.java:139)
at com.nutanix.prism.background.multicluster.ZeusDataSender.buildClusterDataIncrement(ZeusDataSender.java:267)
at com.nutanix.prism.background.multicluster.ZeusDataSender.run(ZeusDataSender.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /appliance/logical/license/license_file
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
at com.nutanix.prism.base.zk.BasicConnectionManagingZkClient.getData(BasicConnectionManagingZkClient.java:157)
at com.nutanix.prism.base.zk.ProtobufManagingZookeeperImpl.read(ProtobufManagingZookeeperImpl.java:65)
... 6 more


INFO 2021-05-12 15:19:01,144Z pool-11-thread-1 -] prism.init.UnregistrationFixerTask.run:158 Fixing unregistration for set: ]
WARN 2021-05-12 15:19:59,285Z http-nio-0.0.0.0-9081-exec-2 8] commands.auth.PAMAuthenticationProvider.checkFailedAttempts:395 Failed to check the Failed Attempts..
INFO 2021-05-12 15:19:59,290Z http-nio-0.0.0.0-9081-exec-2 8] commands.multicluster.DeleteClusterExternalState.doExecute:99 Unregistering Prism Element from Multicluster
INFO 2021-05-12 15:19:59,291Z http-nio-0.0.0.0-9081-exec-2 8] commands.multicluster.DeleteClusterExternalState.unregisterPrismElementFromMulticluster:122 Scheduling reap on demand.
INFO 2021-05-12 15:19:59,292Z pool-299-thread-1 -] prism.init.UnregistrationFixerTask.run:155 Invoked thread for specific cluster 0005ad13-49c9-ed61-7f19-a4bf0167629f
INFO 2021-05-12 15:19:59,293Z pool-299-thread-1 -] prism.init.UnregistrationFixerTask.deleteTempUser:319 com.nutanix.prism.init.UnregistrationFixerTask updating the /userrepository zk-node.
INFO 2021-05-12 15:19:59,336Z pool-299-thread-1 -] prism.proxy.AlertProxyImpl.log:59 AlertProxyImpl manageAlerts request {"operation": "kDelete","get_entities_with_metrics_arg": {"query": {
"where_clause": {"comparison_expr": {"lhs": {"leaf": {"column": "cluster"}},"operator": "kAny","rhs": {"leaf": {"value": {"str_list": {"value_list": u"0005ad13-49c9-ed61-7f19-a4bf0167629f"]}
}}}}},"group_by": {"raw_limit": {"limit": 100000}}}}}
INFO 2021-05-12 15:19:59,658Z pool-299-thread-1 -] prism.proxy.AlertProxyImpl.manageAlerts:177 AlertProxyImpl manageAlerts response num_successful_updates: 93

 

Does anyone have an idea how to fix this or where to look for further info?

 

Best Regards

Nick

If solution presented in knowledge base doesn’t help I doubt anything else will work. Maybe try to create new PC(Prism Central). Something might go wrong during upgrade.


Thanks, we actually noticed that a remote connection wasn’t possible, because we had blocking long running tasks running in Nutanix (cerberus uncharge tasks). They prevented the ‘create remote connection’ tasks from being started. After we killed all those cerberus tasks, a remote connection was possible without a problem. So all the thrown exceptions which one can see in the prism_gateway.log file are unrelated.


Great that you found the solution, can you mark  your reply as an answer so people with similar issues can find the solution faster? Thank you.