Question

X-RAY Rolling Upgrade Failed on AOS 5.10.01

  • 11 January 2019
  • 3 replies
  • 7955 views

Userlevel 3
Badge +6
I tried to exute Rolling Upgrade test from X-RAY which was failed .

Does anyone know how to do ?

i saw this messages
code:
        Failed to issue Genesis RPC. Retries exhausted (kClusterApiError) 



this is error log

2019-01-10 13:18:28 INFO workload.py:390 FIO OLTP: IO started
2019-01-10 13:18:28 INFO _base_step.py:98 OLTP: Starting workload succeeded after 7 seconds
2019-01-10 13:18:28 INFO _base_step.py:101 Rolling Upgrade (EXECUTING): Finished workload.Start (RUN)
2019-01-10 13:18:28 INFO _base_step.py:74 Rolling Upgrade (EXECUTING): Starting test.Wait (RUN)
2019-01-10 13:23:29 INFO _base_step.py:98 Waiting for 300 seconds succeeded after 300 seconds
2019-01-10 13:23:29 INFO _base_step.py:101 Rolling Upgrade (EXECUTING): Finished test.Wait (RUN)
2019-01-10 13:23:29 INFO _base_step.py:74 Rolling Upgrade (EXECUTING): Starting RollingUpgrade.vm_group.MigrateGroup (RUN)
2019-01-10 13:23:29 INFO acropolis_cluster.py:543 Migrating VMS
2019-01-10 13:23:29 INFO task.py:1025 Polling 1 active tasks... (1 / 1 tasks remaining)
2019-01-10 13:23:34 INFO task.py:1025 Polling 1 active tasks... (1 / 1 tasks remaining)
2019-01-10 13:23:39 INFO task.py:1025 Polling 1 active tasks... (1 / 1 tasks remaining)
2019-01-10 13:23:45 INFO task.py:1036 Completed Task (6a1f15d1-d0f8-413e-9ff4-42c81d421fc2): [Terminal status: kSucceeded] (100%)
2019-01-10 13:23:45 INFO _base_step.py:98 Rolling upgrade - OLTP: Migrating VMs from node '0' to node '1' succeeded after 16 seconds
2019-01-10 13:23:45 INFO _base_step.py:101 Rolling Upgrade (EXECUTING): Finished RollingUpgrade.vm_group.MigrateGroup (RUN)
2019-01-10 13:23:45 INFO _base_step.py:74 Rolling Upgrade (EXECUTING): Starting RollingUpgrade.nodes.Shutdown (RUN)
2019-01-10 13:23:45 INFO _base_step.py:116 Annotation (RollingUpgrade.nodes.Shutdown (RUN)): Node 0: Shutting down
2019-01-10 13:23:48 INFO acropolis_cluster.py:241 Requesting genesis shutdown for node 'a77e0934-0649-4df6-8195-367bff85e8b9'
2019-01-10 13:23:49 INFO acropolis_cluster.py:262 Successfully powered off nodes a77e0934-0649-4df6-8195-367bff85e8b9
2019-01-10 13:23:49 INFO acropolis_cluster.py:264 Clearing genesis shutdown token
2019-01-10 13:23:49 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:23:49 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:23:49 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (10 retries remaining)
2019-01-10 13:23:54 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:23:54 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:23:54 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (9 retries remaining)
2019-01-10 13:23:59 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:23:59 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:23:59 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (8 retries remaining)
2019-01-10 13:24:04 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:04 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:04 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (7 retries remaining)
2019-01-10 13:24:09 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:09 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:09 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (6 retries remaining)
2019-01-10 13:24:14 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:14 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:14 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (5 retries remaining)
2019-01-10 13:24:19 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:19 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:19 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (4 retries remaining)
2019-01-10 13:24:24 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:24 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:24 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (3 retries remaining)
2019-01-10 13:24:29 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:29 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:29 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (2 retries remaining)
2019-01-10 13:24:35 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:35 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:35 INFO nutanix_rest_api_client.py:2014 Retrying Genesis request after 5 seconds (1 retries remaining)
2019-01-10 13:24:40 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced error: Caught exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/home/hudsonb/workspace/workspace/OSLAB_euphrates-5.10-stable-release/builds/build-euphrates-5.10-stable-release/python-tree/bdist.linux-x86_64/egg/util/net/rpc.py", line 316, in handle_jsonrpc
TypeError: clear_shutdown_token() takes exactly 2 arguments (1 given)

2019-01-10 13:24:40 ERROR nutanix_rest_api_client.py:2008 Issuing Genesis request produced exception: clear_shutdown_token() takes exactly 2 arguments (1 given)
2019-01-10 13:24:40 ERROR _base_step.py:92 Rolling upgrade - Shutting down node 0 failed after 54 seconds
2019-01-10 13:24:40 INFO _base_step.py:101 Rolling Upgrade (EXECUTING): Finished RollingUpgrade.nodes.Shutdown (RUN)
2019-01-10 13:24:40 ERROR scenario.py:845 Exception during 'RollingUpgrade.nodes.Shutdown (RUN)'
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/curie/scenario.py", line 835, in _execute_step
File "build/bdist.linux-x86_64/egg/curie/steps/_base_step.py", line 79, in __call__
File "build/bdist.linux-x86_64/egg/curie/steps/nodes.py", line 355, in _run
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
raise self._value
CurieException: Failed to issue Genesis RPC. Retries exhausted (kClusterApiError)
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.MigrateGroup (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.Shutdown (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'RollingUpgrade.vm_group.ResetGroupPlacement (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'test.Wait (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'workload.Stop (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.PowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.nodes.WaitForPowerOn (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.cluster.PowerOffCurieVMs (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp.cluster.DeleteCurieVMs (RUN)'
2019-01-10 13:24:40 INFO scenario.py:819 Rolling Upgrade (FAILING) failed: Skipping 'CleanUp._base_step.GenericStep (RUN)'
2019-01-10 13:24:40 INFO scenario.py:749 Rolling Upgrade (FAILING): All steps finished
2019-01-10 13:24:42 INFO scenario.py:512 Rolling Upgrade (FAILED) has finished

3 replies

Badge +5
Hi there..
We are looking into this and someone will be in touch with you.
Thanks,
Kilol
Badge +4
is all the nodes are up and genesis is running on all nodes ?
Userlevel 3
Badge +6
thanks .

Now I switched the cluster . and I'll check this again .

kashi5409

Reply