2021-04-30 16:15:01,154 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1619824498890L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:58.890Z', u'text': u'Rebalance exited with reason {mover_crashed,\n                              {unexpected_exit,\n                               {\'EXIT\',<0.30539.512>,\n                                {{wait_seqno_persisted_failed,\n                                  "G6wxjcA68eaiUDSts-14-823000",502,66,\n                                  [{\'ns_1@172.23.105.217\',\n                                    {\'EXIT\',\n                                     {socket_closed,\n                                      {gen_server,call,\n                                       [{\'janitor_agent-G6wxjcA68eaiUDSts-14-823000\',\n                                         \'ns_1@172.23.105.217\'},\n                                        {if_rebalance,<0.12357.512>,\n                                         {wait_seqno_persisted,502,66}},\n                                        infinity]}}}}]},\n                                 [{ns_single_vbucket_mover,\n                                   \'-wait_seqno_persisted_many/5-fun-2-\',5,\n                                   [{file,"src/ns_single_vbucket_mover.erl"},\n                                    {line,482}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 7ad39a17613690e215a27dd273e92b94'}
2021-04-30 16:15:01,154 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1619824498841L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:58.841Z', u'text': u'Worker <0.30224.512> (for action {move,\n                                     {502,\n                                      [\'ns_1@172.23.106.237\'],\n                                      [\'ns_1@172.23.105.217\'],\n                                      []}}) exited with reason {unexpected_exit,\n                                                                {\'EXIT\',\n                                                                 <0.30539.512>,\n                                                                 {{wait_seqno_persisted_failed,\n                                                                   "G6wxjcA68eaiUDSts-14-823000",\n                                                                   502,66,\n                                                                   [{\'ns_1@172.23.105.217\',\n                                                                     {\'EXIT\',\n                                                                      {socket_closed,\n                                                                       {gen_server,\n                                                                        call,\n                                                                        [{\'janitor_agent-G6wxjcA68eaiUDSts-14-823000\',\n                                                                          \'ns_1@172.23.105.217\'},\n                                                                         {if_rebalance,\n                                                                          <0.12357.512>,\n                                                                          {wait_seqno_persisted,\n                                                                           502,\n                                                                           66}},\n                                                                         infinity]}}}}]},\n                                                                  [{ns_single_vbucket_mover,\n                                                                    \'-wait_seqno_persisted_many/5-fun-2-\',\n                                                                    5,\n                                                                    [{file,\n                                                                      "src/ns_single_vbucket_mover.erl"},\n                                                                     {line,\n                                                                      482}]},\n                                                                   {proc_lib,\n                                                                    init_p,3,\n                                                                    [{file,\n                                                                      "proc_lib.erl"},\n                                                                     {line,\n                                                                      234}]}]}}}'}
2021-04-30 16:15:01,155 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1619824498818L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:58.818Z', u'text': u"Control connection to memcached on 'ns_1@172.23.106.237' disconnected. Check logs for details."}
2021-04-30 16:15:01,155 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1619824497798L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:57.798Z', u'text': u'Bucket "G6wxjcA68eaiUDSts-14-823000" rebalance does not seem to be swap rebalance'}
2021-04-30 16:15:01,155 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1619824497615L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:57.615Z', u'text': u'Started rebalancing bucket G6wxjcA68eaiUDSts-14-823000'}
2021-04-30 16:15:01,155 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1619824497539L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:57.539Z', u'text': u'Shutting down bucket "FGWvi_C_LfdLKvV1-14-771000" on \'ns_1@172.23.106.237\' for deletion'}
2021-04-30 16:15:01,155 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.90', u'tstamp': 1619824497534L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:57.534Z', u'text': u'Shutting down bucket "FGWvi_C_LfdLKvV1-14-771000" on \'ns_1@172.23.107.90\' for deletion'}
2021-04-30 16:15:01,157 | test  | ERROR   | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1619824480682L, u'shortText': u'message', u'serverTime': u'2021-04-30T16:14:40.682Z', u'text': u'Bucket "FGWvi_C_LfdLKvV1-14-771000" rebalance does not seem to be swap rebalance'}
2021-04-30 16:15:01,157 | test  | ERROR   | pool-2-thread-10 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'30af1db075ca265d80ff3eb09e53437c', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e6f82f8b77d4d8ec3926e9e2dd3c9a5f', u'status': u'notRunning'} - rebalance failed