Retry of failed swap rebalance fails with "mover_crashed,\{unexpected_exit,\{\'EXIT\',<0.4022.5>,\{{dcp_wait_for_data_move_failed,"

./testrunner -i /tmp/testexec.29103.ini -p get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=1,num-buckets=1,num-swap=2,percentage_progress=40,GROUP=P0

 {u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n                              {unexpected_exit,\n                               {\'EXIT\',<0.4022.5>,\n                                {{dcp_wait_for_data_move_failed,"default",\n                                  745,\'ns_1@172.23.104.224\',\n                                  [\'ns_1@172.23.104.228\',\n                                   \'ns_1@172.23.104.222\'],\n                                  {error,no_stats_for_this_vbucket}},\n                                 [{ns_single_vbucket_mover,\n                                   \'-wait_dcp_data_move/5-fun-0-\',5,\n                                   [{file,"src/ns_single_vbucket_mover.erl"},\n                                    {line,465}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 870eb853c466fc79804132621f399cfb', u'shortText': u'message', u'serverTime': u'2019-07-20T05:23:21.187Z', u'module': u'ns_orchestrator', u'tstamp': 1563625401187, u'type': u'critical'}

2019-07-20 05:23:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Worker <0.3833.5> (for action {move,{745,\n                                     [\'ns_1@172.23.104.224\',\n                                      \'ns_1@172.23.104.222\'],\n                                     [\'ns_1@172.23.104.228\',\n                                      \'ns_1@172.23.104.222\'],\n                                     []}}) exited with reason {unexpected_exit,\n                                                               {\'EXIT\',\n                                                                <0.4022.5>,\n                                                                {{dcp_wait_for_data_move_failed,\n                                                                  "default",\n                                                                  745,\n                                                                  \'ns_1@172.23.104.224\',\n                                                                  [\'ns_1@172.23.104.228\',\n                                                                   \'ns_1@172.23.104.222\'],\n                                                                  {error,\n                                                                   no_stats_for_this_vbucket}},\n                                                                 [{ns_single_vbucket_mover,\n                                                                   \'-wait_dcp_data_move/5-fun-0-\',\n                                                                   5,\n                                                                   [{file,\n                                                                     "src/ns_single_vbucket_mover.erl"},\n                                                                    {line,\n                                                                     465}]},\n                                                                  {proc_lib,\n                                                                   init_p,3,\n                                                                   [{file,\n                                                                     "proc_lib.erl"},\n                                                                    {line,\n                                                                     232}]}]}}}', u'shortText': u'message', u'serverTime': u'2019-07-20T05:23:21.181Z', u'module': u'ns_vbucket_mover', u'tstamp': 1563625401181, u'type': u'critical'}

./testrunner -i /tmp/testexec.29103.ini -p get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=2,num-buckets=3,num-swap=1,percentage_progress=60,GROUP=P2

{u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n                              {unexpected_exit,\n                               {\'EXIT\',<0.98.18>,\n                                {{dcp_wait_for_data_move_failed,"bucket-1",\n                                  981,\'ns_1@172.23.104.224\',\n                                  [\'ns_1@172.23.104.226\',\n                                   \'ns_1@172.23.104.222\'],\n                                  {error,no_stats_for_this_vbucket}},\n                                 [{ns_single_vbucket_mover,\n                                   \'-wait_dcp_data_move/5-fun-0-\',5,\n                                   [{file,"src/ns_single_vbucket_mover.erl"},\n                                    {line,465}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 8d125b7ab5d729340d579fe7a770d1a5', u'shortText': u'message', u'serverTime': u'2019-07-20T05:41:30.001Z', u'module': u'ns_orchestrator', u'tstamp': 1563626490001, u'type': u'critical'}

[2019-07-20 05:41:41,722] - [rest_client:3250] ERROR - {u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Worker <0.32572.17> (for action {move,{981,\n                                       [\'ns_1@172.23.104.224\',\n                                        \'ns_1@172.23.104.223\',\n                                        \'ns_1@172.23.104.222\'],\n                                       [\'ns_1@172.23.104.224\',\n                                        \'ns_1@172.23.104.226\',\n                                        \'ns_1@172.23.104.222\'],\n                                       []}}) exited with reason {unexpected_exit,\n                                                                 {\'EXIT\',\n                                                                  <0.98.18>,\n                                                                  {{dcp_wait_for_data_move_failed,\n                                                                    "bucket-1",\n                                                                    981,\n                                                                    \'ns_1@172.23.104.224\',\n                                                                    [\'ns_1@172.23.104.226\',\n                                                                     \'ns_1@172.23.104.222\'],\n                                                                    {error,\n                                                                     no_stats_for_this_vbucket}},\n                                                                   [{ns_single_vbucket_mover,\n                                                                     \'-wait_dcp_data_move/5-fun-0-\',\n                                                                     5,\n                                                                     [{file,\n                                                                       "src/ns_single_vbucket_mover.erl"},\n                                                                      {line,\n                                                                       465}]},\n                                                                    {proc_lib,\n                                                                     init_p,3,\n                                                                     [{file,\n                                                                       "proc_lib.erl"},\n                                                                      {line,\n                                                                       232}]}]}}}', u'shortText': u'message', u'serverTime': u'2019-07-20T05:41:29.994Z', u'module': u'ns_vbucket_mover', u'tstamp': 1563626489994, u'type': u'critical'}

./testrunner -i /tmp/testexec.29103.ini -p get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=1,num-buckets=4,num-swap=2,swap-orchestrator=True,percentage_progress=30,GROUP=P0

{u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n                              {unexpected_exit,\n                               {\'EXIT\',<0.1190.33>,\n                                {{dcp_wait_for_data_move_failed,"bucket-2",\n                                  832,\'ns_1@172.23.104.224\',\n                                  [\'ns_1@172.23.104.228\',\n                                   \'ns_1@172.23.104.226\'],\n                                  {error,no_stats_for_this_vbucket}},\n                                 [{ns_single_vbucket_mover,\n                                   \'-wait_dcp_data_move/5-fun-0-\',5,\n                                   [{file,"src/ns_single_vbucket_mover.erl"},\n                                    {line,465}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = deef0995000686fb41b9efeddf81c9d0', u'shortText': u'message', u'serverTime': u'2019-07-20T06:01:32.466Z', u'module': u'ns_orchestrator', u'tstamp': 1563627692466, u'type': u'critical'}

[2019-07-20 06:01:35,822] - [rest_client:3250] ERROR - {u'node': u'ns_1@172.23.104.222', u'code': 0, u'text': u'Worker <0.1326.33> (for action {move,{832,\n                                      [\'ns_1@172.23.104.224\',\n                                       \'ns_1@172.23.104.222\'],\n                                      [\'ns_1@172.23.104.228\',\n                                       \'ns_1@172.23.104.226\'],\n                                      []}}) exited with reason {unexpected_exit,\n                                                                {\'EXIT\',\n                                                                 <0.1190.33>,\n                                                                 {{dcp_wait_for_data_move_failed,\n                                                                   "bucket-2",\n                                                                   832,\n                                                                   \'ns_1@172.23.104.224\',\n                                                                   [\'ns_1@172.23.104.228\',\n                                                                    \'ns_1@172.23.104.226\'],\n                                                                   {error,\n                                                                    no_stats_for_this_vbucket}},\n                                                                  [{ns_single_vbucket_mover,\n                                                                    \'-wait_dcp_data_move/5-fun-0-\',\n                                                                    5,\n                                                                    [{file,\n                                                                      "src/ns_single_vbucket_mover.erl"},\n                                                                     {line,\n                                                                      465}]},\n                                                                   {proc_lib,\n                                                                    init_p,3,\n                                                                    [{file,\n                                                                      "proc_lib.erl"},\n                                                                     {line,\n                                                                      232}]}]}}}', u'shortText': u'message', u'serverTime': u'2019-07-20T06:01:32.460Z', u'module': u'ns_vbucket_mover', u'tstamp': 1563627692460, u'type': u'critical'}