Rebalance failed with badmatch

./testrunner -i centos_x64--rebalance_out.ini  -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=1,num-buckets=4,num-swap=2,swap-orchestrator=True,percentage_progress=30,GROUP=P0,get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=False,force_kill_memached=False,verify_unacked_bytes=True,total_vbuckets=128,std_vbucker_dist=8,chk_max_items=1000,chk_period=30

[2016-04-04 19:24:59,623] - [rest_client:1321] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

[2016-04-04 19:24:59,686] - [rest_client:2443] INFO - Latest logs from UI on 172.23.123.74:

[2016-04-04 19:24:59,686] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.71', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n                              {\'EXIT\',<0.29801.15>,\n                               {{{{{case_clause,\n                                    {error,\n                                     {{{badmatch,\n                                        {error,\n                                         {{badmatch,{error,closed}},\n                                          [{mc_client_binary,cmd_vocal_recv,\n                                            5,\n                                            [{file,"src/mc_client_binary.erl"},\n                                             {line,156}]},\n                                           {mc_client_binary,cmd_vocal,3,\n                                            [{file,"src/mc_client_binary.erl"},\n                                             {line,141}]},\n                                           {dcp_commands,open_connection,4,\n                                            [{file,"src/dcp_commands.erl"},\n                                             {line,60}]},\n                                           {dcp_proxy,connect,4,\n                                            [{file,"src/dcp_proxy.erl"},\n                                             {line,184}]},\n                                           {dcp_proxy,maybe_connect,1,\n                                            [{file,"src/dcp_proxy.erl"},\n                                             {line,170}]},\n                                           {dcp_producer_conn,init,2,\n                                            [{file,\n                                              "src/dcp_producer_conn.erl"},\n                                             {line,30}]},\n                                           {dcp_proxy,init,1,\n                                            [{file,"src/dcp_proxy.erl"},\n                                             {line,47}]},\n                                           {gen_server,init_it,6,\n                                            [{file,"gen_server.erl"},\n                                             {line,304}]}]}}},\n                                       [{dcp_replicator,init,1,\n                                         [{file,"src/dcp_replicator.erl"},\n                                          {line,50}]},\n                                        {gen_server,init_it,6,\n                                         [{file,"gen_server.erl"},{line,304}]},\n                                        {proc_lib,init_p_do_apply,3,\n                                         [{file,"proc_lib.erl"},{line,239}]}]},\n                                      {child,undefined,\'ns_1@172.23.123.71\',\n                                       {dcp_replicator,start_link,\n                                        [\'ns_1@172.23.123.71\',"bucket-3"]},\n                                       temporary,60000,worker,\n                                       [dcp_replicator]}}}},\n                                   [{dcp_sup,start_replicator,2,\n                                     [{file,"src/dcp_sup.erl"},{line,53}]},\n                                    {dcp_sup,\n                                     \'-manage_replicators/2-lc$^2/1-2-\',2,\n                                     [{file,"src/dcp_sup.erl"},{line,69}]},\n                                    {dcp_replication_manager,handle_call,3,\n                                     [{file,"src/dcp_replication_manager.erl"},\n                                      {line,87}]},\n                                    {gen_server,handle_msg,5,\n                                     [{file,"gen_server.erl"},{line,585}]},\n                                    {proc_lib,init_p_do_apply,3,\n                                     [{file,"proc_lib.erl"},{line,239}]}]},\n                                  {gen_server,call,\n                                   [\'dcp_replication_manager-bucket-3\',\n                                    {manage_replicators,\n                                     [\'ns_1@172.23.123.71\',\n                                      \'ns_1@172.23.123.72\',\n                                      \'ns_1@172.23.123.73\',\n                                      \'ns_1@172.23.123.74\']},\n                                    infinity]}},\n                                 {gen_server,call,\n                                  [\'replication_manager-bucket-3\',\n                                   {dcp_takeover,\'ns_1@172.23.123.72\',476},\n                                   infinity]}},\n                                {gen_server,call,\n                                 [{\'janitor_agent-bucket-3\',\n                                   \'ns_1@172.23.105.19\'},\n                                  {if_rebalance,<0.3877.15>,\n                                   {dcp_takeover,\'ns_1@172.23.123.72\',476}},\n                                  infinity]}}}}\n', u'shortText': u'message', u'serverTime': u'2016-04-04T19:24:46.307Z', u'module': u'ns_orchestrator', u'tstamp': 1459823086307, u'type': u'info'}

[2016-04-04 19:24:59,686] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.71', u'code': 0, u'text': u'<0.29701.15> exited with {unexpected_exit,\n                          {\'EXIT\',<0.29801.15>,\n                           {{{{{case_clause,\n                                {error,\n                                 {{{badmatch,\n                                    {error,\n                                     {{badmatch,{error,closed}},\n                                      [{mc_client_binary,cmd_vocal_recv,5,\n                                        [{file,"src/mc_client_binary.erl"},\n                                         {line,156}]},\n                                       {mc_client_binary,cmd_vocal,3,\n                                        [{file,"src/mc_client_binary.erl"},\n                                         {line,141}]},\n                                       {dcp_commands,open_connection,4,\n                                        [{file,"src/dcp_commands.erl"},\n                                         {line,60}]},\n                                       {dcp_proxy,connect,4,\n                                        [{file,"src/dcp_proxy.erl"},\n                                         {line,184}]},\n                                       {dcp_proxy,maybe_connect,1,\n                                        [{file,"src/dcp_proxy.erl"},\n                                         {line,170}]},\n                                       {dcp_producer_conn,init,2,\n                                        [{file,"src/dcp_producer_conn.erl"},\n                                         {line,30}]},\n                                       {dcp_proxy,init,1,\n                                        [{file,"src/dcp_proxy.erl"},\n                                         {line,47}]},\n                                       {gen_server,init_it,6,\n                                        [{file,"gen_server.erl"},\n                                         {line,304}]}]}}},\n                                   [{dcp_replicator,init,1,\n                                     [{file,"src/dcp_replicator.erl"},\n                                      {line,50}]},\n                                    {gen_server,init_it,6,\n                                     [{file,"gen_server.erl"},{line,304}]},\n                                    {proc_lib,init_p_do_apply,3,\n                                     [{file,"proc_lib.erl"},{line,239}]}]},\n                                  {child,undefined,\'ns_1@172.23.123.71\',\n                                   {dcp_replicator,start_link,\n                                    [\'ns_1@172.23.123.71\',"bucket-3"]},\n                                   temporary,60000,worker,\n                                   [dcp_replicator]}}}},\n                               [{dcp_sup,start_replicator,2,\n                                 [{file,"src/dcp_sup.erl"},{line,53}]},\n                                {dcp_sup,\'-manage_replicators/2-lc$^2/1-2-\',\n                                 2,\n                                 [{file,"src/dcp_sup.erl"},{line,69}]},\n                                {dcp_replication_manager,handle_call,3,\n                                 [{file,"src/dcp_replication_manager.erl"},\n                                  {line,87}]},\n                                {gen_server,handle_msg,5,\n                                 [{file,"gen_server.erl"},{line,585}]},\n                                {proc_lib,init_p_do_apply,3,\n                                 [{file,"proc_lib.erl"},{line,239}]}]},\n                              {gen_server,call,\n                               [\'dcp_replication_manager-bucket-3\',\n                                {manage_replicators,\n                                 [\'ns_1@172.23.123.71\',\'ns_1@172.23.123.72\',\n                                  \'ns_1@172.23.123.73\',\'ns_1@172.23.123.74\']},\n                                infinity]}},\n                             {gen_server,call,\n                              [\'replication_manager-bucket-3\',\n                               {dcp_takeover,\'ns_1@172.23.123.72\',476},\n                               infinity]}},\n                            {gen_server,call,\n                             [{\'janitor_agent-bucket-3\',\'ns_1@172.23.105.19\'},\n                              {if_rebalance,<0.3877.15>,\n                               {dcp_takeover,\'ns_1@172.23.123.72\',476}},\n                              infinity]}}}}', u'shortText': u'message', u'serverTime': u'2016-04-04T19:24:46.301Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459823086301, u'type': u'critical'}

[2016-04-04 19:24:59,686] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.105.19', u'code': 0, u'text': u'Bucket "bucket-3" loaded on node \'ns_1@172.23.105.19\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T19:24:05.411Z', u'module': u'ns_memcached', u'tstamp': 1459823045411, u'type': u'info'}

[2016-04-04 19:24:59,687] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.105.19', u'code': 3, u'text': u'Node ns_1@172.23.105.19 joined cluster', u'shortText': u'message', u'serverTime': u'2016-04-04T19:24:04.755Z', u'module': u'ns_cluster', u'tstamp': 1459823044755, u'type': u'info'}

[2016-04-04 19:24:59,687] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.105.19', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.19\'. Version: "4.5.0-2021-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2016-04-04T19:24:04.624Z', u'module': u'menelaus_sup', u'tstamp': 1459823044624, u'type': u'info'}

[2016-04-04 19:24:59,687] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.71', u'code': 0, u'text': u'Bucket "bucket-3" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-04-04T19:23:54.727Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459823034727, u'type': u'info'}

[2016-04-04 19:24:59,687] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.74', u'code': 0, u'text': u'Bucket "bucket-3" loaded on node \'ns_1@172.23.123.74\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T19:23:53.444Z', u'module': u'ns_memcached', u'tstamp': 1459823033444, u'type': u'info'}

[2016-04-04 19:24:59,687] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.71', u'code': 0, u'text': u'Started rebalancing bucket bucket-3', u'shortText': u'message', u'serverTime': u'2016-04-04T19:23:53.329Z', u'module': u'ns_rebalancer', u'tstamp': 1459823033329, u'type': u'info'}

[2016-04-04 19:24:59,688] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.71', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.123.73','ns_1@172.23.105.19',\n                                 'ns_1@172.23.123.74'], EjectNodes = ['ns_1@172.23.123.71',\n                                                                      'ns_1@172.23.123.72'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-04-04T19:23:53.200Z', u'module': u'ns_orchestrator', u'tstamp': 1459823033200, u'type': u'info'}

[2016-04-04 19:24:59,688] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.123.72', u'code': 4, u'text': u"Node 'ns_1@172.23.123.72' saw that node 'ns_1@172.23.105.19' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2016-04-04T19:23:48.772Z', u'module': u'ns_node_disco', u'tstamp': 1459823028772, u'type': u'info'}

[2016-04-04 19:25:07,690] - [bucket_helper:139] INFO - remove bucket bucket-3 ...

[2016-04-04 19:25:37,732] - [rest_client:757] ERROR - http://172.23.123.71:8091/pools/default/buckets/bucket-3 error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}

[2016-04-04 19:25:37,733] - [rest_client:1716] WARNING - Bucket deletion timed out waiting for all nodes