Rebalance fails with "leader activities error"

./testrunner -i /tmp/testexec.2913.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,index_type=scorch -t fts.stable_topology_fts.StableTopFTS.test_snippets_highlighting_of_search_term_in_results,highlight_fields=name,manages.reports,query="""{"query": "Palmer"}""",expected_results=[{"doc_id":"emp10000001","field_name":"name","term":"Palmer"},{"doc_id":"emp10000009","field_name":"manages.reports","term":"Palmer"}],highlight_style=html,items=10,cluster=D+F,D+F

[2018-03-31 17:53:58,781] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes

[2018-03-31 17:53:58,787] - [rest_client:1448] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.97.223', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.22,ns_1@172.23.97.223'}

[2018-03-31 17:53:58,803] - [rest_client:1453] INFO - rebalance operation started

[2018-03-31 17:53:58,816] - [rest_client:1584] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

[2018-03-31 17:53:58,840] - [rest_client:3117] INFO - Latest logs from UI on 172.23.107.22:

[2018-03-31 17:53:58,840] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 0, u'text': u'Rebalance exited with reason {badmatch,\n                              {leader_activities_error,\n                               {default,rebalance},\n                               {domain_conflict,\n                                {activity_token,\n                                 {\'ns_1@172.23.107.22\',\n                                  <<"3c4cdef4d44d1d39371b95afd9946848">>},\n                                 <<"23d27e65dbb56d17fa08e5a4b4dbc475">>,\n                                 default,\n                                 [rebalance],\n                                 []},\n                                {activity,<0.19181.10>,#Ref<0.0.8.12268>,\n                                 default,\n                                 <<"6d5c31c5b8c151241df367f8d7f41cc1">>,\n                                 [{service_janitor,n1ql,\n                                   maybe_complete_pending_failover}],\n                                 majority,\n                                 [quiet]}}}}', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:58.801Z', u'module': u'ns_orchestrator', u'tstamp': 1522544038801, u'type': u'critical'}

[2018-03-31 17:53:58,840] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.22'], EjectNodes = ['ns_1@172.23.97.223'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:58.798Z', u'module': u'ns_orchestrator', u'tstamp': 1522544038798, u'type': u'info'}

[2018-03-31 17:53:58,840] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.107.22\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.97.223\']\n', u'shortText': u'client-side error report', u'serverTime': u'2018-03-31T17:53:58.786Z', u'module': u'menelaus_web', u'tstamp': 1522544038786, u'type': u'warning'}

[2018-03-31 17:53:58,840] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 11, u'text': u'Deleted bucket "default"\n', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:56.737Z', u'module': u'menelaus_web', u'tstamp': 1522544036737, u'type': u'info'}

[2018-03-31 17:53:58,840] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.107.22\' for deletion', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:55.497Z', u'module': u'ns_memcached', u'tstamp': 1522544035497, u'type': u'info'}

[2018-03-31 17:53:58,841] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.97.223', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.97.223\' for deletion', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:55.407Z', u'module': u'ns_memcached', u'tstamp': 1522544035407, u'type': u'info'}

[2018-03-31 17:53:58,841] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.22\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:26.911Z', u'module': u'ns_memcached', u'tstamp': 1522544006911, u'type': u'info'}

[2018-03-31 17:53:58,841] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.97.223', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.97.223\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:26.409Z', u'module': u'ns_memcached', u'tstamp': 1522544006409, u'type': u'info'}

[2018-03-31 17:53:58,841] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 12, u'text': u'Created bucket "default" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,1645215744},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', u'shortText': u'message', u'serverTime': u'2018-03-31T17:53:26.344Z', u'module': u'menelaus_web', u'tstamp': 1522544006344, u'type': u'info'}

[2018-03-31 17:53:58,841] - [rest_client:3118] ERROR - {u'node': u'ns_1@172.23.107.22', u'code': 0, u'text': u'Rebalance completed successfully.', u'shortText': u'message', u'serverTime': u'2018-03-31T17:52:57.017Z', u'module': u'ns_orchestrator', u'tstamp': 1522543977017, u'type': u'info'}

ERROR

======================================================================

ERROR: test_snippets_highlighting_of_search_term_in_results (fts.stable_topology_fts.StableTopFTS)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "pytests/fts/stable_topology_fts.py", line 20, in tearDown

    super(StableTopFTS, self).tearDown()

  File "pytests/fts/fts_base.py", line 2923, in tearDown

    self._cb_cluster.cleanup_cluster(self)

  File "pytests/fts/fts_base.py", line 1759, in cleanup_cluster

    ClusterOperationHelper.cleanup_cluster([node])

  File "lib/membase/helper/cluster_helper.py", line 259, in cleanup_cluster

    wait_for_rebalance=wait_for_rebalance)

  File "lib/membase/api/rest_client.py", line 112, in remove_nodes

    return self.rest.monitorRebalance()

  File "lib/membase/api/rest_client.py", line 1510, in monitorRebalance

    progress = self._rebalance_progress()

  File "lib/membase/api/rest_client.py", line 1609, in _rebalance_progress

    return self._rebalance_status_and_progress()[1]

  File "lib/membase/api/rest_client.py", line 1586, in _rebalance_status_and_progress

    raise RebalanceFailedException(msg)

RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

----------------------------------------------------------------------