[System test]: Eventing rebalance failed because of timeout

[ns_server:error,2018-09-16T03:14:52.566-07:00,ns_1@172.23.104.16:service_rebalancer-eventing<0.30243.109>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:

[{'ns_1@172.23.104.21',nack}]

[user:error,2018-09-16T03:14:52.574-07:00,ns_1@172.23.104.16:<0.6226.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {service_rebalance_failed,eventing,

                              {rebalance_failed,

                               {service_error,

                                <<"eventing rebalance hasn't made progress for past 600 secs">>}}}

[ns_server:error,2018-09-16T03:47:36.826-07:00,ns_1@172.23.104.16:service_rebalancer-eventing<0.13387.133>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:

[{'ns_1@172.23.104.21',nack}]

[user:error,2018-09-16T03:47:36.827-07:00,ns_1@172.23.104.16:<0.6226.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {service_rebalance_failed,eventing,

                              {rebalance_failed,

                               {service_error,

                                <<"eventing rebalance hasn't made progress for past 600 secs">>}}} 

[2018-09-16T02:47:33-07:00, sequoiatools/couchbase-cli:b29353] failover -c 172.23.104.16:8091 --server-failover 172.23.104.18:8091 -u Administrator -p password --force

[2018-09-16T02:47:45-07:00, sequoiatools/couchbase-cli:9b38d0] failover -c 172.23.104.16:8091 --server-failover 172.23.104.19:8091 -u Administrator -p password --force

[2018-09-16T02:47:55-07:00, sequoiatools/couchbase-cli:84ea9d] rebalance -c 172.23.104.16:8091 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]

docker logs 84ea9d

docker start 84ea9d

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[2018-09-16T03:15:00-07:00, sequoiatools/cmd:c7f22a] 60

[2018-09-16T03:16:36-07:00, sequoiatools/couchbase-cli:7051d6] server-add -c 172.23.104.16:8091 --server-add 172.23.104.17:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data

[2018-09-16T03:16:50-07:00, sequoiatools/couchbase-cli:9b5e07] server-add -c 172.23.104.16:8091 --server-add 172.23.104.18:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data

[2018-09-16T03:17:05-07:00, sequoiatools/couchbase-cli:e93281] server-add -c 172.23.104.16:8091 --server-add 172.23.104.19:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing

[2018-09-16T03:17:20-07:00, sequoiatools/couchbase-cli:694c29] rebalance -c 172.23.104.16:8091 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]

docker logs 694c29

docker start 694c29

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[2018-09-16T03:47:45-07:00, sequoiatools/cmd:293cda] 60