[System test]: Eventing rebalance failed due to timeout

[2019-07-08T01:22:21-07:00, sequoiatools/couchbase-cli:6.5:4b78dd] server-add -c 172.23.97.74:8091 --server-add https://172.23.96.215 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing

[2019-07-08T01:22:34-07:00, sequoiatools/couchbase-cli:6.5:7c66b7] rebalance -c 172.23.97.74:8091 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]

docker logs 7c66b7

docker start 7c66b7

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[2019-07-08T01:42:17-07:00, sequoiatools/cmd:558e1b] 60

[2019-07-08T01:43:22-07:00, sequoiatools/cmd:5feacf] 300

[2019-07-08T01:48:42-07:00, sequoiatools/couchbase-cli:6.5:b7ba5e] rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.214 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.214 -u Administrator -p password]

docker logs b7ba5e

docker start b7ba5e

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[2019-07-08T02:00:05-07:00, sequoiatools/cmd:62d857] 60

[2019-07-08T02:01:11-07:00, sequoiatools/cmd:45f38c] 300

[2019-07-08T02:06:40-07:00, sequoiatools/couchbase-cli:6.5:f0f280] server-add -c 172.23.97.74:8091 --server-add https://172.23.96.48 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing

[2019-07-08T02:06:48-07:00, sequoiatools/couchbase-cli:6.5:9fcdc7] rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.214 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.214 -u Administrator -p password]

docker logs 9fcdc7

docker start 9fcdc7

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[2019-07-08T02:17:57-07:00, sequoiatools/cmd:587f17] 60

[ns_server:error,2019-07-08T01:42:11.497-07:00,ns_1@172.23.97.74:service_rebalancer-eventing<0.25119.33>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:

[{'ns_1@172.23.96.212',nack}]

[user:error,2019-07-08T01:42:11.497-07:00,ns_1@172.23.97.74:<0.17436.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,eventing,

                              {rebalance_failed,

                               {service_error,

                                <<"eventing rebalance hasn't made progress for past 600 secs">>}}}.

Rebalance Operation Id = ffe722ea1325050aaa149568fd66b8bf

[ns_server:error,2019-07-08T01:59:59.113-07:00,ns_1@172.23.97.74:service_rebalancer-eventing<0.28820.42>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:

[{'ns_1@172.23.96.215',nack}]

[user:error,2019-07-08T01:59:59.114-07:00,ns_1@172.23.97.74:<0.17436.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,eventing,

                              {rebalance_failed,

                               {service_error,

                                <<"eventing rebalance hasn't made progress for past 600 secs">>}}}.

Rebalance Operation Id = a19d4c5a110aedf672e1c366c5b5768d

[ns_server:error,2019-07-08T02:17:51.452-07:00,ns_1@172.23.97.74:service_rebalancer-eventing<0.25611.48>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:

[{'ns_1@172.23.96.216',nack}]

[user:error,2019-07-08T02:17:51.453-07:00,ns_1@172.23.97.74:<0.17436.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,eventing,

                              {rebalance_failed,

                               {service_error,

                                <<"eventing rebalance hasn't made progress for past 600 secs">>}}}.

Rebalance Operation Id = 020ed7e3443773268c08b82dcf806457