Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.0
-
component test
-
Untriaged
-
Yes
Description
build : 6.0.0-1643
Test Job: http://qa.sc.couchbase.com/job/component_systest_launcher/404/console
Rebalance are failing with time out
error log
[ns_server:error,2018-09-16T03:14:52.566-07:00,ns_1@172.23.104.16:service_rebalancer-eventing<0.30243.109>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:
|
[{'ns_1@172.23.104.21',nack}]
|
[user:error,2018-09-16T03:14:52.574-07:00,ns_1@172.23.104.16:<0.6226.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{rebalance_failed,
|
{service_error,
|
<<"eventing rebalance hasn't made progress for past 600 secs">>}}}
|
[ns_server:error,2018-09-16T03:47:36.826-07:00,ns_1@172.23.104.16:service_rebalancer-eventing<0.13387.133>:service_agent:process_bad_results:810]Service call unset_rebalancer (service eventing) failed on some nodes:
|
[{'ns_1@172.23.104.21',nack}]
|
[user:error,2018-09-16T03:47:36.827-07:00,ns_1@172.23.104.16:<0.6226.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{rebalance_failed,
|
{service_error,
|
<<"eventing rebalance hasn't made progress for past 600 secs">>}}}
|
Test log
[2018-09-16T02:47:33-07:00, sequoiatools/couchbase-cli:b29353] failover -c 172.23.104.16:8091 --server-failover 172.23.104.18:8091 -u Administrator -p password --force
|
[2018-09-16T02:47:45-07:00, sequoiatools/couchbase-cli:9b38d0] failover -c 172.23.104.16:8091 --server-failover 172.23.104.19:8091 -u Administrator -p password --force
|
[2018-09-16T02:47:55-07:00, sequoiatools/couchbase-cli:84ea9d] rebalance -c 172.23.104.16:8091 -u Administrator -p password
|
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
|
|
docker logs 84ea9d
|
docker start 84ea9d
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2018-09-16T03:15:00-07:00, sequoiatools/cmd:c7f22a] 60
|
[2018-09-16T03:16:36-07:00, sequoiatools/couchbase-cli:7051d6] server-add -c 172.23.104.16:8091 --server-add 172.23.104.17:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2018-09-16T03:16:50-07:00, sequoiatools/couchbase-cli:9b5e07] server-add -c 172.23.104.16:8091 --server-add 172.23.104.18:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2018-09-16T03:17:05-07:00, sequoiatools/couchbase-cli:e93281] server-add -c 172.23.104.16:8091 --server-add 172.23.104.19:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
|
[2018-09-16T03:17:20-07:00, sequoiatools/couchbase-cli:694c29] rebalance -c 172.23.104.16:8091 -u Administrator -p password
|
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
|
|
docker logs 694c29
|
docker start 694c29
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2018-09-16T03:47:45-07:00, sequoiatools/cmd:293cda] 60
|
Subsequent rebalance passed.
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.16.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.17.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.18.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.19.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.21.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.104.25.zip
https://s3.amazonaws.com/bugdb/jira/eventing_reb/collectinfo-2018-09-17T054506-ns_1%40172.23.96.96.zip