Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.0
-
Untriaged
-
-
No
Description
Build: 6.5.0-4917, not seen on 6.5.0-4908
Test: MH longevity
Cycle: 2nd
Day: 2nd
Seeing multiple rebalance failure for eventing
[user:error,2019-12-05T15:10:18.944-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.29188.1182>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = da3ae28e304d693f84567e0bdc3cab39
|
|
|
[user:error,2019-12-05T15:35:01.557-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.23000.1196>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = 5ee66f3b1ce6892b90e0b7af66d97beb
|
|
|
|
|
[user:error,2019-12-05T16:11:57.527-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.9006.1218>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = 56a74378a8d5a9737f816839f813f608
|
Rebalance is even failing when it involves only index. We should ignore other services rebalance
[2019-12-05T15:05:15-08:00, sequoiatools/couchbase-cli:6.5:6c7db0] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.164 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
|
[2019-12-05T15:05:44-08:00, sequoiatools/couchbase-cli:6.5:580796] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
|
|
|
docker logs 580796
|
docker start 580796
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T15:10:35-08:00, sequoiatools/cmd:3ee376] 60
|
|
|
|
|
[2019-12-05T15:28:52-08:00, sequoiatools/couchbase-cli:6.5:5bfb5a] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
|
warning using 'json' filter: unexpected end of JSON input []
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
|
|
|
docker logs 5bfb5a
|
docker start 5bfb5a
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T15:35:16-08:00, sequoiatools/cmd:a23d6f] 60
|
|
|
|
|
[2019-12-05T16:07:42-08:00, sequoiatools/couchbase-cli:6.5:c0ad80] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
|
|
|
docker logs c0ad80
|
docker start c0ad80
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T16:12:18-08:00, sequoiatools/cmd:12bade] 60
|
|
Test not recovered after this all rebalance failing
eventing : 3 ===== > [172.23.104.87:8091 172.23.96.148:8091 172.23.98.135:8091]