Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
6.5.0
-
Untriaged
-
-
No
Description
Build: 6.5.0-4917, not seen on 6.5.0-4908
Test: MH longevity
Cycle: 2nd
Day: 2nd
Seeing multiple rebalance failure for eventing
[user:error,2019-12-05T15:10:18.944-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.29188.1182>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = da3ae28e304d693f84567e0bdc3cab39
|
|
|
[user:error,2019-12-05T15:35:01.557-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.23000.1196>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = 5ee66f3b1ce6892b90e0b7af66d97beb
|
|
|
|
|
[user:error,2019-12-05T16:11:57.527-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.9006.1218>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,170}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = 56a74378a8d5a9737f816839f813f608
|
Rebalance is even failing when it involves only index. We should ignore other services rebalance
[2019-12-05T15:05:15-08:00, sequoiatools/couchbase-cli:6.5:6c7db0] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.164 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
|
[2019-12-05T15:05:44-08:00, sequoiatools/couchbase-cli:6.5:580796] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
|
|
|
docker logs 580796
|
docker start 580796
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T15:10:35-08:00, sequoiatools/cmd:3ee376] 60
|
|
|
|
|
[2019-12-05T15:28:52-08:00, sequoiatools/couchbase-cli:6.5:5bfb5a] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
|
warning using 'json' filter: unexpected end of JSON input []
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
|
|
|
docker logs 5bfb5a
|
docker start 5bfb5a
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T15:35:16-08:00, sequoiatools/cmd:a23d6f] 60
|
|
|
|
|
[2019-12-05T16:07:42-08:00, sequoiatools/couchbase-cli:6.5:c0ad80] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
|
|
|
docker logs c0ad80
|
docker start c0ad80
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2019-12-05T16:12:18-08:00, sequoiatools/cmd:12bade] 60
|
|
Test not recovered after this all rebalance failing
eventing : 3 ===== > [172.23.104.87:8091 172.23.96.148:8091 172.23.98.135:8091]
Attachments
Activity
Field | Original Value | New Value |
---|---|---|
Assignee | Jeelan Poola [ jeelan.poola ] | Ankit Prabhu [ ankit.prabhu ] |
Assignee | Ankit Prabhu [ ankit.prabhu ] | Gautham Banasandra [ gautham.banasandra ] |
Is this a Regression? | Yes [ 10450 ] | No [ 10451 ] |
Labels | system-test | approved-for-mad-hatter system-test |
Due Date | 10/Dec/19 |
Link | This issue blocks MB-36676 [ MB-36676 ] |
Resolution | Fixed [ 1 ] | |
Status | Open [ 1 ] | Resolved [ 5 ] |
Status | Resolved [ 5 ] | Closed [ 6 ] |