Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.0.0
-
Untriaged
-
Yes
Description
we are seeing rebalance and failover issues in centos longevity against 5.0.0-2635:
rebalance error:
2017-04-15T19:21:06.523-07:00, ns_orchestrator:4:info:message(ns_1@172.23.108.103) - Starting rebalance, KeepNodes = ['ns_1@172.23.108.100','ns_1@172.23.108.103',
|
'ns_1@172.23.108.104','ns_1@172.23.108.107',
|
'ns_1@172.23.108.108','ns_1@172.23.108.96',
|
'ns_1@172.23.108.97','ns_1@172.23.98.135'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
|
|
|
2017-04-15T19:21:16.171-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket WAREHOUSE
|
2017-04-15T19:21:17.499-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "WAREHOUSE" rebalance does not seem to be swap rebalance
|
2017-04-15T19:22:55.686-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket STOCK
|
2017-04-15T19:22:57.187-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "STOCK" rebalance does not seem to be swap rebalance
|
2017-04-15T19:24:53.420-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket ORDER_LINE
|
2017-04-15T19:24:54.848-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDER_LINE" rebalance does not seem to be swap rebalance
|
2017-04-15T19:31:22.146-07:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.108.103) - <0.25040.828> exited with {unexpected_exit,
|
{'EXIT',<0.8745.828>,
|
{{error,
|
{badrpc,
|
{'EXIT',
|
{{{function_clause,
|
[{couch_set_view_group,handle_info,
|
[{'DOWN',#Ref<11658.0.1529.247523>,
|
process,<11658.7273.70>,normal},
|
{state,
|
{"/data",<<"ORDER_LINE">>,
|
{set_view_group,
|
<<3,159,50,99,49,191,105,13,93,231,72,
|
135,190,43,14,54>>,
|
nil,<<"ORDER_LINE">>,
|
<<"_design/all">>,[],
|
[{set_view,0,
|
<<"function(doc, meta){ emit(meta.id, null) }">>,
|
undefined,
|
{mapreduce_view,
|
[<<"all_ids">>,<<"all_ids">>],
|
nil,[],[]}}],
|
nil,nil,
|
{set_view_index_header,2,0,0,0,0,[],
|
nil,[],false,[],nil,[],[]},
|
main,nil,nil,nil,[],mapreduce_view,
|
".view",prod,
|
couch_set_view_stats_prod,0,nil}},
|
<11658.7109.70>,
|
{set_view_group,
|
<<3,159,50,99,49,191,105,13,93,231,72,
|
135,190,43,14,54>>,
|
<11658.7118.70>,<<"ORDER_LINE">>,
|
<<"_design/all">>,[],
|
[{set_view,0,
|
<<"function(doc, meta){ emit(meta.id, null) }">>,
|
#Ref<11658.0.1529.245720>,
|
{mapreduce_view,
|
[<<"all_ids">>,<<"all_ids">>],
|
{btree,<11658.7118.70>,nil,
|
identity,identity,
|
#Fun<mapreduce_view.15.13640716>,
|
#Fun<mapreduce_view.14.13640716>,
|
7168,6144,true},
|
[],[]}}],
|
{btree,<11658.7118.70>,nil,identity,
|
identity,
|
#Fun<couch_btree.1.130566675>,
|
#Fun<couch_set_view_group.15.104038255>,
|
7168,6144,true},
|
<11658.7086.70>,
|
{set_view_index_header,2,1024,
|
168533731393342116349622361636488795399606305338124734027194576967524690986977013806072176642666772335449621012819375162357424401014038658735986853007277799171343098901488449123649515046721694468205215371744086886521896713715374661428212297977341041960960163152695255950566132401149149375074031230513305354239,
|
2063587534632962122825040546936929183698066512611579169698582715115192385324979242133883264502389797657107818215376893928162799177434980511556674326335804595536840733837532948608020024264271878479056959116858960442390256293844035088600844285714564827100361666712305664,
|
6091210874506492715269802028805559407769805717365771269471964969056616380334129771010369329195892880376875152503960123906210233706576781246464,
|
[{0,0},
|
{1,0},
|
{2,0},
|
{3,0},
|
{4,0},
|
{5,0},
|
{6,0},
|
{7,0},
|
{8,0},
|
{9,0},
|
{10,0},
|
{11,0},
|
{12,0},
|
{13,0},
|
{14,0},
|
{15,0},
|
{16,0},
|
{17,0},
|
{18,0},
|
{19,0},
|
{20,0},
|
{21,0},
|
{22,0},
|
{23,0},
|
{24,0},
|
{25,0},
|
{26,0},
|
{27,0},
|
{28,0},
|
{29,0},
|
{30,0},
|
{31,0},
|
{32,0},
|
{33,0},
|
{35,0},
|
{36,0},
|
{37,0},
|
{38,0},
|
{39,0},
|
{40,0},
|
{41,0},
|
{42,0},
|
{43,0},
|
{44,0},
|
{45,0},
|
{46,0},
|
{47,0},
|
{48,0},
|
{49,0},
|
{50,0},
|
{51,0},
|
{52,0},
|
{53,0},
|
{54,0},
|
{55,0},
|
{56,0},
|
{57,0},
|
{58,0},
|
{59,0},
|
{60,0},
|
{61,0},
|
{62,0},
|
{63,0},
|
{64,0},
|
{65,0},
|
{66,0},
|
{67,0},
|
{68,0},
|
{103,0},
|
{104,0},
|
{105,0},
|
{106,0},
|
{107,0},
|
{108,0},
|
{109,0},
|
{110,0},
|
{111,0},
|
{112,0},
|
{113,0},
|
{114,0},
|
{115,0},
|
{116,0},
|
{117,0},
|
{118,0},
|
{119,0},
|
{153,0},
|
{154,0},
|
{155,0},
|
{156,0},
|
{157,0},
|
{158,0},
|
{159,0},
|
{160,0},
|
{161,0},
|
{162,0},
|
{163,0},
|
{164,0},
|
{165,0},
|
{166,0},
|
{167,0},
|
{168,0},
|
{169,0},
|
{170,0},
|
{172,0},
|
{189,0},
|
{190,0},
|
{191,0},
|
{192,0},
|
{193,0},
|
{194,0},
|
{195,0},
|
{196,0},
|
{197,0},
|
{198,0},
|
{199,0},
|
{200,0},
|
{201,0},
|
{202,0},
|
{203,0},
|
{207,0},
|
{873,0},
|
{874,0},
|
{875,0},
|
{876,0},
|
{877,0},
|
{878,0},
|
{879,0},
|
{880,0},
|
{881,0},
|
{882,0},
|
{883,0},
|
{884,0},
|
{885,0},
|
{886,0},
|
{887,0},
|
{921,0},
|
{1020,0},
|
{1021,0},
|
{1022,0},
|
{1023,0}],
|
nil,
|
[nil],
|
true,
|
[1023],
|
nil,
|
[{224,0},
|
{225,0},
|
{226,0},
|
{227,0},
|
{228,0},
|
{229,0},
|
{230,0},
|
{231,0},
|
{232,0},
|
{233,0},
|
{234,0},
|
{235,0},
|
{236,0},
|
{237,0},
|
{238,0},
|
{239,0},
|
{240,0},
|
{241,0},
|
failover issue:
2017-04-15T23:15:31.364-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Starting vbucket moves for graceful failover of 'ns_1@172.23.108.107'
|
2017-04-15T23:15:31.432-07:00, ns_orchestrator:3:info:message(ns_1@172.23.108.103) - Not rebalancing because rebalance is already in progress.
|
|
|
2017-04-15T23:15:31.633-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "WAREHOUSE" rebalance does not seem to be swap rebalance
|
2017-04-15T23:16:03.286-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "STOCK" rebalance does not seem to be swap rebalance
|
2017-04-15T23:16:47.899-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "ORDER_LINE" rebalance does not seem to be swap rebalance
|
2017-04-15T23:19:47.966-07:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.108.103) - <0.25864.910> exited with {unexpected_exit,
|
{'EXIT',<0.20729.910>,
|
{{error,
|
{badrpc,
|
{'EXIT',
|
{noproc,
|
{gen_server,call,
|
['capi_set_view_manager-ORDER_LINE',
|
{set_vbucket_states,
|
[active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,replica,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,missing,active,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,missing,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,missing,missing,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
pending,pending,pending,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,active,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
pending,pending,pending,pending,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
replica,replica,replica,replica,replica,
|
missing,missing,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,missing,
|
missing,missing,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,replica,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,replica,
|
replica,replica,missing,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,replica,replica,replica,replica,
|
replica,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,replica,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,active,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,active,active,
|
active,active,active,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,active,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
missing,missing,missing,missing,missing,
|
Please take a look - test is still running as it is 7 days longevity
Attachments
Issue Links
| For Gerrit Dashboard: MB-23946 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 77119,2 | MB-23946 Toy manifest to check rebalance failures | master | manifest | Status: MERGED | +2 | +1 |
| 77171,2 | MB-23946 Revert flow control in couchdb on top of spock manifest | master | manifest | Status: MERGED | +2 | +1 |
| 77172,2 | MB-23946 Test rebalance falures on top of spock manifest | master | manifest | Status: MERGED | +2 | +1 |
| 77658,2 | MB-23946 Reverting 70262640 from couchdb | master | manifest | Status: MERGED | +2 | +1 |
| 77942,2 | MB-23946 Revert "MB-22317 Stop the updater for spatial view" | master | couchdb | Status: MERGED | +2 | +1 |
| 79477,3 | MB-23946 Implement handle_info for DOWN message for monitored process | master | couchdb | Status: MERGED | +2 | +1 |
| 79478,2 | MB-23946 Toy manifest to test rebalance failure in longevity tests | master | manifest | Status: MERGED | +2 | +1 |