Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
Untriaged
-
-
Yes
Description
We restarted centos longevity with 5.5.0-2017 which includes fix for MB-28356 - observing lot of memcached exits with status 134 and rebalance failures now:
Service 'memcached' exited with status 134. Restarting. Messages:
|
2018-03-01T11:51:50.044483Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f4a1da74000+0x8f221]
|
2018-03-01T11:51:50.044496Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f4a1da74000+0x8ff6f]
|
2018-03-01T11:51:50.044502Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x5d72e]
|
2018-03-01T11:51:50.044507Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x3cc51]
|
2018-03-01T11:51:50.044513Z CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8() [0x7f4a1e8c3000+0x1937c]
|
2018-03-01T11:51:50.044517Z CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8(event_base_loop+0x46f) [0x7f4a1e8c3000+0x1c7cf]
|
2018-03-01T11:51:50.044522Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x3b2b4]
|
2018-03-01T11:51:50.044526Z CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f4a1f7ba000+0x8387]
|
2018-03-01T11:51:50.044531Z CRITICAL /lib64/libpthread.so.0() [0x7f4a1f17a000+0x7dc5]
|
2018-03-01T11:51:50.044556Z CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f4a1d19b000+0xf776d] hide
|
Service 'memcached' exited with status 134. Restarting. Messages:
|
2018-03-01T12:02:02.435045Z WARNING (default) DCP (Producer) eq_dcpq:xdcr:dcp_f4268e62702130298bf87f17cc481219/default/default_172.23.97.239:11210_0:grt1Bjtc7yTjp5yVwpfndg== - (vb 800) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 23323. Client requested seqnos:{21750,18446744073709551615} snapshot:{0,40409} uuid:241399521537618
|
2018-03-01T12:02:02.443382Z WARNING (default) DCP (Producer) eq_dcpq:xdcr:dcp_f4268e62702130298bf87f17cc481219/default/default_172.23.97.239:11210_0:grt1Bjtc7yTjp5yVwpfndg== - (vb 800) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 23323. Client requested seqnos:{16841,18446744073709551615} snapshot:{0,39309} uuid:241399521537618
|
2018-03-01T12:02:02.451255Z WARNING (default) DCP (Producer) eq_dcpq:xdcr:dcp_f4268e62702130298bf87f17cc481219/default/default_172.23.97.239:11210_0:grt1Bjtc7yTjp5yVwpfndg== - (vb 800) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 23323. Client requested seqnos:{10188,18446744073709551615} snapshot:{0,39297} uuid:241399521537618 hide
|
Rebalance failures:
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.6172.206>,
|
{bulk_set_vbucket_state_failed,
|
[{'ns_1@172.23.96.56',
|
{'EXIT',
|
{{{{{case_clause,
|
{error,
|
{{{badmatch,
|
{error,
|
{{badmatch,{error,econnrefused}},
|
[{dcp_proxy,connect,5,
|
[{file,"src/dcp_proxy.erl"},
|
{line,228}]},
|
{dcp_proxy,maybe_connect,2,
|
[{file,"src/dcp_proxy.erl"},
|
{line,210}]},
|
{dcp_producer_conn,init,2,
|
[{file,
|
"src/dcp_producer_conn.erl"},
|
{line,31}]},
|
{dcp_proxy,init,1,
|
[{file,"src/dcp_proxy.erl"},
|
{line,57}]},
|
{gen_server,init_it,6,
|
[{file,"gen_server.erl"},
|
{line,304}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},
|
{line,239}]}]}}},
|
[{dcp_replicator,init,1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,50}]},
|
{gen_server,init_it,6,
|
[{file,"gen_server.erl"},
|
{line,304}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},
|
{line,239}]}]},
|
{child,undefined,
|
{'ns_1@172.23.97.239',true},
|
{dcp_replicator,start_link,
|
['ns_1@172.23.97.239',"STOCK",
|
true]},
|
temporary,60000,worker,
|
[dcp_replicator]}}}},
|
[{dcp_sup,start_replicator,2,
|
[{file,"src/dcp_sup.erl"},
|
{line,54}]},
|
{dcp_sup,
|
'-manage_replicators/3-lc$^3/1-3-',
|
2,
|
[{file,"src/dcp_sup.erl"},
|
{line,81}]},
|
{dcp_replication_manager,
|
handle_call,3,
|
[{file,
|
"src/dcp_replication_manager.erl"},
|
{line,87}]},
|
{gen_server,handle_msg,5,
|
[{file,"gen_server.erl"},
|
{line,585}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},
|
{line,239}]}]},
|
{gen_server,call,
|
['dcp_replication_manager-STOCK',
|
{manage_replicators,
|
['ns_1@172.23.108.103',
|
'ns_1@172.23.97.237',
|
'ns_1@172.23.97.238',
|
'ns_1@172.23.97.239'],
|
true},
|
infinity]}},
|
{gen_server,call,
|
['replication_manager-STOCK',
|
{change_vbucket_replication,779,
|
'ns_1@172.23.97.239'},
|
infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-STOCK',
|
'ns_1@172.23.96.56'},
|
{if_rebalance,<0.10452.205>,
|
{update_vbucket_state,779,replica,
|
undefined,'ns_1@172.23.97.239'}},
|
infinity]}}}}]}}}} hide
|
ns_orchestrator 000
|
ns_1@172.23.108.103
|
12:02:03 PM Thu Mar 1, 2018
|
<0.930.206> exited with {unexpected_exit,
|
{'EXIT',<0.6172.206>,
|
{bulk_set_vbucket_state_failed,
|
[{'ns_1@172.23.96.56',
|
{'EXIT',
|
{{{{{case_clause,
|
{error,
|
{{{badmatch,
|
{error,
|
{{badmatch,{error,econnrefused}},
|
[{dcp_proxy,connect,5,
|
[{file,"src/dcp_proxy.erl"},
|
{line,228}]},
|
{dcp_proxy,maybe_connect,2,
|
[{file,"src/dcp_proxy.erl"},
|
{line,210}]},
|
{dcp_producer_conn,init,2,
|
...show
|
Attaching logs and screenshots
Attachments
Issue Links
- is duplicated by
-
MB-28477 memcached exits with status 134 in eventing component level system tests
- Closed
-
MB-28532 memcached crashes in ubuntu longevity
- Closed
-
MB-28559 [Upgrade] Swap rebalance failed when upgrade from 4.6.1 to vulcan 2036
- Closed
- is triggering
-
MB-28501 query service exits observed in centos longevity
- Closed
-
MB-28511 GSI stack trace in query service exits - centos longevity
- Closed
- relates to
-
MB-28356 rebalance failure in centos longevity with error mover_crashed and bulk_set_vbucket_state_failed
- Closed
-
MB-28428 Compression and XATTR can crash memcached
- Closed
-
MB-28626 memcached exits with status 134 and rebalance failures in centos longevity
- Closed