Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.5.0
-
Untriaged
-
-
Yes
-
KV-Engine MH 2nd Beta
Description
Build: 6.5.0-4218 , it passed on 6.5.0-4169
Test: Rebalance component with durability
Test Steps:
- Create 6 node cluster all kv
- Create 3 buckets with replica 1,1,2
- load data
- as with no majority and with majority (bucket having replica 1 and 2) – >for mad-hatter
- without durability for alice
- Rebalance in kv node
- Rebalance out kv node
- Swap kv nodes
- Update replica of bucket as 2
- Failover -> full recovery
- Failover -> delta recovery
- Graceful Failover and rebalance out 1 KV
- Rebalance-in KV which were out due to failover
- reset bucket replica (1,1,2)
- Rebalance-in 2 kv
- Rebalance-out 2 kv
- Swap rebalance 2 kv
- Rebalance in 1 kv and out 2 kv
- STOP and START memcached
- Add back all the nodes for cluster reset
Seeing memcached crash with status 0 and 139 observed dumps
2019-09-10T03:42:57.506-07:00, ns_log:0:info:message(ns_1@172.23.96.48) - Service 'memcached' exited with status 139. Restarting. Messages:
|
2019-09-10T03:42:57.360041-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14543e]
|
2019-09-10T03:42:57.360047-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x145754]
|
2019-09-10T03:42:57.360054-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f1c80e4d000+0xf5e0]
|
2019-09-10T03:42:57.360064-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1c7bbed000+0x1905dd]
|
2019-09-10T03:42:57.360071-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1c7bbed000+0xd7991]
|
2019-09-10T03:42:57.360077-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1c7bbed000+0x12ff9c]
|
2019-09-10T03:42:57.360081-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f1c8342e000+0x8f27]
|
2019-09-10T03:42:57.360086-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f1c80e4d000+0x7e25]
|
2019-09-10T03:42:57.360113-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f1c80a8a000+0xf834d]
|
This also leads to rebalance failed with mover crashed
2019-09-10T03:42:57.508-07:00, ns_memcached:0:info:message(ns_1@172.23.96.48) - Control connection to memcached on 'ns_1@172.23.96.48' disconnected. Check logs for details.
|
2019-09-10T03:42:57.513-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.74) - Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.24449.315>,
|
{{{{{child_interrupted,
|
{'EXIT',<25738.1266.57>,socket_closed}},
|
[{dcp_replicator,spawn_and_wait,1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,249}]},
|
{dcp_replicator,handle_call,3,
|
[{file,"src/dcp_replicator.erl"},
|
{line,121}]},
|
{gen_server,try_handle_call,4,
|
[{file,"gen_server.erl"},{line,636}]},
|
{gen_server,handle_msg,6,
|
[{file,"gen_server.erl"},{line,665}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,247}]}]},
|
{gen_server,call,
|
[<25738.1271.57>,
|
{setup_replication,[246,249,262,272]},
|
infinity]}},
|
{gen_server,call,
|
['replication_manager-other-2',
|
{change_vbucket_replication,246,
|
'ns_1@172.23.96.48'},
|
infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-other-2',
|
'ns_1@172.23.96.18'},
|
{if_rebalance,<0.29961.313>,
|
{update_vbucket_state,271,active,
|
undefined,undefined,undefined}},
|
infinity]}}}}}.
|
Rebalance Operation Id = 62ee07a91285ff48209266a2cfdd930d
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-