Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.0
-
6.5.0-3298
-
Triaged
-
Centos 64-bit
-
-
No
-
KV-Engine Mad-Hatter Beta
Description
- Create a cluster of 2 nodes. Create a bucket with replica = 1
- Load documents - Durability = Yes
- Start loading documents again
- Add 2 node to the cluster
- Click rebalance
Expected Result:
Rebalance should succeed, and there should not be any ambiguous exception.
Actual Result:
Rebalance fails and there are memached exits with an expectation failure:
2019-05-22T07:43:46.389742-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2019-05-22T07:43:46.389794-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc: 440
|
Backtrace:
2019-05-22T07:43:46.437359-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-3298). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/0c339244-c234-0a0a-170229de-4e0c7f90.dmp before terminating.
|
2019-05-22T07:43:46.437381-07:00 CRITICAL Stack backtrace of crashed thread:
|
2019-05-22T07:43:46.437531-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x12f2f0]
|
2019-05-22T07:43:46.437548-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x1449de]
|
2019-05-22T07:43:46.437556-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x144cf4]
|
2019-05-22T07:43:46.437567-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fc71ab0e000+0xf5d0]
|
2019-05-22T07:43:46.437588-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7fc71a741000+0x36207]
|
2019-05-22T07:43:46.437605-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fc71a741000+0x378f8]
|
2019-05-22T07:43:46.437634-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fc71b243000+0x91195]
|
2019-05-22T07:43:46.437648-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1406ad]
|
2019-05-22T07:43:46.437662-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc71b243000+0x8ef86]
|
2019-05-22T07:43:46.437676-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc71b243000+0x8efd1]
|
2019-05-22T07:43:46.437716-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc71b243000+0x8f213]
|
2019-05-22T07:43:46.437728-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0xd6a98]
|
2019-05-22T07:43:46.437737-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0xd2dd3]
|
2019-05-22T07:43:46.437743-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0xd472e]
|
2019-05-22T07:43:46.437752-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0x186e17]
|
2019-05-22T07:43:46.437763-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0xe2055]
|
2019-05-22T07:43:46.437770-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0x12e195]
|
2019-05-22T07:43:46.437776-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0x12e65d]
|
2019-05-22T07:43:46.437782-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fc715f8d000+0x127a40]
|
2019-05-22T07:43:46.437786-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fc71d0de000+0x8f27]
|
2019-05-22T07:43:46.437791-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fc71ab0e000+0x7dd5]
|
2019-05-22T07:43:46.437816-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fc71a741000+0xfdead]
|
2019-05-22 07:43:04,036 | ERROR | MainProcess | pool-2-thread-5 | [rest_client:print_UI_logs:2617] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.97.52', u'tstamp': 1558536181903L, u'shortText': u'message', u'serverTime': u'2019-05-22T07:43:01.903Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.15972.4>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,155}]},\n {mc_client_binary,set_vbucket,4,\n [{file,"src/mc_client_binary.erl"},\n {line,412}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,550}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,228}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {gen_server,call,\n [\'ns_memcached-default\',\n {set_vbucket,511,active,\n [[\'ns_1@172.23.97.52\',\n \'ns_1@172.23.97.53\']]},\n 180000]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.97.52\'},\n {if_rebalance,<0.15439.4>,\n {update_vbucket_state,511,active,paused,\n undefined,\n [[\'ns_1@172.23.97.52\',\n \'ns_1@172.23.97.53\']]}},\n infinity]}}}}}. Operation Id = 61c70092824b5bf54d00e10e4f88b195'}
|
2019-05-22 07:43:04,036 | ERROR | MainProcess | pool-2-thread-5 | [rest_client:print_UI_logs:2617] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.97.52', u'tstamp': 1558536181898L, u'shortText': u'message', u'serverTime': u'2019-05-22T07:43:01.898Z', u'text': u'<0.15604.4> exited with {unexpected_exit,\n {\'EXIT\',<0.15972.4>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},{line,155}]},\n {mc_client_binary,set_vbucket,4,\n [{file,"src/mc_client_binary.erl"},{line,412}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},{line,550}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},{line,228}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {gen_server,call,\n [\'ns_memcached-default\',\n {set_vbucket,511,active,\n [[\'ns_1@172.23.97.52\',\'ns_1@172.23.97.53\']]},\n 180000]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\'ns_1@172.23.97.52\'},\n {if_rebalance,<0.15439.4>,\n {update_vbucket_state,511,active,paused,\n undefined,\n [[\'ns_1@172.23.97.52\',\'ns_1@172.23.97.53\']]}},\n infinity]}}}}'}
|