Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-4617
-
Untriaged
-
-
1
-
Unknown
Description
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini sdk_timeout=60,bucket_eviction_policy=fullEviction,randomize_value=True,doc_size=1024,bucket_storage=magma -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica,nodes_init=4,num_items=5000000,doc_size=4096,sdk_timeout=60,replicas=2,GROUP=P0'
|
Rebalance Failure
2021-04-06 22:45:44,444 | test | ERROR | pool-4-thread-25 | [rest_client:_rebalance_status_and_progress:1574] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'6e391ab431fc0b3c56d27a05ca969160', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c15430136e23ca8195205eaf166b085d', u'status': u'notRunning'} - rebalance failed
|
2021-04-06 22:45:44,489 | test | INFO | pool-4-thread-25 | [rest_client:print_UI_logs:2675] Latest logs from UI on 172.23.120.170:
|
2021-04-06 22:45:44,489 | test | ERROR | pool-4-thread-25 | [rest_client:print_UI_logs:2677] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.123', u'tstamp': 1617774343380L, u'shortText': u'message', u'serverTime': u'2021-04-06T22:45:43.380Z', u'text': u"Control connection to memcached on 'ns_1@172.23.121.123' disconnected. Check logs for details."}
|
2021-04-06 22:45:44,490 | test | ERROR | pool-4-thread-25 | [rest_client:print_UI_logs:2677] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.121.123', u'tstamp': 1617774343373L, u'shortText': u'message', u'serverTime': u'2021-04-06T22:45:43.373Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n2021-04-06T22:45:42.445446-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f854e0fc000+0x36387]\n2021-04-06T22:45:42.445472-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f854e0fc000+0x37a78]\n2021-04-06T22:45:42.445505-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f854ebff000+0x91195]\n2021-04-06T22:45:42.445520-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x15ff12]\n2021-04-06T22:45:42.445537-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f854ebff000+0x8ef86]\n2021-04-06T22:45:42.445559-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f854ebff000+0x8efd1]\n2021-04-06T22:45:42.445587-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f854ebff000+0xb9dfe]\n2021-04-06T22:45:42.445599-07:00 CRITICAL #11 /lib64/libpthread.so.0() [0x7f854e4ca000+0x7ea5]\n2021-04-06T22:45:42.445629-07:00 CRITICAL #12 /lib64/libc.so.6(clone+0x6d) [0x7f854e0fc000+0xfe8dd]"}
|
2021-04-06 22:45:44,490 | test | ERROR | pool-4-thread-25 | [rest_client:print_UI_logs:2677] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1617774343214L, u'shortText': u'message', u'serverTime': u'2021-04-06T22:45:43.214Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.8526.4>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,158}]},\n {mc_client_binary,set_vbucket,4,\n [{file,"src/mc_client_binary.erl"},\n {line,396}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,574}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,224}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]},\n {gen_server,call,\n [\'ns_memcached-default\',\n {get_vbucket_high_seqno,479},\n 180000]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.121.115\'},\n {if_rebalance,<0.5188.4>,\n {update_vbucket_state,485,active,\n undefined,undefined,\n [[\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.123\',\n \'ns_1@172.23.121.116\'],\n [\'ns_1@172.23.121.115\']]}},\n infinity]}}}}}.\nRebalance Operation Id = b0f47fc2e8c82a8a2a626eaefb613c8e'}
|
2021-04-06 22:45:44,490 | test | ERROR | pool-4-thread-25 | [rest_client:print_UI_logs:2677] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1617774343156L, u'shortText': u'message', u'serverTime': u'2021-04-06T22:45:43.156Z', u'text': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.120.170.'}
|
2021-04-06 22:45:44,490 | test | ERROR | pool-4-thread-25 | [rest_client:print_UI_logs:2677] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1617774343042L, u'shortText': u'message', u'serverTime': u'2021-04-06T22:45:43.042Z', u'text': u'Worker <0.8513.4> (for action {move,{485,\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.123\',\n \'ns_1@172.23.121.116\'],\n [\'ns_1@172.23.121.115\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.8526.4>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 158}]},\n {mc_client_binary,\n set_vbucket,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 396}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 574}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 224}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 249}]}]},\n {gen_server,\n call,\n [\'ns_memcached-default\',\n {get_vbucket_high_seqno,\n 479},\n 180000]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.121.115\'},\n {if_rebalance,\n <0.5188.4>,\n {update_vbucket_state,\n 485,\n active,\n undefined,\n undefined,\n [[\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.123\',\n \'ns_1@172.23.121.116\'],\n [\'ns_1@172.23.121.115\']]}},\n infinity]}}}}'}
|
|
Stack Trace:
2021-04-06 22:45:59,867 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:725] 172.23.121.115: Stack Trace of first crash - 4ab2c353-fc70-4cb9-0cbf25ae-96c1b669.dmp
|
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007fde4b1f4387 in raise () from /lib64/libc.so.6
|
#0 0x00007fde4b1f4387 in raise () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x00007fde4b1f5a78 in abort () from /lib64/libc.so.6
|
No symbol table info available.
|
#2 0x00007fde4bd52195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
|
terminating = false
|
t = <optimized out>
|
#3 0x000000000055ff12 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:93
|
No locals.
|
#4 0x00007fde4bd4ff86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
|
No locals.
|
#5 0x00007fde4bd4ffd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
|
No locals.
|
#6 0x00007fde4bd7adfe in std::execute_native_thread_routine (__p=0x7fde45f73090) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:91
|
__t = {_M_t = {_M_t = {<std::_Tuple_impl<0, std::thread::_State*, std::default_delete<std::thread::_State> >> = {<std::_Tuple_impl<1, std::default_delete<std::thread::_State> >> = {<std::_Head_base<1, std::default_delete<std::thread::_State>, true>> = {<std::default_delete<std::thread::_State>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, std::thread::_State*, false>> = {_M_head_impl = 0x7fde45f73090}, <No data fields>}, <No data fields>}}}
|
#7 0x00007fde4b593ea5 in start_thread () from /lib64/libpthread.so.0
|
No symbol table info available.
|
#8 0x00007fde4b2bc8dd in clone () from /lib64/libc.so.6
|
No symbol table info available.
|