Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35669

Rebalance swap (9 -> 9) failed with with error message mover_crashed in build 6.5.0-4064

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Rebalance test 9->9 node swap failed with below error message in 6.5.0-4064 build . 

       

      "completionMessage":"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.14666.20>,\n bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.96.109',\n {'EXIT',\n {{{badmatch,{error,closed,\n [{mc_client_binary,cmd_vocal_recv,5,\n [

      {file,\"src/mc_client_binary.erl\"},\n {line,153}]},\n {mc_client_binary,\n wait_for_seqno_persistence,3,\n [{file,"src/mc_client_binary.erl"}

      ,\n {line,691}]},\n {ns_memcached,\n 'wait_for_seqno_persistence/3-fun-0',\n 3,\n [

      {file,\"src/ns_memcached.erl\"}

      ,\n {line,1258}]},\n

      {ns_memcached,\n 'perform_very_long_call/3-fun-0',2,\n [
      {file,"src/ns_memcached.erl"}

      ,\n {line,343}]},\n {ns_memcached_sockets_pool,\n 'executing_on_socket/3-fun-0',3,\n [

      {file,\n \"src/ns_memcached_sockets_pool.erl\"}

      ,\n {line,73}]},\n {async,'async_init/4-fun-2',3,\n [

      {file,\"src/async.erl\"}

      ,\n {line,211}]}]},\n {gen_server,call,\n [

      {'janitor_agent-bucket-1',\n 'ns_1@172.23.96.109'}

      ,\n {if_rebalance,<0.14434.20>,\n {update_vbucket_state,226,replica,\n undefined,'ns_1@172.23.96.101'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4,\n [

      {file,\"src/janitor_agent.erl\"}

      ,\n {line,432}]},\n {proc_lib,init_p,3,\n [

      {file,\"proc_lib.erl\"}

      ,{line,232}]}]}}}}."}

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            As per discussion, assigning back to Dan to check backtrace (doesn't match exception seen).

            drigby Dave Rigby added a comment - As per discussion, assigning back to Dan to check backtrace (doesn't match exception seen).
            owend Daniel Owen added a comment -

            Was using the incorrect version of couchbase-server. Here is the correct BT

            (gdb) bt
            #0  0x00007fa2f3e451f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
            #1  0x00007fa2f3e468e8 in __GI_abort () at abort.c:90
            #2  0x00007fa2f4999195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
            #3  0x00000000005476dd in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86
            #4  0x00007fa2f4996f86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
            #5  0x00007fa2f4996fd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
            #6  0x00007fa2f4997213 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fa1b8000940, tinfo=tinfo@entry=0x7fa2efc9cd90 <typeinfo for gsl::fail_fast>, dest=dest@entry=0x7fa2ef7c8610 <gsl::fail_fast::~fail_fast()>)
                at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93
            #7  0x00007fa2ef843e08 in gsl::fail_fast_assert (cond=<optimized out>, message=0x7fa2ef9f5eb8 "GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 537")
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/gsl-lite.h:473
            #8  0x00007fa2ef858667 in EPBucket::flushVBucket (this=0x7fa2edcf2600, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:537
            #9  0x00007fa2ef8a6352 in Flusher::flushVB (this=0x7fa2c084e700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293
            #10 0x00007fa2ef8a699d in Flusher::step (this=0x7fa2c084e700, task=0x7fa2c07b1750) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:210
            #11 0x00007fa2ef89fd7f in ExecutorThread::run (this=0x7fa2c084d300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:153
            #12 0x00007fa2f67abf27 in run (this=0x7fa2cb116ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
            #13 platform_thread_wrap (arg=0x7fa2cb116ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
            #14 0x00007fa2f41dae25 in start_thread (arg=0x7fa1c77f6700) at pthread_create.c:308
            #15 0x00007fa2f3f0834d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
            
            

            The issues is a pre-condition failure

            GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 537
            

            owend Daniel Owen added a comment - Was using the incorrect version of couchbase-server. Here is the correct BT (gdb) bt #0 0x00007fa2f3e451f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007fa2f3e468e8 in __GI_abort () at abort.c:90 #2 0x00007fa2f4999195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x00000000005476dd in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86 #4 0x00007fa2f4996f86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47 #5 0x00007fa2f4996fd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57 #6 0x00007fa2f4997213 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fa1b8000940, tinfo=tinfo@entry=0x7fa2efc9cd90 <typeinfo for gsl::fail_fast>, dest=dest@entry=0x7fa2ef7c8610 <gsl::fail_fast::~fail_fast()>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93 #7 0x00007fa2ef843e08 in gsl::fail_fast_assert (cond=<optimized out>, message=0x7fa2ef9f5eb8 "GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 537") at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/gsl-lite.h:473 #8 0x00007fa2ef858667 in EPBucket::flushVBucket (this=0x7fa2edcf2600, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:537 #9 0x00007fa2ef8a6352 in Flusher::flushVB (this=0x7fa2c084e700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293 #10 0x00007fa2ef8a699d in Flusher::step (this=0x7fa2c084e700, task=0x7fa2c07b1750) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:210 #11 0x00007fa2ef89fd7f in ExecutorThread::run (this=0x7fa2c084d300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:153 #12 0x00007fa2f67abf27 in run (this=0x7fa2cb116ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #13 platform_thread_wrap (arg=0x7fa2cb116ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #14 0x00007fa2f41dae25 in start_thread (arg=0x7fa1c77f6700) at pthread_create.c:308 #15 0x00007fa2f3f0834d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 The issues is a pre-condition failure GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 537
            owend Daniel Owen added a comment -

            This is the expect that is failing in EPBucket::flushVBucket.

            if (hcs) {
                                Expects(hcs > vbstate.highCompletedSeqno);
                                vbstate.highCompletedSeqno = *hcs;
                            }
            

            owend Daniel Owen added a comment - This is the expect that is failing in EPBucket::flushVBucket. if (hcs) { Expects(hcs > vbstate.highCompletedSeqno); vbstate.highCompletedSeqno = *hcs; }
            owend Daniel Owen added a comment - Looks like duplicate of https://issues.couchbase.com/browse/MB-35631
            drigby Dave Rigby added a comment -

            Duplicate of MB-35631.

            drigby Dave Rigby added a comment - Duplicate of MB-35631 .

            People

              sharath.sulochana Sharath Sulochana (Inactive)
              sharath.sulochana Sharath Sulochana (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty