Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61238

memcached crash on 7.6.0-2180 (MP1): violating invariant "getItemsForCursor: Limits must be > 0"

    XMLWordPrintable

Details

    Description

      Steps:

      1. Initialise a 1 node KV cluster
      2. The cluster is configured with just 2 vbuckets.
      3. Create a Magma bucket with 250GB memory and start loading data
      4. Data load consists of doc sizes which are in the range of (2MB - 18MB).
      5. Loaded about 13,000 docs.
      6. Rebalanced-in a node and observed the following during rebalance
        Since the bucket only has 2vbuckets, during rebalance-in, only 1 vbucket was moving from the source to the destination node.
      7. Rebalance exited with reason {mover_crashed,
        {unexpected_exit,
        {'EXIT',<0.20525.1>,
        {{{{badmatch,{error,timeout}},
        [{mc_client_binary,stats_recv,4,
        [{file,"src/mc_client_binary.erl"},
        {line,172}]},
        {mc_client_binary,stats,4,
        [{file,"src/mc_client_binary.erl"},
        {line,499}]},
        {ns_memcached,do_handle_call,3,
        [{file,"src/ns_memcached.erl"},
        {line,620}]},
        {ns_memcached,worker_loop,3,
        [{file,"src/ns_memcached.erl"},
        {line,243}]},
        {proc_lib,init_p_do_apply,3,
        [{file,"proc_lib.erl"},{line,240}]}]},
        {gen_server,call,
        ['ns_memcached-test_bucket',
        {get_dcp_docs_estimate,1,
        "replication:ns_1@172.31.30.47->ns_1@ec2-54-88-120-100.compute-1.amazonaws.com:test_bucket"},
        300000]}},
        {gen_server,call,
        [{'janitor_agent-test_bucket',
        'ns_1@172.31.30.47'},
        {if_rebalance,<0.20500.1>,
        {wait_dcp_data_move,
        ['ns_1@ec2-54-88-120-100.compute-1.amazonaws.com'],
        1}},
        infinity]}}}}}.
        Rebalance Operation Id = 472f8c4c222090b8a956b2c3217f2438  

      1. Stack backtrace of crashed thread:
           #0  /opt/couchbase/bin/memcached() [0x400000+0x835001]
           #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x852ec5]
           #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x85320f]
           #3  /lib64/libc.so.6() [0x7f7a77a00000+0x54df0]
           #4  /lib64/libc.so.6() [0x7f7a77a00000+0xa154c]
           #5  /lib64/libc.so.6(raise+0x16) [0x7f7a77a00000+0x54d46]
           #6  /lib64/libc.so.6(abort+0xd3) [0x7f7a77a00000+0x287f3]
           #7  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xa89ab]
           #8  /opt/couchbase/bin/memcached() [0x400000+0x844fe6]
           #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb82fa]
           #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb8365]
           #11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb85b7]
           #12 /opt/couchbase/bin/memcached() [0x400000+0x92c015]
           #13 /opt/couchbase/bin/memcached() [0x400000+0xbf5f4]
           #14 /opt/couchbase/bin/memcached() [0x400000+0x47e318]
           #15 /opt/couchbase/bin/memcached() [0x400000+0x5ab271]
           #16 /opt/couchbase/bin/memcached() [0x400000+0x5b37e3]
           #17 /opt/couchbase/bin/memcached() [0x400000+0x5b3949]
           #18 /opt/couchbase/bin/memcached() [0x400000+0x5b9bb1]
           #19 /opt/couchbase/bin/memcached() [0x400000+0x78c76f]
           #20 /opt/couchbase/bin/memcached() [0x400000+0x4f77f2]
           #21 /opt/couchbase/bin/memcached() [0x400000+0x78a50a]
           #22 /opt/couchbase/bin/memcached() [0x400000+0x78d8c3]
           #23 /opt/couchbase/bin/memcached() [0x400000+0x92800f]
           #24 /opt/couchbase/bin/memcached() [0x400000+0x910b37]
           #25 /opt/couchbase/bin/memcached() [0x400000+0x92abaa]
           #26 /opt/couchbase/bin/memcached() [0x400000+0x786850]
           #27 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xe4aa3]
           #28 /lib64/libc.so.6() [0x7f7a77a00000+0x9f802]
           #29 /lib64/libc.so.6() [0x7f7a77a00000+0x3f450] 

      1. Cb-collect logs are attached.

      Attachments

        For Gerrit Dashboard: MB-61238
        # Subject Branch Project Status CR V

        Activity

          People

            vibhav.sp Vibhav S P
            vibhav.sp Vibhav S P
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty