Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38945

memcahced crashed: Checkpoint::expelItem: not found in index Item[0x7f1cbe64c060] with key:cid:0x1:set_vbucket_state

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Step 1: Rebalance Overview

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.220 | kv       | Cluster node |
      | 172.23.105.221 | None     | <--- IN ---  |
      | 172.23.105.223 | None     | <--- IN ---  |
      | 172.23.105.225 | None     | <--- IN ---  |
      | 172.23.105.226 | None     | <--- IN ---  |
      | 172.23.105.227 | None     | <--- IN ---  |
      | 172.23.105.208 | None     | <--- IN ---  |
      +----------------+----------+--------------+
      

      Step 2: Rebalance Overview

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.225 | kv       | Cluster node |
      | 172.23.105.226 | kv       | Cluster node |
      | 172.23.105.220 | kv       | Cluster node |
      | 172.23.105.223 | kv       | Cluster node |
      | 172.23.105.227 | kv       | Cluster node |
      | 172.23.105.221 | kv       | Cluster node |
      | 172.23.105.208 | [u'kv']  | --- OUT ---> |
      +----------------+----------+--------------+
      

      Step 3: Rebalance Overview

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.225 | kv       | Cluster node |
      | 172.23.105.226 | [u'kv']  | --- OUT ---> |
      | 172.23.105.220 | kv       | Cluster node |
      | 172.23.105.223 | kv       | Cluster node |
      | 172.23.105.227 | [u'kv']  | --- OUT ---> |
      | 172.23.105.221 | kv       | Cluster node |
      +----------------+----------+--------------+
      

      Step 4: Rebalance Overview

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.225 | [u'kv']  | --- OUT ---> |
      | 172.23.105.220 | kv       | Cluster node |
      | 172.23.105.223 | [u'kv']  | --- OUT ---> |
      | 172.23.105.221 | kv       | Cluster node |
      +----------------+----------+--------------+
      

      CORE DUMPS SEEN ON SERVER 172.23.105.220
      CORE DUMPS SEEN ON SERVER 172.23.105.221

      memcached.log.000093.txt:2020-04-23T10:34:35.921561-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      memcached.log.000093.txt:2020-04-23T10:34:35.921614-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Checkpoint::expelItem: not found in index Item[0x7f1cbe64c060] with key:cid:0x1:set_vbucket_state
      memcached.log.000093.txt:2020-04-23T10:34:36.366072-07:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1121). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/1a15f0b3-b8c6-dcd6-6b69d78f-182e7352.dmp before terminating.
      memcached.log.000093.txt:2020-04-23T10:34:36.366110-07:00 CRITICAL Stack backtrace of crashed thread:
      memcached.log.000093.txt:2020-04-23T10:34:36.366289-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
      memcached.log.000093.txt:2020-04-23T10:34:36.366300-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
      memcached.log.000093.txt:2020-04-23T10:34:36.366309-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
      memcached.log.000093.txt:2020-04-23T10:34:36.366316-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f1db85c6000+0xf5d0]
      memcached.log.000093.txt:2020-04-23T10:34:36.366339-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f1db81f9000+0x36207]
      memcached.log.000093.txt:2020-04-23T10:34:36.366358-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f1db81f9000+0x378f8]
      memcached.log.000093.txt:2020-04-23T10:34:36.366387-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f1db8cfb000+0x91195]
      memcached.log.000093.txt:2020-04-23T10:34:36.366396-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x145e02]
      memcached.log.000093.txt:2020-04-23T10:34:36.366412-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8ef86]
      memcached.log.000093.txt:2020-04-23T10:34:36.366429-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8efd1]
      memcached.log.000093.txt:2020-04-23T10:34:36.366443-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8f213]
      memcached.log.000093.txt:2020-04-23T10:34:36.366453-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x5ab5d]
      memcached.log.000093.txt:2020-04-23T10:34:36.366460-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x627b4]
      memcached.log.000093.txt:2020-04-23T10:34:36.366467-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x6d5f4]
      memcached.log.000093.txt:2020-04-23T10:34:36.366476-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x133cc4]
      memcached.log.000093.txt:2020-04-23T10:34:36.366480-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f1dbaba4000+0x8ee7]
      memcached.log.000093.txt:2020-04-23T10:34:36.366485-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f1db85c6000+0x7dd5]
      memcached.log.000093.txt:2020-04-23T10:34:36.366515-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f1db81f9000+0xfdead]
      

      Saw the crashes in couple of tests:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_magma.ini -p num_items=5000000,GROUP=P0,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=32,rerun=false,enable_dp=False,infra_log_level=debug,log_level=debug -t rebalance_new.rebalance_out.RebalanceOutTests.incremental_rebalance_out_with_ops,nodes_init=7,replicas=1,num_items=50000,value_size=1024,GROUP=P0;non_dgm -m rest'
       
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_magma.ini -p num_items=5000000,GROUP=P0,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=32,rerun=false,enable_dp=False,infra_log_level=debug,log_level=debug -t rebalance_new.rebalance_out.RebalanceOutTests.rebalance_out_with_failover_full_addback_recovery,nodes_init=5,replicas=1,num_items=10000,GROUP=P0;SET1,skip_cleanup=True -m rest'
      

      Attachments

        Activity

          People

            ben.huddleston Ben Huddleston
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty