Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46047

[Collections] Memcached crash at couchstore_open_doc_with_docinfo in bySeqnoScanCallback

    XMLWordPrintable

Details

    Description

      Script to Repo

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,bucket_spec=multi_bucket.30_buckets,cycles=20000,nodes_in=2,quota_percent=100'

      Steps to Reproduce
      1. Create a 3 node kv init cluster
      2. Create 30 buckets, each with 34 scopes, and each scope with 1 collection with no documents
      3. Rebalance-in 2 kv nodes
      4. Rebalance-out 2 kv nodes and repeat steps 3-4

      Rebalance-out failed

      +----------------+----------+-----------------------+---------------+--------------+
      		 
      | Nodes          | Services | Version               | CPU           | Status       |
      		 
      +----------------+----------+-----------------------+---------------+--------------+
      		 
      | 172.23.105.215 | kv       | 7.0.0-5050-enterprise | 73.0162832773 | Cluster node |
      		 
      | 172.23.106.237 | kv       | 7.0.0-5050-enterprise | 64.6982646983 | --- OUT ---> |
      		 
      | 172.23.105.220 | kv       | 7.0.0-5050-enterprise | 37.896928683  | Cluster node |
      		 
      | 172.23.107.90  | kv       | 7.0.0-5050-enterprise | 63.7854725332 | --- OUT ---> |
      		 
      | 172.23.105.217 | kv       | 7.0.0-5050-enterprise | 58.929956061  | Cluster node |
      		 
      | 172.23.105.219 | kv       | 7.0.0-5050-enterprise | 55.4980595084 | Cluster node |
      		 
      +----------------+----------+-----------------------+---------------+--------------+

      1 coredump is seen on 172.23.106.237:
      Attachments have info about bt
      checking for "CRITICAL" on this node:

      [root@localhost logs]# grep CRITICAL memcached.log.*
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662363-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-5050). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/8cee594d-541f-4dc8-d4bcd289-a201eff3.dmp before terminating.
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662417-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662689-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0x66d9f3]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662714-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x68d98a]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662735-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x68dcc8]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662743-07:00 CRITICAL     #3  /lib64/libpthread.so.0() [0x7f8955c39000+0xf5d0]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662766-07:00 CRITICAL     #4  /opt/couchbase/bin/memcached() [0x400000+0x4efce0]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662787-07:00 CRITICAL     #5  /opt/couchbase/bin/memcached(couchstore_open_doc_with_docinfo+0xd9) [0x400000+0x4e8669]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662834-07:00 CRITICAL     #6  /opt/couchbase/bin/memcached() [0x400000+0x42ef9f]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662855-07:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x4e7519]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662874-07:00 CRITICAL     #8  /opt/couchbase/bin/memcached() [0x400000+0x4fc349]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662895-07:00 CRITICAL     #9  /opt/couchbase/bin/memcached() [0x400000+0x4fc04e]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662914-07:00 CRITICAL     #10 /opt/couchbase/bin/memcached(couchstore_changes_since+0x1d6) [0x400000+0x4e9726]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662933-07:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0x42d8c1]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662950-07:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0x48a933]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662964-07:00 CRITICAL     #13 /opt/couchbase/bin/memcached() [0x400000+0x48d54a]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662977-07:00 CRITICAL     #14 /opt/couchbase/bin/memcached() [0x400000+0x48f891]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.662989-07:00 CRITICAL     #15 /opt/couchbase/bin/memcached() [0x400000+0x48fb04]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663004-07:00 CRITICAL     #16 /opt/couchbase/bin/memcached() [0x400000+0x29bee7]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663018-07:00 CRITICAL     #17 /opt/couchbase/bin/memcached() [0x400000+0x297030]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663040-07:00 CRITICAL     #18 /opt/couchbase/bin/memcached() [0x400000+0x6d7290]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663060-07:00 CRITICAL     #19 /opt/couchbase/bin/memcached() [0x400000+0x6bef6a]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663078-07:00 CRITICAL     #20 /opt/couchbase/bin/memcached() [0x400000+0x6da249]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663093-07:00 CRITICAL     #21 /opt/couchbase/bin/memcached() [0x400000+0x2955bd]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663140-07:00 CRITICAL     #22 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f895636f000+0xcdd40]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663146-07:00 CRITICAL     #23 /lib64/libpthread.so.0() [0x7f8955c39000+0x7dd5]
      		 
      memcached.log.000278.txt:2021-04-30T16:14:58.663180-07:00 CRITICAL     #24 /lib64/libc.so.6(clone+0x6d) [0x7f895586c000+0xfdead]

      Note that debug info is installed on all nodes

      [root@localhost crash]# rpm -qa | grep couchbase-server

      couchbase-server-7.0.0-5050.x86_64

      couchbase-server-debuginfo-7.0.0-5050.x86_64

      Attachments

        1. bt_full.txt
          17 kB
        2. output.log
          84 kB
        3. rebalance_fail_ST.txt
          8 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-46221 memcached crash in CouchKVStore::commit

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              paolo.cocchi Paolo Cocchi
              sumedh.basarkod Sumedh Basarkod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty