Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56646

Crash in processCreateVBucketsComplete during KV node shutdown

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 node cluster (1 kv +index+n1ql , 1 kv and 1 cbas node)
      2. Create an Ephemeral bucket
      3. Load 10k docs in bucket.
      4. Create a dataset on that bucket.
      5. Let the data ingestion complete on dataset.
      6. Now disconnect local link so that the dataset becomes disconnected.
      7. Now delete 50% data from KV bucket.
      8. Reconnect the local link so that dataset becomes connected.
      9. Verify that the dataset now only has 5K docs.

      Note:
      Found this issue while scanning failures on weekly run. dmp file was missing from cbcollect, so couldn't get backtraces, MIniDmp was found on node 172.23.97.199 .

      Logs:

      2023-04-26T00:41:51.255813-07:00 CRITICAL Detected previous crash
      		 
      2023-04-26T00:41:51.255957-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.2.0-5318). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/ad644654-ded3-453d-cb05468a-202520cc.dmp before terminating.
      		 
      2023-04-26T00:41:51.255972-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      2023-04-26T00:41:51.255973-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x7431f8]
      		 
      2023-04-26T00:41:51.255974-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x794cea]
      		 
      2023-04-26T00:41:51.255975-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x795028]
      		 
      2023-04-26T00:41:51.255976-07:00 CRITICAL    #3  /lib/x86_64-linux-gnu/libpthread.so.0() [0x7ff5e9c53000+0x13140]
      		 
      2023-04-26T00:41:51.255977-07:00 CRITICAL    #4  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7ff5e7a54000+0x38ce1]
      		 
      2023-04-26T00:41:51.255978-07:00 CRITICAL    #5  /lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7ff5e7a54000+0x22537]
      		 
      2023-04-26T00:41:51.255979-07:00 CRITICAL    #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5e7f87000+0x9963c]
      		 
      2023-04-26T00:41:51.255980-07:00 CRITICAL    #7  /opt/couchbase/bin/memcached() [0x400000+0x74d71b]
      		 
      2023-04-26T00:41:51.255981-07:00 CRITICAL    #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5e7f87000+0xa48f6]
      		 
      2023-04-26T00:41:51.255982-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5e7f87000+0xa4961]
      		 
      2023-04-26T00:41:51.255983-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5e7f87000+0xa4bf4]
      		 
      2023-04-26T00:41:51.255984-07:00 CRITICAL    #11 /opt/couchbase/bin/../lib/libstdc++.so.6(__cxa_bad_typeid+0) [0x7ff5e7f87000+0x990e4]
      		 
      2023-04-26T00:41:51.255986-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x25d52c]
      		 
      2023-04-26T00:41:51.255989-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x2be058]
      		 
      2023-04-26T00:41:51.255990-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x35221d]
      		 
      2023-04-26T00:41:51.255991-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x356ead]
      		 
      2023-04-26T00:41:51.255992-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x36003e]
      		 
      2023-04-26T00:41:51.255993-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x6bbd79]
      		 
      2023-04-26T00:41:51.255994-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x6b543a]
      		 
      2023-04-26T00:41:51.255995-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6bd12e]
      		 
      2023-04-26T00:41:51.255995-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x81b240]
      		 
      2023-04-26T00:41:51.255996-07:00 CRITICAL    #21 /opt/couchbase/bin/memcached() [0x400000+0x805cda]
      		 
      2023-04-26T00:41:51.255997-07:00 CRITICAL    #22 /opt/couchbase/bin/memcached() [0x400000+0x81e1f9]
      		 
      2023-04-26T00:41:51.255998-07:00 CRITICAL    #23 /opt/couchbase/bin/memcached() [0x400000+0x6b5134]
      		 
      2023-04-26T00:41:51.255998-07:00 CRITICAL    #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5e7f87000+0xcdd40]
      		 
      2023-04-26T00:41:51.255999-07:00 CRITICAL    #25 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7ff5e9c53000+0x7ea7]
      		 
      2023-04-26T00:41:51.256159-07:00 CRITICAL    #26 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7ff5e7a54000+0xfca2f]

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.22494.ini replicas=0,GROUP=P0;durability,kv_quota_percent=70,bucket_storage=couchstore,get-cbcollect-info=False,upgrade_version=7.2.0-5318 -t cbas.cbas_bucket_operations.CBASBucketOperations.test_delete_some_docs_in_KV_after_dataset_disconnect_and_then_reconnect_dataset,bucket_type=ephemeral,bucket_eviction_policy=noEviction,num_items=100000,compression_mode=active,GROUP=P0;durability;NotForPersistenceTest,services_init=kv:n1ql:index-cbas-kv,nodes_init=3,cbas_spec=local_datasets,bucket_spec=analytics.default,cluster_kv_infra=bkt_spec'

      Job: http://qa.sc.couchbase.com/job/test_suite_executor-TAF/239400/consoleFull

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty