Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39005

ActiveStream seqnoAdvanced may dereference null vBucket

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV Sprint 2020-April

    Description

      Build: 7.0.0-1902
      Test: -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale: 3
      Day: 1

      Memcached Crash seen on node: 172.23.96.254

      2020-04-26T22:39:46.514852-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-1902). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/6d1f701c-aa15-436f-3352d32d-0e18fa7e.dmp before terminating.
      		 
      2020-04-26T22:39:46.514875-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      2020-04-26T22:39:46.515073-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1379ad]
      		 
      2020-04-26T22:39:46.515086-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14d66e]
      		 
      2020-04-26T22:39:46.515094-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14d984]
      		 
      2020-04-26T22:39:46.515102-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f7963add000+0xf5e0]
      		 
      2020-04-26T22:39:46.515112-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x74aa5]
      		 
      2020-04-26T22:39:46.515117-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x86f4f]
      		 
      2020-04-26T22:39:46.515122-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x893db]
      		 
      2020-04-26T22:39:46.515126-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x8b690]
      		 
      2020-04-26T22:39:46.515130-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x8b9de]
      		 
      2020-04-26T22:39:46.515137-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x144443]
      		 
      2020-04-26T22:39:46.515162-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f7967941000+0x13ccef]
      		 
      2020-04-26T22:39:46.515168-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f7966497000+0x10677]
      		 
      2020-04-26T22:39:46.515173-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f7963add000+0x7e25]
      		 
      2020-04-26T22:39:46.515205-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f796371a000+0xf834d]

      Step:

      [2020-04-26T22:33:38-07:00, sequoiatools/couchbase-cli:7.0:dca88f] failover -c 172.23.97.74:8091 --server-failover 172.23.121.77:8091 -u Administrator -p password --force
      		 
      [2020-04-26T22:33:53-07:00, sequoiatools/couchbase-cli:7.0:f1fcb3] rebalance -c 172.23.97.74:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs f1fcb3
      		 
      docker start f1fcb3
      		 
       
      		 
      *Unable to display progress bar on this os
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      		 
      [2020-04-26T22:39:56-07:00, sequoiatools/cmd:f75f6c] 60

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.18.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.151.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1587968459/collectinfo-2020-04-27T062100-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          is caused by

          Task - A task that needs to be done. MB-37402 Collections: Handling max-visible seqno

          • Major - Major loss of function.
          • Resolved
          For Gerrit Dashboard: MB-39005
          # Subject Branch Project Status CR V
          126832,2 MB-39005: Check vb ptr before use in completeBackfill master kv_engine Status: MERGED +2 +1

          Activity

            People

              james.harrison James Harrison (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty