Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41619

[Collections] - Minidumps seen in system test during rebalance in + collections data load + bucket level durability set in

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Script to Repro

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_second_cluster.yml -test tests/rebalance/test_allRebalance_collections.yml -scope tests/rebalance/scope_rebalance_collection_couchstore.yml -scale 10 -repeat 0 -log_level 0 -version 7.0.0-3186 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=0 -show_topology=true
      

      We had added bucket level durability setting in our system test and were testing it out. Basically this is the first time we ran this test.

      grep CRITICAL on memcached log on 172.23.120.73

      memcached<0.129.0>: 2020-09-23T05:01:04.459746-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      memcached<0.129.0>: 2020-09-23T05:01:04.459831-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): stoll
      memcached<0.129.0>: terminate called after throwing an instance of 'std::out_of_range'
      memcached<0.129.0>:   what():  stoll
       
      [ns_server:info,2020-09-23T05:01:04.776-07:00,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:224]memcached<0.129.0>: 2020-09-23T05:01:04.659132-07:00 WARNING 84: Slow operation. {"cid":"fbf5374772365785/cb6ab9121f71e088/a11a0c00","duration":"502 ms","trace":"request=1430439590469449:502599 store=0:0 sync_write.ack_local=1430439590532144:0 store=1430439590483842:50 sync_write.ack_remote=1430440091996779:0 sync_write.prepare=1430439590513987:502085 store=0:0 store=1430440092984449:1","command":"SET","peer":"{"ip":"172.23.104.254","port":47240}","bucket":"default","packet":{"bodylen":84,"cas":0,"datatype":["JSON"],"extlen":8,"key":"<ud>.00000000000000004453</ud>","keylen":21,"magic":"ClientRequest","opaque":793249,"opcode":"SET","vbucket":144}}
      memcached<0.129.0>: 2020-09-23T05:01:04.659885-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3186). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/80502074-2e61-4c72-20b7efad-17199217.dmp before terminating.
      memcached<0.129.0>: 2020-09-23T05:01:04.659930-07:00 CRITICAL Stack backtrace of crashed thread:
      memcached<0.129.0>: 2020-09-23T05:01:04.660106-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x193e8d]
      memcached<0.129.0>: 2020-09-23T05:01:04.660122-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1aa20a]
      memcached<0.129.0>: 2020-09-23T05:01:04.660135-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1aa548]
      memcached<0.129.0>: 2020-09-23T05:01:04.660142-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f368b990000+0xf5f0]
      memcached<0.129.0>: 2020-09-23T05:01:04.660164-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f368b5c2000+0x36337]
      memcached<0.129.0>: 2020-09-23T05:01:04.660191-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f368b5c2000+0x37a28]
      memcached<0.129.0>: 2020-09-23T05:01:04.660227-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f368c0c5000+0x91195]
      memcached<0.129.0>: 2020-09-23T05:01:04.660240-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1a4bd2]
      memcached<0.129.0>: 2020-09-23T05:01:04.660257-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f368c0c5000+0x8ef86]
      memcached<0.129.0>: 2020-09-23T05:01:04.660279-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f368c0c5000+0x8efd1]
      memcached<0.129.0>: 2020-09-23T05:01:04.660293-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f368c0c5000+0x8f213]
      memcached<0.129.0>: 2020-09-23T05:01:04.660318-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x37) [0x7f368c0c5000+0xb7867]
      memcached<0.129.0>: 2020-09-23T05:01:04.660335-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0x1f1987]
      memcached<0.129.0>: 2020-09-23T05:01:04.660345-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0x237394]
      memcached<0.129.0>: 2020-09-23T05:01:04.660353-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0x239fc1]
      memcached<0.129.0>: 2020-09-23T05:01:04.660380-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0xb8d58]
      memcached<0.129.0>: 2020-09-23T05:01:04.660393-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0xbc479]
      memcached<0.129.0>: 2020-09-23T05:01:04.660399-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0xbe4eb]
      memcached<0.129.0>: 2020-09-23T05:01:04.660405-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0xbe74e]
      memcached<0.129.0>: 2020-09-23T05:01:04.660420-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0x17ea03]
      memcached<0.129.0>: 2020-09-23T05:01:04.660427-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f368f678000+0x8265f]
      memcached<0.129.0>: 2020-09-23T05:01:04.660434-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f368e142000+0x10947]
      memcached<0.129.0>: 2020-09-23T05:01:04.660442-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f368b990000+0x7e65]
      memcached<0.129.0>: 2020-09-23T05:01:04.660490-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f368b5c2000+0xfe88d]
      

      Bt full of 80502074-2e61-4c72-20b7efad-17199217.dmp on 172.23.120.73
      See bt_full_sys_test.txt

      cbcollect_info attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty