Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26142

FATAL Breakpad caught crash in memcached version 5.1.0-1220.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • Enterprise Edition 5.1.0 build 1220
    • Untriaged
    • Yes

    Description

      The rqg jobs work fine with 5.1.0-1219 but fail with 1220.
      Checkins done:
      http://172.23.120.24/builds/latestbuilds/couchbase-server/5.1.0/1220/CHANGELOG
      changed projects :

      	kv_engine changed from e188c86dbf002213146d782adff086ac7a24c401 to master
      		[+] 2e56ef3 Merge "Merge remote-tracking branch 'couchbase/spock'"
      		[+] db135e8 [RocksDB] Enable support for RocksDB as default
      		[+] 1f77f4e Merge remote-tracking branch 'couchbase/spock'
      		[+] 496d9b9 MB-26037: Allow DCP rollback on vbuckets in pending state
      		[+] 4871a42 MB-25991: Don't directly modify StoredValue during expiration
      		[+] c26e1ef MB-25938: Create a copy of the xattrs before modifying them
      
      

      Crash reported in log:

      2017-09-21T16:31:09.397277-07:00 FATAL Breakpad caught crash in memcached version 5.1.0-1220. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/4d714af6-22a7-be33-255da0e0-6e7096fb.dmp before terminating.
      2017-09-21T16:31:09.397452-07:00 FATAL Stack backtrace of crashed thread:
      2017-09-21T16:31:09.397707-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x3a185]
      2017-09-21T16:31:09.397779-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3d4) [0x400000+0x98ee4]
      2017-09-21T16:31:09.397841-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP7siginfoPv+0x97) [0x400000+0x991f7]
      2017-09-21T16:31:09.397915-07:00 FATAL     /lib64/libpthread.so.0() [0x7fc166e97000+0xf7e0]
      2017-09-21T16:31:09.398048-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/../lib/librocksdb.so.5.6(_ZNSt10_HashtableISsSt4pairIKSsN7rocksdb14OptionTypeInfoEESaIS4_ENSt8__detail10_Select1stESt8equal_toISsESt4hashISsENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb1ELb0ELb1EEEEC1IPKS4
      2017-09-21T16:31:09.398167-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/../lib/librocksdb.so.5.6() [0x7fc15b8b4000+0x1256d3]
      2017-09-21T16:31:09.398228-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe74f]
      2017-09-21T16:31:09.398283-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0x12f75]
      2017-09-21T16:31:09.398337-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe366]
      2017-09-21T16:31:09.398390-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0x1271a]
      2017-09-21T16:31:09.398452-07:00 FATAL     /lib64/libdl.so.2() [0x7fc1670b4000+0xf66]
      2017-09-21T16:31:09.398507-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe366]
      2017-09-21T16:31:09.398560-07:00 FATAL     /lib64/libdl.so.2() [0x7fc1670b4000+0x129c]
      2017-09-21T16:31:09.398614-07:00 FATAL     /lib64/libdl.so.2(dlopen+0x31) [0x7fc1670b4000+0xee1]
      2017-09-21T16:31:09.398669-07:00 FATAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(cb_dlopen+0x1f) [0x7fc1674d7000+0x7a7f]
      2017-09-21T16:31:09.398723-07:00 FATAL     /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0(load_engine+0x3c) [0x7fc16894a000+0x466c]
      2017-09-21T16:31:09.398780-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x4ac76]
      2017-09-21T16:31:09.398837-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x4b284]
      2017-09-21T16:31:09.398902-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x1d1d1]
      2017-09-21T16:31:09.398982-07:00 FATAL     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fc165009000+0x1ed1d]
      2017-09-21T16:31:09.399041-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x15cb1]
      
      

      Attaching cbcollect_info.

      Test run:

      ./testrunner -i /root/tuqvm.ini gsi_type=plasma -t rqg.test_rqg.RQGTests.test_rqg_concurrent,test_file_path=b/resources/rqg/simple_table_db/query_tests_using_templates/query_50000_aggregate_functions.txt.zip,database=multiple_table_db,reset_database=True,concurreny_count=10,index_quota_percent=30,password=password,use_mysql=True,replicas=0,create_secondary_indexes=True,run_query_with_primary=True,record_failure=True,failure_record_path=/tmp,nodes_init=1,build_index_batch_size=1000,build_secondary_index_in_seq=True,nodes_init=1,number_of_buckets=1,total_queries=10000,skip_cleanup=false
      

      Test fails during giving permission to the cbadminbucket user as follows:

      2017-09-21 16:33:01 | ERROR | MainProcess | test_thread | [rest_client._http_request] PUT http://172.23.105.209:8091/settings/rbac/users/local/cbadminbucket body: name=cbadminbucket&roles=admin headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password
      Traceback (most recent call last):
        File "pytests/basetestcase.py", line 268, in setUp
          self.add_built_in_server_user(node=self.master)
        File "pytests/basetestcase.py", line 2060, in add_built_in_server_user
          status = RbacBase().add_user_role(rolelist, RestConnection(node), 'builtin')
        File "pytests/security/rbac_base.py", line 70, in add_user_role
          response = rest.add_set_builtin_user(userid,payload)
        File "lib/membase/api/rest_client.py", line 3865, in add_set_builtin_user
          raise Exception(content)
      Exception: ["Unexpected server error, request logged."]
      ERROR
       
      
      

      I can help with the rqg setup and give the specific machine on which the crash is seen if needed.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-26142
          # Subject Branch Project Status CR V

          Activity

            People

              ajay.bhullar Ajay Bhullar
              Prerna.Manaktala Prerna Manaktala (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty