Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26142

FATAL Breakpad caught crash in memcached version 5.1.0-1220.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • Enterprise Edition 5.1.0 build 1220
    • Untriaged
    • Yes

    Description

      The rqg jobs work fine with 5.1.0-1219 but fail with 1220.
      Checkins done:
      http://172.23.120.24/builds/latestbuilds/couchbase-server/5.1.0/1220/CHANGELOG
      changed projects :

      	kv_engine changed from e188c86dbf002213146d782adff086ac7a24c401 to master
      		[+] 2e56ef3 Merge "Merge remote-tracking branch 'couchbase/spock'"
      		[+] db135e8 [RocksDB] Enable support for RocksDB as default
      		[+] 1f77f4e Merge remote-tracking branch 'couchbase/spock'
      		[+] 496d9b9 MB-26037: Allow DCP rollback on vbuckets in pending state
      		[+] 4871a42 MB-25991: Don't directly modify StoredValue during expiration
      		[+] c26e1ef MB-25938: Create a copy of the xattrs before modifying them
      
      

      Crash reported in log:

      2017-09-21T16:31:09.397277-07:00 FATAL Breakpad caught crash in memcached version 5.1.0-1220. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/4d714af6-22a7-be33-255da0e0-6e7096fb.dmp before terminating.
      2017-09-21T16:31:09.397452-07:00 FATAL Stack backtrace of crashed thread:
      2017-09-21T16:31:09.397707-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x3a185]
      2017-09-21T16:31:09.397779-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3d4) [0x400000+0x98ee4]
      2017-09-21T16:31:09.397841-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP7siginfoPv+0x97) [0x400000+0x991f7]
      2017-09-21T16:31:09.397915-07:00 FATAL     /lib64/libpthread.so.0() [0x7fc166e97000+0xf7e0]
      2017-09-21T16:31:09.398048-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/../lib/librocksdb.so.5.6(_ZNSt10_HashtableISsSt4pairIKSsN7rocksdb14OptionTypeInfoEESaIS4_ENSt8__detail10_Select1stESt8equal_toISsESt4hashISsENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb1ELb0ELb1EEEEC1IPKS4
      2017-09-21T16:31:09.398167-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/../lib/librocksdb.so.5.6() [0x7fc15b8b4000+0x1256d3]
      2017-09-21T16:31:09.398228-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe74f]
      2017-09-21T16:31:09.398283-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0x12f75]
      2017-09-21T16:31:09.398337-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe366]
      2017-09-21T16:31:09.398390-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0x1271a]
      2017-09-21T16:31:09.398452-07:00 FATAL     /lib64/libdl.so.2() [0x7fc1670b4000+0xf66]
      2017-09-21T16:31:09.398507-07:00 FATAL     /lib64/ld-linux-x86-64.so.2() [0x7fc168f6f000+0xe366]
      2017-09-21T16:31:09.398560-07:00 FATAL     /lib64/libdl.so.2() [0x7fc1670b4000+0x129c]
      2017-09-21T16:31:09.398614-07:00 FATAL     /lib64/libdl.so.2(dlopen+0x31) [0x7fc1670b4000+0xee1]
      2017-09-21T16:31:09.398669-07:00 FATAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(cb_dlopen+0x1f) [0x7fc1674d7000+0x7a7f]
      2017-09-21T16:31:09.398723-07:00 FATAL     /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0(load_engine+0x3c) [0x7fc16894a000+0x466c]
      2017-09-21T16:31:09.398780-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x4ac76]
      2017-09-21T16:31:09.398837-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x4b284]
      2017-09-21T16:31:09.398902-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x1d1d1]
      2017-09-21T16:31:09.398982-07:00 FATAL     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fc165009000+0x1ed1d]
      2017-09-21T16:31:09.399041-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x15cb1]
      
      

      Attaching cbcollect_info.

      Test run:

      ./testrunner -i /root/tuqvm.ini gsi_type=plasma -t rqg.test_rqg.RQGTests.test_rqg_concurrent,test_file_path=b/resources/rqg/simple_table_db/query_tests_using_templates/query_50000_aggregate_functions.txt.zip,database=multiple_table_db,reset_database=True,concurreny_count=10,index_quota_percent=30,password=password,use_mysql=True,replicas=0,create_secondary_indexes=True,run_query_with_primary=True,record_failure=True,failure_record_path=/tmp,nodes_init=1,build_index_batch_size=1000,build_secondary_index_in_seq=True,nodes_init=1,number_of_buckets=1,total_queries=10000,skip_cleanup=false
      

      Test fails during giving permission to the cbadminbucket user as follows:

      2017-09-21 16:33:01 | ERROR | MainProcess | test_thread | [rest_client._http_request] PUT http://172.23.105.209:8091/settings/rbac/users/local/cbadminbucket body: name=cbadminbucket&roles=admin headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password
      Traceback (most recent call last):
        File "pytests/basetestcase.py", line 268, in setUp
          self.add_built_in_server_user(node=self.master)
        File "pytests/basetestcase.py", line 2060, in add_built_in_server_user
          status = RbacBase().add_user_role(rolelist, RestConnection(node), 'builtin')
        File "pytests/security/rbac_base.py", line 70, in add_user_role
          response = rest.add_set_builtin_user(userid,payload)
        File "lib/membase/api/rest_client.py", line 3865, in add_set_builtin_user
          raise Exception(content)
      Exception: ["Unexpected server error, request logged."]
      ERROR
       
      
      

      I can help with the rqg setup and give the specific machine on which the crash is seen if needed.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ajay.bhullar Ajay Bhullar
              Prerna.Manaktala Prerna Manaktala (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty