Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30097

[System test] : Memcached crashed with 139 with active compression

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • centos cluster2

    Description

      Build: Enterprise Edition 5.5.0 build 2884

      While running system test we enabled compression: Active on default bucket, there are memcached crashed and rebalance failed

      Service 'memcached' exited with status 139. Restarting. Messages: 2018-06-13T02:13:45.804778Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0x117106] 2018-06-13T02:13:45.804793Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0x83677] 2018-06-13T02:13:45.804806Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0xcc55f] 2018-06-13T02:13:45.804819Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0x1236fb] 2018-06-13T02:13:45.804835Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0xeea26] 2018-06-13T02:13:45.804847Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0x8254b] 2018-06-13T02:13:45.804856Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fb7d05cb000+0xc0c36] 2018-06-13T02:13:45.804868Z CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fb7e6aaf000+0x8f07] 2018-06-13T02:13:45.804883Z CRITICAL /lib64/libpthread.so.0() [0x7fb7e4c22000+0x7e25] 2018-06-13T02:13:45.804946Z CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fb7e485f000+0xf834d]
       
      ns_1@172.23.96.2122:13:45 AM   Wed Jun 13, 2018

      Service 'memcached' exited with status 139. Restarting. Messages: 2018-06-13T02:13:45.820449Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0x117106] 2018-06-13T02:13:45.820466Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0x83677] 2018-06-13T02:13:45.820481Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0xcc55f] 2018-06-13T02:13:45.820492Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0x1236fb] 2018-06-13T02:13:45.820500Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0xeea26] 2018-06-13T02:13:45.820509Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0x8254b] 2018-06-13T02:13:45.820517Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7519cb000+0xc0c36] 2018-06-13T02:13:45.820527Z CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa767ee3000+0x8f07] 2018-06-13T02:13:45.820539Z CRITICAL /lib64/libpthread.so.0() [0x7fa766056000+0x7e25] 2018-06-13T02:13:45.820601Z CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fa765c93000+0xf834d]
      ns_1@172.23.96.2072:13:45 AM   Wed Jun 13, 2018

      Supportal link : https://supportal.couchbase.com/snapshot/c17b2de91da651e7d7e5bff75676517c::0 

       Its a part of same standard system test for vulcan with compression enabled on the "default" bucket. Here are the steps we are doing for the test.

      • Create 16 node cluster (9 - KV , 3 - index , 2- query, 2-fts), 2 node XDCR cluster , 1 sysn gateway
      • Create 10 bucket (1- default , 9 tpcc)

      bucket-create -c 172.23.96.206 -u Administrator -p password --bucket default --bucket-ramsize 9002 --bucket-type couchbase --bucket-replica 1 --enable-flush 1 --wait --password password --bucket-eviction-policy fullEviction --compression-mode active

      * other bucket does not have compression enabled

      • create views 

      [2018-06-13T02:13:30-07:00, appropriate/curl:551d7a] -s -X PUT -u Administrator:password -H Content-Type:application/json http://172.23.96.206:8092/default/_design/scale -d {"views":{"stats":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 500) && (doc.rating < 520)){ emit(meta.id, doc.ops_sec); }} }", "reduce": "_stats"},"padd":{"map":"function(doc, meta){ if(doc.profile){ if (doc.rating < 200){ emit(meta.id, doc.padding); }} }"},"array":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 200) && (doc.rating< 300)){ emit(doc.active_hosts, null); }} }"}}}
      [pull] appropriate/curl
      [2018-06-13T02:13:33-07:00, appropriate/curl:d50db0] -s -X PUT -u Administrator:password -H Content-Type:application/json http://172.23.96.206:8092/default/_design/scale -d {"views":{"stats":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 500) && (doc.rating < 520)){ emit(meta.id, doc.ops_sec); }} }", "reduce": "_stats"},"padd":{"map":"function(doc, meta){ if(doc.profile){ if (doc.rating < 200){ emit(meta.id, doc.padding); }} }"},"array":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 200) && (doc.rating< 300)){ emit(doc.active_hosts, null); }} }"},"stats":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 500) && (doc.rating < 520)){ emit(meta.id, doc.ops_sec); }} }", "reduce": "_stats"},"padd":{"map":"function(doc, meta){ if(doc.profile){ if (doc.rating < 200){ emit(meta.id, doc.padding); }} }"},"array":{"map":"function(doc, meta){ if(doc.profile){ if((doc.rating > 200) && (doc.rating< 300)){ emit(doc.active_hosts, null); }} }"}}}
      [pull] appropriate/curl
      [2018-06-13T02:13:36-07:00, appropriate/curl:3e8cd5] -s -X PUT -u Administrator:password -H Content-Type:application/json http://172.23.96.206:8092/ORDER_LINE/_design/all -d {"views":{"all_ids":{"map":"function(doc, meta){ emit(meta.id, null) }"}}}
      [pull] appropriate/curl
      [2018-06-13T02:13:40-07:00, appropriate/curl:185cb7] -s -X PUT -u Administrator:password -H Content-Type:application/json http://172.23.96.206:8092/ORDER_LINE/_design/all -d {"views":{"all_ids":{"map":"function(doc, meta){ emit(meta.id, null) }"},"all_ids":{"map":"function(doc, meta){ emit(meta.id, null) }"}}}

      * Load default bucket

      • Remove KV node <-- start failing 

      I am able to reproduce again http://supportal.couchbase.com/snapshot/77459cb2d452225b750c28f0537c6154::0

      Attachments

        For Gerrit Dashboard: MB-30097
        # Subject Branch Project Status CR V

        Activity

          People

            sriram Sriram Ganesan (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty