Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51057

[magma]: memcached crashed during replica update on xdcr source cluster.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 7.1.0
    • None
    • couchbase-bucket
    • 7.1.0 build 2305
    • Untriaged
    • 1
    • Unknown

    Description

      Steps:

      1. Step 1: Create a 4 node cluster
      2. Step 1*: Create a 2 node XDCR remote cluster
      3. Step 2: Create required buckets and collections.
      4. Step 2*: Create required buckets and collections on XDCR remote.
      5. Step 1: Create 20000000 items sequentially
      6. Step 2: Update 20000000 RandonKey keys to create 50 percent fragmentation
      7. Step 3: Create 20000000 items sequentially
      8. Step 4: Update 20000000 RandonKey keys to create 50 percent fragmentation
      9. Step 5: Rebalance in with Loading of docs
      10. Step 6: Rebalance Out with Loading of docs
      11. Step 7: Rebalance In_Out with Loading of docs
      12. Step 8: Swap with Loading of docs
      13. Step 9: Failover 1 node and RebalanceOut that node with loading in parallel
      14. Step 11: Failover a node and FullRecovery that node
      15. Step 12: Failover a node and DeltaRecovery that node with loading in parallel
      16. Step 13: Updating the bucket replica to 2

      Service 'memcached' exited with status 139. Restarting. Messages:
      #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x78821a]
      #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x788558]
      #3 /lib64/libpthread.so.0() [0x7f3d6fe2d000+0xf5f0]
      #4 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(bufferevent_incref_and_lock_+0x1) [0x7f3d6f0e8000+0xe6a1]
      #5 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(bufferevent_trigger+0x22) [0x7f3d6f0e8000+0xead2]
      #6 /opt/couchbase/bin/memcached() [0x400000+0x1b29b9]
      #7 /opt/couchbase/bin/memcached() [0x400000+0x81579b]
      #8 /opt/couchbase/bin/memcached() [0x400000+0x80f1d8]
      #9 /opt/couchbase/bin/memcached() [0x400000+0x811499]
      #10 /opt/couchbase/bin/memcached() [0x400000+0x812086]
      #11 /opt/couchbase/bin/memcached() [0x400000+0x813a46]
      #12 /opt/couchbase/bin/memcached() [0x400000+0x1b2269]
      #13 /opt/couchbase/bin/memcached() [0x400000+0x79c939]
      #14 /lib64/libpthread.so.0() [0x7f3d6fe2d000+0x7e65]
      #15 /lib64/libc.so.6(clone+0x6d) [0x7f3d6d677000+0xfe88d]
      

      Note: Haven't seen this issue on 2255 build.

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job1.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=4,graceful=True,skip_cleanup=True,num_items=20000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=25,index_nodes=0,xdcr_collections=10,xdcr_remote_nodes=2,cbas_nodes=0,fts_nodes=0,ops_rate=50000,ramQuota=10240,doc_ops=create:update:delete:read,rebl_ops_rate=20000,key_type=RandomKey,vbuckets=1024,mutation_perc=30 -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            From examining the backtrace this is the same issue as MB-50967.

            drigby Dave Rigby added a comment - From examining the backtrace this is the same issue as MB-50967 .

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty