Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39417

memcached crashed and recovered during data load.

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 node cluster
      2. Create a magma bucket with replicas=0
      3. Load 0-50M items of size 4096
      4. Delete all 0-50M of the items created
      5. Load 50-100M items. Total Items = 50M
      6. Load another 100M-150M items. Total Items = 100M
      7. Observed a memcached crash during step 6.

      Stack Trace from the dmp file on 172.23.97.37:

      Stack Trace of 8e4cb2ee-492e-400d-60f7c5b8-b30ab903.dmp
      rm -rf /opt/couchbase/var/lib/couchbase/crash/8e4cb2ee-492e-400d-60f7c5b8-b30ab903.core
      //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/8e4cb2ee-492e-400d-60f7c5b8-b30ab903.dmp > /opt/couchbase/var/lib/couchbase/crash/8e4cb2ee-492e-400d-60f7c5b8-b30ab903.core
      gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/8e4cb2ee-492e-400d-60f7c5b8-b30ab903.core -ex bt all -ex quit
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007ff3f6cd2c4a in je_dallocx () from /opt/couchbase/bin/../lib/libjemalloc.so.2
       #0  0x00007ff3f6cd2c4a in je_dallocx () from /opt/couchbase/bin/../lib/libjemalloc.so.2
       #1  0x000000000047ee64 in Connection::~Connection() ()
       #2  0x000000000047f079 in Connection::~Connection() ()
       #3  0x0000000000492585 in conn_destroy(Connection*) ()
       #4  0x0000000000486835 in Connection::event_callback(bufferevent*, short, void*) ()
       #5  0x00007ff3f6a94e70 in bufferevent_run_deferred_callbacks_unlocked () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
       #6  0x00007ff3f6a9dd01 in event_process_active_single_queue.isra.31 () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
       #7  0x00007ff3f6a9e54f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
       #8  0x0000000000469189 in worker_libevent(void*) ()
       #9  0x00007ff3f79cd827 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
       #10 0x00007ff3f500ae65 in start_thread () from /lib64/libpthread.so.0
       #11 0x00007ff3f4d3388d in clone () from /lib64/libc.so.6
      

      Memcached logs on node: ns_1@172.23.97.37

      Service 'memcached' exited with status 139. Restarting. Messages:
      2020-05-15T05:15:49.043404-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x7f079]
      2020-05-15T05:15:49.043416-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x92585]
      2020-05-15T05:15:49.043425-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x86835]
      2020-05-15T05:15:49.043437-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fda90642000+0xee70]
      2020-05-15T05:15:49.043446-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fda90642000+0x17d01]
      2020-05-15T05:15:49.043453-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7fda90642000+0x1854f]
      2020-05-15T05:15:49.043469-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x69189]
      2020-05-15T05:15:49.043478-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fda91579000+0x10827]
      2020-05-15T05:15:49.043488-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fda8ebbf000+0x7e65]
      2020-05-15T05:15:49.043547-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fda8e7f1000+0xfe88d] 
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty