Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45491

[Magma] With disk full and write commit failures, add 1 node to add more disk to the cluster leads to memcached crash.

    XMLWordPrintable

Details

    • Untriaged
    • Hide
      Couldn't attach the logs here, they being quite big.
      Logs: root@172.23.120.105:/root/MB-45491
      Show
      Couldn't attach the logs here, they being quite big. Logs: root@172.23.120.105 :/root/ MB-45491
    • 1
    • Unknown

    Description

      Steps:

      1. Create a 3 node cluster
      2. Create magma bucket and load 5M docs of size 1kb
      3. Fill up the disk on all the 3 nodes by writing some dummy data.
      4. Ensure that write commits are failing on all nodes by inserting more data on all 3 nodes.
      5. Add 1 node and rebalance.
      6. Rebalance stuck and memcached crashed on existing node in the cluster.

      172.23.121.115

      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007f5b1845d387 in raise () from /lib64/libc.so.6
       #0  0x00007f5b1845d387 in raise () from /lib64/libc.so.6
       No symbol table info available.
       #1  0x00007f5b1845ea78 in abort () from /lib64/libc.so.6
       No symbol table info available.
       #2  0x00007f5b18fbb195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
               terminating = false
               t = <optimized out>
       #3  0x000000000055ff12 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:93
       No locals.
       #4  0x00007f5b18fb8f86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
       No locals.
       #5  0x00007f5b18fb8fd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
       No locals.
       #6  0x00007f5b18fe3dfe in std::execute_native_thread_routine (__p=0x7f5b14377090) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:91
               __t = {_M_t = {_M_t = {<std::_Tuple_impl<0, std::thread::_State*, std::default_delete<std::thread::_State> >> = {<std::_Tuple_impl<1, std::default_delete<std::thread::_State> >> = {<std::_Head_base<1, std::default_delete<std::thread::_State>, true>> = {<std::default_delete<std::thread::_State>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, std::thread::_State*, false>> = {_M_head_impl = 0x7f5b14377090}, <No data fields>}, <No data fields>}}}
       #7  0x00007f5b187fcea5 in start_thread () from /lib64/libpthread.so.0
       No symbol table info available.
       #8  0x00007f5b185258dd in clone () from /lib64/libc.so.6
       No symbol table info available.
      

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini sdk_timeout=60,bucket_eviction_policy=fullEviction,randomize_value=True,doc_size=1024 -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_add_nodes,nodes_init=3,num_items=5000000,doc_size=4096,sdk_timeout=60,replicas=1,GROUP=P0'
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty