Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24474

memcached (ephemeral bucket) crashes during rebalance from 9 to 10 nodes

    XMLWordPrintable

Details

    Description

      Build 5.0.0-2915. Last known good build is 5.0.0-2909.

      Probably a duplicate of MB-24472. Though it started crashing only recently and only with ephemeral buckets.

      Setup:

      • 9 -> 10 nodes
      • 1 ephemeral bucket (NRU)
      • 500M documents
      • 50K mixed ops/sec

      [ns_server:error,2017-05-20T21:21:13.367-07:00,ns_1@172.23.96.100:<0.15484.7>:ns_single_vbucket_mover:spawn_and_wait:110]Got unexpected exit signal {'EXIT',<0.10921.7>,
                                  {unexpected_exit,
                                   {'EXIT',<0.13145.7>,
                                    {{{{badmatch,{error,closed}},
                                       [{mc_client_binary,cmd_vocal_recv,5,
                                         [{file,"src/mc_client_binary.erl"},
                                          {line,149}]},
                                        {mc_client_binary,select_bucket,2,
                                         [{file,"src/mc_client_binary.erl"},
                                          {line,336}]},
                                        {ns_memcached,ensure_bucket,2,
                                         [{file,"src/ns_memcached.erl"},
                                          {line,1225}]},
                                        {ns_memcached,handle_info,2,
                                         [{file,"src/ns_memcached.erl"},{line,718}]},
                                        {gen_server,handle_msg,5,
                                         [{file,"gen_server.erl"},{line,604}]},
                                        {ns_memcached,init,1,
                                         [{file,"src/ns_memcached.erl"},{line,168}]},
                                        {gen_server,init_it,6,
                                         [{file,"gen_server.erl"},{line,304}]},
                                        {proc_lib,init_p_do_apply,3,
                                         [{file,"proc_lib.erl"},{line,239}]}]},
                                      {gen_server,call,
                                       ['ns_memcached-bucket-1',
                                        {get_dcp_docs_estimate,1009,
                                         "replication:ns_1@172.23.96.108->ns_1@172.23.96.107:bucket-1"},
                                        180000]}},
                                     {gen_server,call,
                                      [{'janitor_agent-bucket-1',
                                        'ns_1@172.23.96.108'},
                                       {if_rebalance,<0.10921.7>,
                                        {wait_dcp_data_move,
                                         ['ns_1@172.23.96.109','ns_1@172.23.96.107'],
                                         1009}},
                                       infinity]}}}}}
      

      2017-05-20T21:21:02.235021-07:00 WARNING Breakpad caught crash in memcached version 5.0.0-2915. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3944930a-f59f-3b3e-6feae241-7e37425b.dmp before terminating.
      2017-05-20T21:21:02.235660-07:00 WARNING Stack backtrace of crashed thread:
      2017-05-20T21:21:02.238630-07:00 WARNING     /opt/couchbase/bin/memcached() [0x4348f5]
      2017-05-20T21:21:02.238688-07:00 WARNING     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3d4) [0x486564]
      2017-05-20T21:21:02.238717-07:00 WARNING     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x97) [0x486877]
      2017-05-20T21:21:02.238752-07:00 WARNING     /lib64/libpthread.so.0() [0x7f0b081b6100]
      2017-05-20T21:21:02.238842-07:00 WARNING     /lib64/libc.so.6(gsignal+0x37) [0x7f0b062945f7]
      2017-05-20T21:21:02.238903-07:00 WARNING     /lib64/libc.so.6(abort+0x148) [0x7f0b06295ce8]
      2017-05-20T21:21:02.238984-07:00 WARNING     /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165) [0x7f0b06b999d5]
      2017-05-20T21:21:02.239013-07:00 WARNING     /opt/couchbase/bin/memcached() [0x435b6c]
      2017-05-20T21:21:02.239080-07:00 WARNING     /lib64/libstdc++.so.6() [0x7f0b06b97946]
      2017-05-20T21:21:02.239146-07:00 WARNING     /lib64/libstdc++.so.6() [0x7f0b06b97973]
      2017-05-20T21:21:02.239208-07:00 WARNING     /lib64/libstdc++.so.6() [0x7f0b06b97b93]
      2017-05-20T21:21:02.239239-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01d1a473]
      2017-05-20T21:21:02.239263-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01d1cacd]
      2017-05-20T21:21:02.239288-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01c8d2e6]
      2017-05-20T21:21:02.239309-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01c8d590]
      2017-05-20T21:21:02.239330-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01c8db58]
      2017-05-20T21:21:02.239352-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01c888da]
      2017-05-20T21:21:02.239372-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01c88c34]
      2017-05-20T21:21:02.239396-07:00 WARNING     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f0b01ce9ba3]
      2017-05-20T21:21:02.239419-07:00 WARNING     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f0b087eab5e]
      2017-05-20T21:21:02.239448-07:00 WARNING     /lib64/libpthread.so.0() [0x7f0b081aedc5]
      2017-05-20T21:21:02.239528-07:00 WARNING     /lib64/libc.so.6(clone+0x6d) [0x7f0b06355ced]
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty