Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19557

mcd crash during rebalance

    XMLWordPrintable

Details

    Description

      During a swap rebalance of system test run 8x2 xdcr cluster, rebalance exited relating to memcached exiting. dmp was generated and backtrace attached. 4.5.0-2497 ubuntu 14.04

      Service 'memcached' exited with status 139. Restarting. Messages: 2016-05-09T10:24:10.855330-07:00 WARNING (default) Failed to fetch data from database, vBucket=188 key=7DF3DDC3-142_14215 error=document not found [none]
      2016-05-09T10:24:10.855643-07:00 WARNING (default) Failed to fetch data from database, vBucket=252 key=7DF3DDC3-142_14221 error=document not found [none]
      2016-05-09T10:24:10.856967-07:00 WARNING (default) Failed to fetch data from database, vBucket=665 key=C9465100-91_329 error=document not found [none]
      2016-05-09T10:24:10.859953-07:00 WARNING Breakpad caught crash in memcached. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/375bcde6-3b9e-6b0b-4bcb2bbe-7d321a98.dmp before terminating.
      2016-05-09T10:24:10.859987-07:00 WARNING Stack backtrace of crashed thread:
      

      The rebalance at the time exited duing get dcp_docs

      <0.1540.14> exited with {unexpected_exit,
                               {'EXIT',<0.1574.14>,
                                {{{{badmatch,{error,closed}},
                                   [{mc_client_binary,cmd_vocal_recv,5,
                                     [{file,"src/mc_client_binary.erl"},{line,156}]},
                                    {mc_client_binary,select_bucket,2,
                                     [{file,"src/mc_client_binary.erl"},{line,351}]},
                                    {ns_memcached,ensure_bucket,2,
                                     [{file,"src/ns_memcached.erl"},{line,1299}]},
                                    {ns_memcached,handle_info,2,
                                     [{file,"src/ns_memcached.erl"},{line,748}]},
                                    {gen_server,handle_msg,5,
                                     [{file,"gen_server.erl"},{line,604}]},
                                    {ns_memcached,init,1,
                                     [{file,"src/ns_memcached.erl"},{line,177}]},
                                    {gen_server,init_it,6,
                                     [{file,"gen_server.erl"},{line,304}]},
                                    {proc_lib,init_p_do_apply,3,
                                     [{file,"proc_lib.erl"},{line,239}]}]},
                                  {gen_server,call,
                                   ['ns_memcached-default',
                                    {get_dcp_docs_estimate,744,
                                     "replication:ns_1@172.23.105.62->ns_1@172.23.106.213:default"},
                                    180000]}},
                                 {gen_server,call,
                                  [{'janitor_agent-default','ns_1@172.23.105.62'},
                                   {if_rebalance,<0.15378.13>,
                                    {wait_dcp_data_move,
                                     ['ns_1@172.23.106.14','ns_1@172.23.106.213'],
                                     744}},
                                   infinity]}}}} hide
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              tommie Tommie McAfee (Inactive)
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty