Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8345

nodes down due to memcached crash at destination cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 2.1.0
    • 2.1.0
    • ns_server
    • Security Level: Public
    • None
    • windows 2008 r2 64bit
    • Windows 64-bit

    Description

      Environment: 6 windows 2008 r2 64 bit
      Run test
      ./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source

      Test failed with error could not reach node
      https://friendpaste.com/4gUHpCTiJLdRDYYzp0hRUd

      Check diag around the failed time, I saw erlang crash in one node

      Mon May 20 17:05:53.190281 Pacific Daylight Time 3: (default) Shutting down tap connections!
      Mon May 20 17:05:53.191257 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.125"
      Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.123"
      Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_30"
      Mon May 20 17:05:54.208835 Pacific Daylight Time 3: (default) Had to wait 999 ms for shutdown
      Mon May 20 17:05:55.209812 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
      Mon May 20 17:05:56.210789 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
      Mon May 20 17:05:57.212742 Pacific Daylight Time 3: (default) Had to wait 1001 ms for shutdown
      [user:info,2013-05-20T17:06:26.069,ns_1@10.3.2.124:ns_memcached-default<0.31624.6>:ns_memcached:terminate:735]Control connection to memcached on 'ns_1@10.3.2.124' disconnected: {{badmatch,
      {error,
      closed}},
      [

      {mc_client_binary, cmd_binary_vocal_recv, 5}

      ,

      {mc_client_binary, select_bucket, 2}

      ,

      {ns_memcached, ensure_bucket, 2}

      ,

      {ns_memcached, handle_info, 2}

      ,

      {gen_server, handle_msg, 5}

      ,

      {ns_memcached, init,1}

      ,

      {gen_server, init_it, 6}

      ,

      {proc_lib, init_p_do_apply, 3}

      ]}
      [error_logger:error,2013-05-20T17:06:26.069,ns_1@10.3.2.124:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: erlang:apply/2
      pid: <0.31640.6>
      registered_name: []
      exception error: no match of right hand side value

      {error,closed}

      in function mc_binary:quick_stats_recv/3
      in call from mc_binary:quick_stats_loop/5
      in call from mc_binary:quick_stats/5
      in call from ns_memcached:do_handle_call/3
      in call from ns_memcached:worker_loop/3
      ancestors: ['ns_memcached-default','single_bucket_sup-default',
      <0.31607.6>]
      messages: []
      links: [<0.31624.6>]
      dictionary: []
      trap_exit: false
      status: running
      heap_size: 2584
      stack_size: 24
      reductions: 567619
      neighbours:

      Link to manifest of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

      This bug may relate to bug MB-8266

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty