Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36720

memcached crashed during rebalance - monotonic invariant exception on the PassiveDM HPS

    XMLWordPrintable

Details

    Description

      QE Note:

      -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=1,standard_buckets=1,num-swap=2,GROUP=P0;MHbeta;durability;windows,durability=MAJORITY
      

      Steps:
      1. Create a 4 node cluster, a bucket with replica=1.
      2. Load 250k items
      3. Swap rebalance 2 nodes:
      Nodes in Cluster = [u'ns_1@172.23.105.225', u'ns_1@172.23.105.220', u'ns_1@172.23.105.223', u'ns_1@172.23.105.221']
      Adding Nodes = [172.23.105.226:8091, 172.23.105.227:8091 ]
      Ejecting Nodes = ['ns_1@172.23.105.225', 'ns_1@172.23.105.223']

      Rebalance cluster and memcached crashed on ns_1@172.23.105.227

      Worker <0.9617.3> (for action {move,{682,
      ['ns_1@172.23.105.223',
      'ns_1@172.23.105.225'],
      ['ns_1@172.23.105.227',
      'ns_1@172.23.105.226'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT',
      <0.12629.3>,
      {{{badmatch,
      {error,
      closed}},
      [{mc_client_binary,
      cmd_vocal_recv,
      5,
      [{file,
      "src/mc_client_binary.erl"},
      {line,
      155}]},
      {mc_client_binary,
      wait_for_seqno_persistence,
      3,
      [{file,
      "src/mc_client_binary.erl"},
      {line,
      696}]},
      {ns_memcached,
      '-wait_for_seqno_persistence/3-fun-0-',
      3,
      [{file,
      "src/ns_memcached.erl"},
      {line,
      1272}]},
      {ns_memcached,
      '-perform_very_long_call/3-fun-0-',
      2,
      [{file,
      "src/ns_memcached.erl"},
      {line,
      344}]},
      {ns_memcached_sockets_pool,
      '-executing_on_socket/3-fun-0-',
      3,
      [{file,
      "src/ns_memcached_sockets_pool.erl"},
      {line,
      92}]},
      {async,
      '-async_init/4-fun-1-',
      3,
      [{file,
      "src/async.erl"},
      {line,
      197}]}]},
      {gen_server,
      call,
      [{'janitor_agent-default',
      'ns_1@172.23.105.227'},
      {if_rebalance,
      <0.18669.2>,
      {update_vbucket_state,
      682,
      active,
      undefined,
      undefined,
      [['ns_1@172.23.105.227',
      'ns_1@172.23.105.226']]}},
      infinity]}}}}
      

      Working fine on 6.5.0-4719

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty