Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35209

Rebalance Out failed with reason wait_seqno_persisted_failed. Durability=Majority, bucket replica=0

    XMLWordPrintable

Details

    Description

      Steps to Reproduce
      1. Create a 3 node cluster

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.155 | [u'kv']  | Cluster node |
      | 172.23.105.159 | None     | <--- IN ---  |
      | 172.23.105.205 | None     | <--- IN ---  |
      +----------------+----------+--------------+
      

      2. Add a default bucket with replicas=0
      3. Load 50k items to the bucket with durability=MAJORITY
      Bucket statistics:

      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      | Bucket  | Type    | Replicas | TTL | Items | RAM Quota  | RAM Used  | Disk Used |
      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      | default | membase | 0        | 0   | 50000 | 6716129280 | 120314264 | 177073826 |
      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      

      Cluster statistics:

      +----------------+----------+-----------------+------------
      | Node           | Services | Active Items | Replica 
      +----------------+----------+-----------------+------------
      | 172.23.105.205 | [u'kv']  | 16648        | 0
      | 172.23.105.155 | [u'kv']  | 16684        | 0
      | 172.23.105.159 | [u'kv']  | 16668        | 0
      +----------------+----------+-----------------+------------
      

      4. Rebalance out 1 node and start data loading in a separate thread

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.205 | [u'kv']  | --- OUT ---> |
      | 172.23.105.155 | [u'kv']  | Cluster node |
      | 172.23.105.159 | [u'kv']  | Cluster node |
      +----------------+----------+--------------+
      

      Rebalance failed:

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.14396.0>,
      {{wait_seqno_persisted_failed,"default",972,
      58,
      [{'ns_1@172.23.105.159',
      {'EXIT',
      {socket_closed,
      {gen_server,call,
      [{'janitor_agent-default',
      'ns_1@172.23.105.159'},
      {if_rebalance,<0.6472.0>,
      {wait_seqno_persisted,972,58}},
      infinity]}}}}]},
      [{ns_single_vbucket_mover,
      '-wait_seqno_persisted_many/5-fun-2-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},
      {line,488}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 89ccca0dadbb9290c709f4a87ec310b8
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              drigby Dave Rigby (Inactive)
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty