Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38682

Rebalance failed in the second step while updating replica from 0 > 1 > 2.

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Create a 4 node cluster and a default bucket with replica=0

        +----------------+----------+--------------+
        | Nodes          | Services | Status       |
        +----------------+----------+--------------+
        | 172.23.105.220 | kv       | Cluster node |
        | 172.23.105.221 | None     | <--- IN ---  |
        | 172.23.105.223 | None     | <--- IN ---  |
        | 172.23.105.225 | None     | <--- IN ---  |
        +----------------+----------+--------------+
        

      2. Load 10M items in the bucket:

        2020-04-09 00:41:28,311 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
        +---------+---------+----------+-----+---------+------------+------------+------------+
        | Bucket  | Type    | Replicas | TTL | Items   | RAM Quota  | RAM Used   | Disk Used  |
        +---------+---------+----------+-----+---------+------------+------------+------------+
        | default | membase | 0        | 0   | 10000000 | 7881097216 | 2143179672 | 2143007310 |
        +---------+---------+----------+-----+---------+------------+------------+------------+
        

      3. Update replica to 1 and rebalance. Update all the items create earlier.
      4. Update replica from 1 to 2 and rebalance and update all the items again in parallel.
      5. Rebalance in the step above failed.

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.2514.3>,
      {{dcp_wait_for_data_move_failed,"default",
      123,'ns_1@172.23.122.225',
      ['ns_1@172.23.122.208','ns_1@172.23.121.79'],
      {error,no_stats_for_this_vbucket}},
      [{ns_single_vbucket_mover,
      '-wait_dcp_data_move/5-fun-0-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},
      {line,465}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 455535458d4b34402d37b446165332a2
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty