Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36489

[Volume Test] Rebalance Stuck after decrement of replica number of the bucket from 2 to 1.

    XMLWordPrintable

Details

    Description

      Steps to Reproduce:

      1. Create a 6 node cluster.

        +----------------+----------+--------------+
        | Nodes          | Services | Status       |
        +----------------+----------+--------------+
        | 172.23.106.134 | [u'kv']  | Cluster node |
        | 172.23.106.136 | None     | <--- IN ---  |
        | 172.23.106.137 | None     | <--- IN ---  |
        | 172.23.106.138 | None     | <--- IN ---  |
        | 172.23.105.168 | None     | <--- IN ---  |
        | 172.23.106.82  | None     | <--- IN ---  |
        +----------------+----------+--------------+
         

      1. Create a bucket with compression = Off, eviction policy=valueOnly, replicas=1.
      2. Load 1K docs with durability = MAJORITY.
      3. Rebalance In 1 node(172.23.106.83) with 200 creates, 400 updates in parallel with durability = MAJORITY.
      4. Rebalance Out 1 node(172.23.105.168) with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.
      5. Rebalance In 2 nodes(172.23.106.85, 172.23.105.168) and Rebalance Out 1 node(172.23.106.83) with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.
      6. Swap Rebalance 1 node(IN=172.23.106.86 ,OUT=172.23.106.82) with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.
      7. Update Replica number of the bucket from 1 to 2.
      8. Rebalance In 1 node(172.23.106.82) with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.
      9. Rebalance the cluster with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.
      10.  While Step 9 is in progress, Stop the memcached process and start it again after 20 seconds.
      11.  Start 200 creates, 400 updates, 200 deletes with durability = MAJORITY.
      12.  While Step 11 is in progress, failover a node (172.23.106.137).
      13.  Rebalance Out the node failed over in Step 12.
      14.  Rebalance In 1 node (172.23.106.83)
      15.  Start 200 creates, 400 updates, 200 deletes with durability = MAJORITY.
      16.  While Step 15 is in progress, failover a node (172.23.106.83)
      17.  Fully Recover the node failed over in Step 16.
      18.  Rebalance the cluster.
      19.  Start 200 creates, 400 updates, 200 deletes with durability = MAJORITY.
      20.  While Step 19 is in progress, failover a node (172.23.106.83)
      21.  Delta Recovery the node failed over in Step 20.
      22.  Rebalance the cluster.
      23.  Decrement the replica number of the bucket from 2 to 1.
      24.  Rebalance the cluster with 200 creates, 400 updates, 200 deletes in parallel with durability = MAJORITY.

      Rebalance is stuck at 99.90234375%.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              james.harrison James Harrison (Inactive)
              prateek.kumar Prateek Kumar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty