Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35245

[Chain Rebalance]: Subsequent rebalance in of 2 nodes failed because Disk ckpt streamed as Memory ckpt

    XMLWordPrintable

Details

    Description

      Steps to reproduce:
      1. Initialize a 2 node cluster with default bucket having replica=1

      +----------------+----------+--------------+
      		 
      | Nodes          | Services | Status       |
      		 
      +----------------+----------+--------------+
      		 
      | 172.23.105.220 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.221 | None     | <--- IN ---  |
      		 
      +----------------+----------+--------------+

      2. Loaded 10k items into the bucket

      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      		 
      | Bucket  | Type    | Replicas | TTL | Items | RAM Quota  | RAM Used  | Disk Used |
      		 
      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      		 
      | default | membase | 1        | 0   | 10000 | 4477419520 | 141117168 | 76989548  |
      		 
      +---------+---------+----------+-----+-------+------------+-----------+-----------+

      3. Rebalance in 2 nodes and load 10k more items in parallel. Lots of DurabilityImpossibleExceptions are seen which is expected as per. Rebalance completed properly.

      +----------------+----------+--------------+
      		 
      | Nodes          | Services | Status       |
      		 
      +----------------+----------+--------------+
      		 
      | 172.23.105.220 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.221 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.223 | None     | <--- IN ---  |
      		 
      | 172.23.105.225 | None     | <--- IN ---  |
      		 
      +----------------+----------+--------------+

      4. Repeat step 3. Rebalance in 1 more node and load 10k more docs.
      Again many DurabilityImpossibleExceptions are seen.

      +----------------+----------+--------------+
      		 
      | Nodes          | Services | Status       |
      		 
      +----------------+----------+--------------+
      		 
      | 172.23.105.225 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.220 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.223 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.221 | [u'kv']  | Cluster node |
      		 
      | 172.23.105.226 | None     | <--- IN ---  |
      		 
      | 172.23.105.227 | None     | <--- IN ---  |
      		 
      +----------------+----------+--------------+

      Rebalance failed.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-35270 [Durability] Rebalance stuck during replica update

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35244 [Chain Rebalance]: Rebalance stuck while rebalancing in node with doc ops running in parallel having durability=majority

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35274 Rebalance In 2 nodes is struck which were hard failover and moved out from the cluster earlier

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty