Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41801

Reported Resident Ratio drops low after graceful failover and recovery

    XMLWordPrintable

Details

    Description

      Summary:
      Resident ratio dropped low on nodes that were failovered and recovered.

      Steps to Reproduce:
      1. Create a 6 node cluster
      2020-10-01 03:24:03,920 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview
      -----------------------++-------------

      Nodes Services Status

      -----------------------++-------------

      172.23.105.211 kv Cluster node
      172.23.105.212 None <--- IN —
      172.23.105.213 None <--- IN —
      172.23.105.215 None <--- IN —
      172.23.105.217 None <--- IN —
      172.23.105.219 None <--- IN —

      -----------------------++-------------
      2. Create a single bucket with 1000 collections and a total of 10k items
      -------------------------+----------------------------------------------------+----------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      -------------------------+----------------------------------------------------+----------

      3Y2OCngAZQmemyo couchbase 2 none 0 10000 12582912000 957175360 635367063

      -------------------------+----------------------------------------------------+----------
      3. Drop 400 collections and recreate them whilst doing the below steps (no more docs are created, and number of items get reduced as collections are dropped )

      4. Failover .211 and .212
      2020-10-01 03:32:50,365 | test | INFO | MainThread | [collections_drop_recreate_rebalance:wait_for_failover_or_assert:53] 2 nodes failed over as expected in 0.0750000476837 seconds

      5. Delta recover and rebalance
      2020-10-01 03:32:50,500 | test | INFO | pool-1-thread-28 | [table_view:display:72] Rebalance Overview
      -----------------------++-------------

      Nodes Services Status

      -----------------------++-------------

      172.23.105.215 kv Cluster node
      172.23.105.212 kv Cluster node
      172.23.105.213 kv Cluster node
      172.23.105.217 kv Cluster node
      172.23.105.219 kv Cluster node
      172.23.105.211 kv Cluster node

      -----------------------++-------------
      Now the resident ratio drops low as seen in the screenshot.
      from supportal logs, the low resident ratio is on the .211 and .212 nodes

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jwalker Jim Walker
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty