Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3494

[Operator 2.6.4] Required a break point to stop infinite rebalances which can lead to potential cascading failures.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.6.4
    • 2.6.4
    • operator
    • None
    • 0

    Description

      Couchbase Cluster Description

      • Set up the cluster as per the required specifications
      • Each node is an m5.4xlarge instance. (16 vCPUs and 64GB RAM) 
      • 6 Data Service, 4 Index Service and Query Service Nodes.
      • 10 Buckets (with 1 replica), Full Eviction and Auto-failover set to 5s.
      • ~210GB data per bucket → ~2TB data loaded onto cluster.
      • 50 Primary Indexes with 1 Replica each. (Total 100 Indexes)
      • DeltaRecovery Upgrade to update Couchbase Server from 7.2.5 to 7.6.1
      • Continuous data and query workload on all buckets during the update process.

      Running on 20-25 % CPU load
      Running with 50-60% RAM consumption.

      With current CPU and memory consumption, the cluster is managing to stay in healthy condition but with higher load, infinite rebalance can huge pressure on cluster potentially leading to cascading failures.



      Attachments

        1. Screenshot 2024-05-20 at 3.22.15 PM.png
          929 kB
          Manik Mahajan
        2. Screenshot 2024-05-20 at 4.42.45 PM.png
          1.05 MB
          Manik Mahajan
        3. Screenshot 2024-05-20 at 4.47.54 PM.png
          1.06 MB
          Manik Mahajan

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              justin.ashworth Justin Ashworth
              manik.mahajan Manik Mahajan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty