Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3491

Swap Rebalance Upgrade CB 7.2.5 to 7.6.1 with K8s 1.25 and CAO 2.6.4

    XMLWordPrintable

Details

    • Task
    • Resolution: Resolved
    • Major
    • 2.6.4
    • None
    • None
    • None

    Description

      Update 2024-06-05

      Tests Completed

      Date CAO Build Status
      2024-05-18 2.6.4-108 Fail
      2024-05-22 2.6.4-111 Pass
      2024-05-29 2.6.4-119 Pass
      2024-06-04 2.6.4-122 Pass

      Update 2024-05-29

      Performing Swap Rebalance Upgrade

      Kubernetes Version 1.25 EKS
      Couchbase Server 7.2.5 → 7.6.1
      Operator 2.6.4-119

      Cluster Setup

      • Each node is an m5.4xlarge instance. (16 vCPUs and 64GB RAM) 
      • 6 Data Service, 4 Index Service & Query Service Nodes.
      • 10 Buckets (with 3 replica), Full Eviction and Auto-failover set to 5s.
      • 125GB data per bucket → ~1.25TB data loaded onto cluster before beginning of upgrade.
      • 50 Primary Indexes with 1 Replica each. (Total 100 Indexes with Index Storage: Plasma)

      Upgrade Process

      • SwapRebalance Upgrade to update Couchbase Server from 7.2.5 to 7.6.1.
      • Continuous query and data workload on the buckets during the update process.
      • Around 35-50% CPU load on all servers during the upgrade.

      Swap Rebalance Upgrade was successful

      Logs

      Before the Upgrade

      http://supportal.couchbase.com/snapshot/ca8f26dd83c1f592a980665e9c78360e::0

      2024-05-29A_PreSwapRebalance_cbopinfo-20240529T110501+0530.tar.gz

      During the Upgrade

      http://supportal.couchbase.com/snapshot/ca8f26dd83c1f592a980665e9c78360e::1

      2024-05-29A_DuringSwapRebalance_cbopinfo-20240529T183603+0530.tar.gz

      After the Upgrade

      http://supportal.couchbase.com/snapshot/ca8f26dd83c1f592a980665e9c78360e::2

      2024-05-29A_PostSwapRebalance_cbopinfo-20240529T195359+0530.tar.gz

       

       

      Attachments

        1. 2024-05-21_K8s_1.25_CB_7.2.5_PreSwapRebalance_cbopinfo-20240521T125228+0530.tar.gz
          1.18 MB
          Aryaan Bhaskar
        2. 2024-05-29A_DuringSwapRebalance_cbopinfo-20240529T183603+0530.tar.gz
          1.77 MB
          Aryaan Bhaskar
        3. 2024-05-29A_PostSwapRebalance_cbopinfo-20240529T195359+0530.tar.gz
          1.92 MB
          Aryaan Bhaskar
        4. 2024-05-29A_PreSwapRebalance_cbopinfo-20240529T110501+0530.tar.gz
          1.18 MB
          Aryaan Bhaskar
        5. 2024-06-04A_PostSwapUpgrade_cbopinfo-20240605T100524+0530.tar.gz
          4.45 MB
          Aryaan Bhaskar
        6. 2024-06-04A_PreSwapUpgrade_cbopinfo-20240604T105157+0530.tar.gz
          2.78 MB
          Aryaan Bhaskar
        7. cb_7.6.1_during_swap_rebalance_query_index_upgrades_cbopinfo-20240518T020924+0530.tar.gz
          2.65 MB
          Aryaan Bhaskar
        8. image-2024-05-18-02-22-52-798.png
          873 kB
          Aryaan Bhaskar
        9. image-2024-05-18-02-24-01-207.png
          779 kB
          Aryaan Bhaskar
        10. image-2024-05-18-02-25-07-321.png
          923 kB
          Aryaan Bhaskar
        11. image-2024-05-18-02-26-09-622.png
          958 kB
          Aryaan Bhaskar
        12. image-2024-05-18-02-26-38-122.png
          997 kB
          Aryaan Bhaskar
        13. image-2024-05-22-11-09-11-279.png
          527 kB
          Aryaan Bhaskar
        14. image-2024-05-22-11-09-35-939.png
          488 kB
          Aryaan Bhaskar
        15. K8s_1.25_CB_7.2.5_PreSwapRebalanceUpgrade_cbopinfo-20240517T154713+0530.tar.gz
          1.88 MB
          Aryaan Bhaskar

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              aryaan.bhaskar Aryaan Bhaskar
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty