Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3597

Failed over pods are recreated with older version instead of upgrade version during failover/delta recovery upgrade

    XMLWordPrintable

Details

    • 18 -Lost to Eternity
    • 1

    Description

      Cluster Setup

      • Kind cluster locally run on Mac
      • 6 nodes in cluster
      • 9 buckets
      • Initial Cluster version : 7.6.0-2176
      • Upgrade Cluster version : 7.6.1-3200

      Steps taken in the scenario

      • Created a cluster
      • Created 9 buckets
      • Issued an upgrade with delta recovery
      • When one of the pods(cb-example-0001) was picked up for failover, manually deleted 2 other pods

      $kubectl delete pod cb-example-0004
      		 
      $kubectl delete pod cb-example-0005

      • The pods are failed over and then operator attempts to recreate it back
      • Instead of recreating with upgrade version 7.6.1, the pods are created with 7.6.0.
      • The upgrade then continues from cb-example-0001.
      • Then later cb-example-0004 and cb-example-0005 are upgraded as well.

      Issue

      • The operator should create pods with upgrade version instead of older version

      {"level":"info","ts":"2024-07-25T05:08:29Z","logger":"cluster","msg":"Pods failed over","cluster":"default/cb-example"}
      		 
      {"level":"info","ts":"2024-07-25T05:08:29Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0004","image":"couchbase/server:7.6.0"}
      		 
       
      		 
      {"level":"info","ts":"2024-07-25T05:09:10Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0005","image":"couchbase/server:7.6.0"}

      Operator logs : 
      https://cb-engineering.s3.amazonaws.com/K8S-3597/cbopinfo-20240725T111733+0530.tar.gz

      Cluster logs : 

      https://cb-engineering.s3.amazonaws.com/K8S-3598/collectinfo-2024-07-25T114628-ns_1%40cb-example-0005.cb-example.default.svc.zip

      https://cb-engineering.s3.amazonaws.com/K8S-3598/collectinfo-2024-07-25T114628-ns_1%40cb-example-0001.cb-example.default.svc.zip

       The cao tool and operator images were built locally on this commit

      commit c2e920ddbcfa9b4819d47ad81d0a35c359dd1dc6 (HEAD -> master, origin/master, origin/HEAD)
      		 
      Author: usamah jassat <usamah.jassat@couchbase.com>
      		 
      Date:   Wed Jul 17 15:11:19 2024 +0100    K8S-3581: don't attempt backend migration when rebalance required
      		 
          
      		 
          Change-Id: I2d2b6d6d4f8dbb0a30db5bd54a05631d17631eee
      		 
          Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/212890
      		 
          Reviewed-by: Yusuf Ramzan <yusuf.ramzan@couchbase.com>
      		 
          Tested-by: Build Bot <build@couchbase.com>

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            usamah.jassat Usamah Jassat
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty