Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62677

[Mini Volume] System state is marked as corrupted after scaling of Columnar cluster

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • Unknown
    • Analytics Sprint 46

    Description

      Issue

      System state is marked as corrupted for multiple nodes in a Columnar instance after triggering a scaling operation for that particular cluster.

      For example - consider node 010, System state is marked as healthy before scaling operation.

      2024-07-10T14:07:40.867+00:00 INFO CBAS.bootstrap.NCApplication [main] System state: HEALTHY
      		 
      2024-07-10T14:07:40.867+00:00 INFO CBAS.bootstrap.NCApplication [main] Node ID: 43186f0aaf75b714f25d14d7b125168a

       

      Scaling operation is triggered reducing the number of nodes in a Columnar instance from 16 -> 8.

      2024-07-10T14:11:58.957Z, ns_orchestrator:0:info:message(ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com) - Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-005.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-008.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-009.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-010.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-012.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-014.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                       'ns_1@svc-da-node-016.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com'], EjectNodes = ['ns_1@svc-da-node-001.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-004.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-006.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-007.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-011.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-013.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com',
      		 
                                                                                                                                    'ns_1@svc-da-node-015.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 6e2b484b00870dd3472762b3102289d4


      2024-07-10T14:12:11.093Z, ns_orchestrator:0:info:message(ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com) - Rebalance completed successfully.
      		 
      Rebalance Operation Id = 6e2b484b00870dd3472762b3102289d4

       

      System state is marked as corrupted when Analytics driver is restarted after completion of scaling operation.

      2024-07-10T14:12:14.812+00:00 INFO CBAS.bootstrap.NCApplication [main] System state: CORRUPTED
      		 
      2024-07-10T14:12:14.812+00:00 INFO CBAS.bootstrap.NCApplication [main] Node ID: 43186f0aaf75b714f25d14d7b125168a

      Note

      This issue is seen for Columnar instance 2203 which contains the fix for MB-62615, hence filing a new ticket.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sujay.gad Sujay Gad
            sujay.gad Sujay Gad
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty