Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55241

Analytics rebalance is triggered as soon as Capella cluster deployment is complete on 7.1.3

    XMLWordPrintable

Details

    Description

      Issue
      Analytics rebalance is triggered as soon as Capella cluster deployment is complete for a 7.1.3 cluster. This issue is applicable for all 3 Providers - AWS, GCP and Azure. Also, this issue is observed only on 7.1.3 clusters and not on 7.0.4 clusters.

      Activity log for 7.1.3 Capella cluster

      Cluster deployment completed at 9:25:56 GMT. Rebalance was triggered internally as soon as the cluster deployment was completed and finished in about 20 secs or so.
      From my observation, this rebalance can take upto a minute or so to complete.

      From System Event logs for 1 of the nodes present in the cluster, we can see that Analytics rebalance is getting triggered as soon as the cluster deployment is complete.

      {"timestamp":"2023-01-24T03:56:02.298Z","component":"analytics","severity":"info","event_id":5286,"description":"Analytics Cluster State Updated","extra_attributes":{"prev_state":"UNUSABLE","new_state":"PENDING"},"uuid":"67d5e61d-5b14-4f18-98a8-b8126f8fecb2","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"timestamp":"2023-01-24T03:56:02.303Z","component":"analytics","severity":"info","event_id":5286,"description":"Analytics Cluster State Updated","extra_attributes":{"prev_state":"PENDING","new_state":"RECOVERING"},"uuid":"c462315d-6cc0-4947-be49-b30c9e04b7de","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"component":"indexing","uuid":"48d8df52-7eaa-4824-a34a-fa851ae51a25","timestamp":"2023-01-24T03:56:04.066Z","sub_component":"Indexer","severity":"info","event_id":2048,"description":"Indexer Settings Changed","extra_attributes":{"group":"SettingsChange","module":"settingsManager:applySettings","old_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":false,"indexer.plasma.mainIndex.enableInMemoryCompression":false},"new_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":true,"indexer.plasma.mainIndex.enableInMemoryCompression":true}},"node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"component":"indexing","uuid":"91324f1c-0646-4c9e-a0c0-31acfda814cd","timestamp":"2023-01-24T03:56:04.069Z","sub_component":"Indexer","severity":"info","event_id":2048,"description":"Indexer Settings Changed","extra_attributes":{"group":"SettingsChange","module":"settingsManager:applySettings","old_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":false,"indexer.plasma.mainIndex.enableInMemoryCompression":false},"new_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":true,"indexer.plasma.mainIndex.enableInMemoryCompression":true}},"node":"svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"component":"indexing","uuid":"3457ecc3-a12e-4679-b08d-38da791581a9","timestamp":"2023-01-24T03:56:04.070Z","sub_component":"Indexer","severity":"info","event_id":2048,"description":"Indexer Settings Changed","extra_attributes":{"group":"SettingsChange","module":"settingsManager:applySettings","old_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":false,"indexer.plasma.mainIndex.enableInMemoryCompression":false},"new_setting":{"indexer.plasma.backIndex.enableInMemoryCompression":true,"indexer.plasma.mainIndex.enableInMemoryCompression":true}},"node":"svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"timestamp":"2023-01-24T03:56:07.285Z","event_id":2,"component":"ns_server","description":"Rebalance initiated","severity":"info","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","uuid":"610fb94d-5874-401f-b31f-9cf201d6f2d6","extra_attributes":{"operation_id":"c2ca24c2b43ba7cab3a657da9d2b731c","nodes_info":{"active_nodes":["ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com"],"keep_nodes":["ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]}}}
      		 
      {"component":"analytics","uuid":"7339dd03-3577-42c1-bddb-e9d5cd57d2b0","timestamp":"2023-01-24T03:56:07.621Z","sub_component":"cbas","severity":"info","event_id":5123,"description":"Analytics Topology Change Started","extra_attributes":{"topology":{"id":"52545dd845cb0b1ba3787f5f5a3d3143","num_eject_nodes":0,"num_keep_nodes":3,"type":"topology-change-rebalance"}},"node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"timestamp":"2023-01-24T03:56:08.343Z","component":"analytics","severity":"info","event_id":5286,"description":"Analytics Cluster State Updated","extra_attributes":{"prev_state":"RECOVERING","new_state":"ACTIVE"},"uuid":"194b1bc4-ac12-465d-9800-0473122a5078","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"timestamp":"2023-01-24T03:56:08.362Z","component":"analytics","severity":"info","event_id":5284,"description":"Analytics Partitions Topology Updated","extra_attributes":{"revision":2,"balanced":true,"num_replicas":2},"uuid":"d3be2893-cc44-436a-88e1-1e531dc031df","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"component":"analytics","uuid":"4eedb189-2b89-4865-aab0-81d06871ebc6","timestamp":"2023-01-24T03:56:14.755Z","sub_component":"cbas","severity":"info","event_id":5125,"description":"Analytics Topology Change Completed","extra_attributes":{"topology":{"id":"52545dd845cb0b1ba3787f5f5a3d3143","num_eject_nodes":0,"num_keep_nodes":3,"type":"topology-change-rebalance"}},"node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com"}
      		 
      {"timestamp":"2023-01-24T03:56:14.799Z","event_id":3,"component":"ns_server","description":"Rebalance completed","severity":"info","node":"svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","otp_node":"ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","uuid":"8e3112b5-c199-44d4-8d8f-78c6f0ed2793","extra_attributes":{"operation_id":"c2ca24c2b43ba7cab3a657da9d2b731c","nodes_info":{"active_nodes":["ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com"],"keep_nodes":["ns_1@svc-dqisa-node-001.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-002.p5b64cp7upmlir.nonprod-project-avengers.com","ns_1@svc-dqisa-node-003.p5b64cp7upmlir.nonprod-project-avengers.com"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]},"time_taken":7513,"completion_message":"Rebalance completed successfully."}}

      Same issue is not observed when 7.0.4 cluster is deployed having all services from Capella environment.

      NOTE
      This issue causes most of our CP level tests to fail with ErrClusterStateNotNormal, Cannot perform action due to the cluster not being in a Normal state error.
      Example test http://qe-jenkins1.sc.couchbase.com/job/cp-cli-provisioned-system-test/84/
      We have currently added couple of mins sleep after cluster deployment is complete in order to avoid hitting this particular issue.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sujay.gad Sujay Gad
            sujay.gad Sujay Gad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty