Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60023

[System Test] Rebalance failure - inactivity_timeout

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Rebalance failure because of inactivity timeout. All the subsequent rebalances have also failed.

      {"stageInfo":{"index":{"totalProgress":39.20855119823854,"perNodeProgress":{"ns_1@172.23.106.30":0.3920855119823854,"ns_1@172.23.97.108":0.3920855119823854,"ns_1@172.23.97.109":0.3920855119823854},"startTime":"2023-12-06T17:32:01.078-08:00","completedTime":false,"timeTaken":1301665},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.96.230":1,"ns_1@172.23.96.245":1,"ns_1@172.23.97.66":1,"ns_1@172.23.96.198":1},"startTime":"2023-12-06T17:31:58.033-08:00","completedTime":"2023-12-06T17:32:01.068-08:00","timeTaken":3035},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.97.100":1,"ns_1@172.23.97.67":1},"startTime":"2023-12-06T17:32:01.069-08:00","completedTime":"2023-12-06T17:32:01.078-08:00","timeTaken":9}},"rebalanceId":"3a3507eb6c53427ead6b7fb93841afd7","nodesInfo":{"active_nodes":["ns_1@172.23.106.30","ns_1@172.23.96.198","ns_1@172.23.96.230","ns_1@172.23.96.245","ns_1@172.23.97.100","ns_1@172.23.97.108","ns_1@172.23.97.109","ns_1@172.23.97.66","ns_1@172.23.97.67"],"keep_nodes":["ns_1@172.23.106.30","ns_1@172.23.96.198","ns_1@172.23.96.230","ns_1@172.23.96.245","ns_1@172.23.97.100","ns_1@172.23.97.108","ns_1@172.23.97.109","ns_1@172.23.97.66","ns_1@172.23.97.67"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.97.67","startTime":"2023-12-06T17:31:54.727-08:00","completedTime":"2023-12-06T17:53:42.744-08:00","timeTaken":1308016,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,index,\n                              {worker_died,\n                               {'EXIT',<0.13379.191>,\n                                {task_failed,rebalance,inactivity_timeout}}}}."}
      

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701916923/collectinfo-2023-12-07T025059-ns_1%40172.23.97.67.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty