Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60358

[System Test] [File based rebalance] : Indexing rebalance progress stuck at a % for over 10 hours(possibly hung)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.0
    • 7.6.0
    • qe

    • Enterprise Edition 7.6.0 build 1987
    • Untriaged
    • Centos 64-bit
    • 0
    • Unknown

    Description

      Script to Repro

      ./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.0-1987 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
      

      Step at which it hangs

      [2024-01-11T16:23:58-08:00, sequoiatools/couchbase-cli:7.6:0daf19] rebalance -c 172.23.106.109:8091 --server-remove 172.23.96.252 -u Administrator -p password
      

      I saw the indexing rebalance was stuck at same % for over 10 hours

      balakumaran.g@Balakumarans-MacBook-Pro-2 cbcollect_info_ns_1@172.23.96.203_20231215-091644 %  date; curl -u Administrator:password http://172.23.106.121:8091/pools/default/rebalanceProgress | jq
                      
      Fri Jan 12 17:32:15 IST 2024
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  1087  100  1087    0     0   1987      0 --:--:-- --:--:-- --:--:--  2012
      {
        "status": "running",
        "ns_1@172.23.96.170": {
          "progress": 1
        },
        "ns_1@172.23.106.110": {
          "progress": 1
        },
        "ns_1@172.23.106.121": {
          "progress": 0.9600292968749999
        },
        "ns_1@172.23.121.72": {
          "progress": 0
        },
        "ns_1@172.23.97.242": {
          "progress": 1
        },
        "ns_1@172.23.96.252": {
          "progress": 0.9600292968749999
        },
        "ns_1@172.23.96.203": {
          "progress": 0
        },
        "ns_1@172.23.105.63": {
          "progress": 0
        },
        "ns_1@172.23.97.243": {
          "progress": 1
        },
        "ns_1@172.23.104.213": {
          "progress": 0
        },
        "ns_1@172.23.96.253": {
          "progress": 0.9600292968749999
        },
        "ns_1@172.23.121.94": {
          "progress": 0
        },
        "ns_1@172.23.106.124": {
          "progress": 0
        },
        "ns_1@172.23.97.244": {
          "progress": 1
        },
        "ns_1@172.23.106.164": {
          "progress": 0.9600292968749999
        },
        "ns_1@172.23.97.245": {
          "progress": 0
        },
        "ns_1@172.23.104.215": {
          "progress": 0
        },
        "ns_1@172.23.97.226": {
          "progress": 0
        },
        "ns_1@172.23.96.186": {
          "progress": 1
        },
        "ns_1@172.23.121.87": {
          "progress": 0
        },
        "ns_1@172.23.104.227": {
          "progress": 0
        },
        "ns_1@172.23.105.237": {
          "progress": 0
        },
        "ns_1@172.23.105.238": {
          "progress": 0
        },
        "ns_1@172.23.97.229": {
          "progress": 1
        },
        "ns_1@172.23.97.189": {
          "progress": 1
        },
        "ns_1@172.23.106.109": {
          "progress": 1
        },
        "ns_1@172.23.120.59": {
          "progress": 0.9600292968749999
        }
      }
      balakumaran.g@Balakumarans-MacBook-Pro-2
      

      cbcollect_info attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty