Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60164

[System Test][File based rebalance] :- Indexing rebalance hangs during KV swap rebalance

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • No

    Description

      Script to Repro

      ./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.0-1931 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
      

      Step at which indexing rebalance hangs

      [2023-12-17T04:38:32-08:00, sequoiatools/couchbase-cli:7.6:ea7cae] server-add -c 172.23.106.109:8091 --server-add https://172.23.124.27 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2023-12-17T04:38:54-08:00, sequoiatools/couchbase-cli:7.6:9f7b91] rebalance -c 172.23.106.109:8091 --server-remove 172.23.96.170 -u Administrator -p password
      

      Rebalance progress is hung at the same percentage for almost 12 hours.

      balakumaran.g@Balakumarans-MacBook-Pro-2 cbcollect_info_ns_1@172.23.96.203_20231215-091644 %  date; curl -u Administrator:password http://172.23.106.121:8091/pools/default/rebalanceProgress | jq   
       
      Mon Dec 18 14:01:38 IST 2023
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  1129  100  1129    0     0   1873      0 --:--:-- --:--:-- --:--:--  1891
      {
        "status": "running",
        "ns_1@172.23.96.170": {
          "progress": 1
        },
        "ns_1@172.23.106.110": {
          "progress": 1
        },
        "ns_1@172.23.121.61": {
          "progress": 1
        },
        "ns_1@172.23.96.251": {
          "progress": 0.99
        },
        "ns_1@172.23.106.121": {
          "progress": 0.99
        },
        "ns_1@172.23.97.242": {
          "progress": 1
        },
        "ns_1@172.23.96.252": {
          "progress": 1
        },
        "ns_1@172.23.96.203": {
          "progress": 0
        },
        "ns_1@172.23.105.63": {
          "progress": 0
        },
        "ns_1@172.23.97.243": {
          "progress": 1
        },
        "ns_1@172.23.104.213": {
          "progress": 0
        },
        "ns_1@172.23.96.253": {
          "progress": 0.99
        },
        "ns_1@172.23.121.94": {
          "progress": 0
        },
        "ns_1@172.23.97.244": {
          "progress": 1
        },
        "ns_1@172.23.106.124": {
          "progress": 1
        },
        "ns_1@172.23.106.164": {
          "progress": 0.99
        },
        "ns_1@172.23.97.245": {
          "progress": 0
        },
        "ns_1@172.23.104.215": {
          "progress": 0
        },
        "ns_1@172.23.97.226": {
          "progress": 0
        },
        "ns_1@172.23.96.186": {
          "progress": 0
        },
        "ns_1@172.23.120.167": {
          "progress": 0
        },
        "ns_1@172.23.104.227": {
          "progress": 0
        },
        "ns_1@172.23.124.27": {
          "progress": 1
        },
        "ns_1@172.23.121.87": {
          "progress": 0.99
        },
        "ns_1@172.23.105.237": {
          "progress": 0
        },
        "ns_1@172.23.105.238": {
          "progress": 0
        },
        "ns_1@172.23.97.229": {
          "progress": 1
        },
        "ns_1@172.23.97.189": {
          "progress": 1
        },
        "ns_1@172.23.106.109": {
          "progress": 1
        },
        "ns_1@172.23.120.59": {
          "progress": 0.99
        }
      }
      balakumaran.g@Balakumarans-MacBook-Pro-2 cbcollect_info_ns_1@172.23.96.203_20231215-091644 % 
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty