Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59723

[System Test] : indexing rebalance progress fluctuates up and down

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • No

    Description

      ./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.0-1820 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
      

      Rebalance progress

      balakumaran.g@Balakumarans-MacBook-Pro-2 TAF %  date; curl -u Administrator:password http://172.23.96.253:8091/pools/default/rebalanceProgress | jq | grep -A 2 '0.9'
       
      Mon Nov 20 17:15:56 IST 2023
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  1176  100  1176    0     0   1538      0 --:--:-- --:--:-- --:--:--  1551
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.106.121": {
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.121.72": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.96.203": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.121.94": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.97.245": {
      --
          "progress": 0.9689361702127659
        }
      }
      balakumaran.g@Balakumarans-MacBook-Pro-2 TAF %  date; curl -u Administrator:password http://172.23.96.253:8091/pools/default/rebalanceProgress | jq | grep -A 2 '0.9'
       
      Mon Nov 20 17:15:59 IST 2023
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  1176  100  1176    0     0   1696      0 --:--:-- --:--:-- --:--:--  1714
          "progress": 0.9997872340425532
        },
        "ns_1@172.23.106.121": {
          "progress": 0.9997872340425532
        },
        "ns_1@172.23.121.72": {
      --
          "progress": 0.9997872340425532
        },
        "ns_1@172.23.96.203": {
      --
          "progress": 0.9997872340425532
        },
        "ns_1@172.23.121.94": {
      --
          "progress": 0.9997872340425532
        },
        "ns_1@172.23.97.245": {
      --
          "progress": 0.9997872340425532
        }
      }
      balakumaran.g@Balakumarans-MacBook-Pro-2 TAF %  date; curl -u Administrator:password http://172.23.96.253:8091/pools/default/rebalanceProgress | jq | grep -A 2 '0.9'
       
      Mon Nov 20 17:16:05 IST 2023
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  1176  100  1176    0     0    758      0  0:00:01  0:00:01 --:--:--   761
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.106.121": {
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.121.72": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.96.203": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.121.94": {
      --
          "progress": 0.9689361702127659
        },
        "ns_1@172.23.97.245": {
      --
          "progress": 0.9689361702127659
        }
      }
      

      From the above stats you can see

      at Nov 20 17:15:56 IST 2023 - Indexing progress for 172.23.106.121 was 0.9689361702127659
      at Nov 20 17:15:59 IST 2023 - Indexing progress for 172.23.106.121 was 0.9997872340425532
      at Nov 20 17:16:05 IST 2023 - Indexing progress for 172.23.106.121 was 0.9689361702127659

      I saw this fluctuation for almost 15-20 mins for rebalanceId :- c02364a302d9abea71bc0c999f0d792d. This doesn't affect any tests but could cause confusion in customer environments if they are tracking rebalance progress. I have been running system tests for a while and haven't noticed this until very recently, so very well could be related to file based rebalance(thought not 100% sure).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            harikishore.chaparala Hari Kishore (Inactive)
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty