Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48458

[Volume Test] : Seeing higher rebalance times compared to 7.0.0 when scan & i/o utilisation workloads are skewed

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Build : 7.0.2-6667
      Test : -test tests/2i/cheshirecat/test_idx_cc_vol_10K.yml -scope tests/2i/cheshirecat/scope_idx_cc_vol_10K.yml
      Scale : 5

      For the 10K indexes Volume test (plasma) with 7.0.2-6667, the rebalance times are higher than the run that was done with 7.0.0-5170. Overall, 8 rebalance operations in 7.0.0-5170 took ~15 hrs, while in 7.0.2, it is taking ~28 hrs.

      Two rebalance especially are taking 9 hrs and 6 hrs. The higher rebalance time should be investigated.

      Rebalance # Rebalance Type Start Time End Time Duration in 7.0.2-6667 Duration in 7.0.0-5170 Delta
      1 Add idx node 2021-09-11T00:41:59 2021-09-11T01:37:16 0:56 1:45  -0:49
      2 Remove idx node 2021-09-11T01:55:07 2021-09-11T11:06:05 9:11 1:05  +8:06
      3 Swap rebalance Idx node 2021-09-11T11:24:17 2021-09-11T13:49:54 2:25 1:21  +1:04
      4 Remove KV node 2021-09-11T14:37:18 2021-09-11T20:38:24 6:01 0:29  +4:32
      5 Add KV node 2021-09-11T20:55:59 2021-09-11T20:58:20 0:03 0:55  -0:52
      6 Failover-recovery-addback idx 2021-09-11T21:15:58 2021-09-12T00:59:15 3:44 2:44  +1:00
      7 Failover-remove idx node 2021-09-12T01:26:37 2021-09-12T05:21:15 3:56 2:31  +1:25
      8 Add idx node 2021-09-12T05:33:03 2021-09-12T08:35:02 3:02 4:04  -1:02
      TOTAL       28:18 14:54  +13:24

      Logs :

      Taken after end of rebalance # 2
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631383660/collectinfo-2021-09-11T182656-ns_1%40172.23.99.25.zip

      Taken after end of rebalance # 4
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631421170/collectinfo-2021-09-12T045208-ns_1%40172.23.99.21.zip

      If you need logs from any specific time, pls let me know. 

      This is a regression from 7.0.0-5170. This test did not completely run on 7.0.2 previously because rebalances were failing due to MB-47600.

      Attachments

        Issue Links

          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. MB-48497 Index Rebalance improvements

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              yogendra.acharya Yogendra Acharya (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty