Loading...

Details

Type: Bug
Resolution: Cannot Reproduce
Priority: Critical
Fix Version/s: None
Affects Version/s: 5.5.0
Component/s: couchbase-bucket
Labels:
Environment:
CentOS 7
E5-2680 v3 (48 vCPU)
64 GB RAM
Samsung PM863a SATA SSD

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.100.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.101.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.102.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.103.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.104.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.105.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.106.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.107.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.108.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.109.zip

Show
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.100.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.101.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.102.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.103.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.104.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.105.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.106.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.107.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.108.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-30/172.23.96.109.zip
Is this a Regression?:
Yes

Description

Test scenario:

9 nodes
1 bucket, 1 replica, full eviction
1B items (~1KB), 5-10% resident ratio
15K ops/sec (90% read, 10% update), 10% cache miss ratio (before rebalance)
Swap rebalance of one node (172.23.96.108 -> 172.23.96.109)

I noticed this issue when I was running another test with 10TB of data. Later I realized that high latency can also be observed during a very simple test with just 1TB.

Latency on *.100 (doesn't participate in rebalance):

The following data is collected for "GET"

[   0 -    9]us ( 50.89%)  4217092 | ############################################

[  10 -   19]us ( 90.10%)  3250224 | #################################

[  20 -   29]us ( 91.30%)    98719 | #

[  30 -   39]us ( 91.37%)     6069 |

[  40 -   49]us ( 91.37%)      424 |

[  50 -   59]us ( 91.37%)       48 |

[  60 -   69]us ( 91.38%)       56 |

[  70 -   79]us ( 91.38%)       60 |

[  80 -   89]us ( 91.38%)       61 |

[  90 -   99]us ( 91.38%)       73 |

[ 100 -  109]us ( 91.38%)      102 |

[ 110 -  119]us ( 91.38%)      166 |

[ 120 -  129]us ( 91.38%)      231 |

[ 130 -  139]us ( 91.39%)      371 |

[ 140 -  149]us ( 91.39%)      467 |

[ 150 -  159]us ( 91.40%)      546 |

[ 160 -  169]us ( 91.41%)      617 |

[ 170 -  179]us ( 91.42%)      702 |

[ 180 -  189]us ( 91.42%)      626 |

[ 190 -  199]us ( 91.43%)      624 |

[ 200 -  209]us ( 91.44%)      678 |

[ 210 -  219]us ( 91.45%)      570 |

[ 220 -  229]us ( 91.45%)      636 |

[ 230 -  239]us ( 91.46%)      597 |

[ 240 -  249]us ( 91.47%)      762 |

[ 250 -  259]us ( 91.48%)      937 |

[ 260 -  269]us ( 91.50%)     1166 |

[ 270 -  279]us ( 91.51%)     1576 |

[ 280 -  289]us ( 91.54%)     2018 |

[ 290 -  299]us ( 91.57%)     2556 |

[ 300 -  309]us ( 91.61%)     3125 |

[ 310 -  319]us ( 91.65%)     3835 |

[ 320 -  329]us ( 91.71%)     4519 |

[ 330 -  339]us ( 91.77%)     4924 |

[ 340 -  349]us ( 91.83%)     5432 |

[ 350 -  359]us ( 91.90%)     5850 |

[ 360 -  369]us ( 91.98%)     6133 |

[ 370 -  379]us ( 92.06%)     6466 |

[ 380 -  389]us ( 92.14%)     6818 |

[ 390 -  399]us ( 92.23%)     7189 |

[ 400 -  409]us ( 92.32%)     7583 |

[ 410 -  419]us ( 92.41%)     8088 |

[ 420 -  429]us ( 92.52%)     8448 |

[ 430 -  439]us ( 92.62%)     8663 |

[ 440 -  449]us ( 92.73%)     9060 |

[ 450 -  459]us ( 92.84%)     9393 |

[ 460 -  469]us ( 92.96%)     9531 |

[ 470 -  479]us ( 93.07%)     9493 |

[ 480 -  489]us ( 93.19%)     9905 |

[ 490 -  499]us ( 93.32%)    10279 |

[ 500 -  509]us ( 93.45%)    10655 |

[ 510 -  519]us ( 93.58%)    11191 |

[ 520 -  529]us ( 93.72%)    11343 |

[ 530 -  539]us ( 93.86%)    11940 |

[ 540 -  549]us ( 94.01%)    12545 |

[ 550 -  559]us ( 94.17%)    12764 |

[ 560 -  569]us ( 94.33%)    13374 |

[ 570 -  579]us ( 94.49%)    13636 |

[ 580 -  589]us ( 94.66%)    13890 |

[ 590 -  599]us ( 94.83%)    13867 |

[ 600 -  609]us ( 94.99%)    13695 |

[ 610 -  619]us ( 95.16%)    13473 |

[ 620 -  629]us ( 95.32%)    13360 |

[ 630 -  639]us ( 95.47%)    12884 |

[ 640 -  649]us ( 95.62%)    12226 |

[ 650 -  659]us ( 95.76%)    11402 |

[ 660 -  669]us ( 95.89%)    11039 |

[ 670 -  679]us ( 96.02%)    10524 |

[ 680 -  689]us ( 96.14%)    10039 |

[ 690 -  699]us ( 96.25%)     9660 |

[ 700 -  709]us ( 96.37%)     9435 |

[ 710 -  719]us ( 96.48%)     9184 |

[ 720 -  729]us ( 96.59%)     9100 |

[ 730 -  739]us ( 96.70%)     8982 |

[ 740 -  749]us ( 96.80%)     8784 |

[ 750 -  759]us ( 96.91%)     8742 |

[ 760 -  769]us ( 97.01%)     8504 |

[ 770 -  779]us ( 97.11%)     8284 |

[ 780 -  789]us ( 97.21%)     8055 |

[ 790 -  799]us ( 97.30%)     7760 |

[ 800 -  809]us ( 97.39%)     7497 |

[ 810 -  819]us ( 97.48%)     7419 |

[ 820 -  829]us ( 97.57%)     6902 |

[ 830 -  839]us ( 97.65%)     6584 |

[ 840 -  849]us ( 97.72%)     6272 |

[ 850 -  859]us ( 97.79%)     5946 |

[ 860 -  869]us ( 97.86%)     5669 |

[ 870 -  879]us ( 97.92%)     5194 |

[ 880 -  889]us ( 97.99%)     5089 |

[ 890 -  899]us ( 98.04%)     4728 |

[ 900 -  909]us ( 98.10%)     4630 |

[ 910 -  919]us ( 98.15%)     4386 |

[ 920 -  929]us ( 98.20%)     4211 |

[ 930 -  939]us ( 98.25%)     4081 |

[ 940 -  949]us ( 98.30%)     3908 |

[ 950 -  959]us ( 98.34%)     3770 |

[ 960 -  969]us ( 98.39%)     3590 |

[ 970 -  979]us ( 98.43%)     3479 |

[ 980 -  989]us ( 98.47%)     3432 |

[ 990 -  999]us ( 98.51%)     3161 |

[   1 -    1]ms ( 99.49%)    81667 |

[   2 -    2]ms ( 99.74%)    20163 |

[   3 -    3]ms ( 99.82%)     6470 |

[   4 -    4]ms ( 99.84%)     2055 |

[   5 -    5]ms ( 99.85%)      733 |

[   6 -    6]ms ( 99.85%)      480 |

[   7 -    7]ms ( 99.86%)      497 |

[   8 -    8]ms ( 99.87%)      558 |

[   9 -    9]ms ( 99.87%)      490 |

[  10 -   10]ms ( 99.88%)      495 |

[  11 -   11]ms ( 99.89%)      490 |

[  12 -   12]ms ( 99.89%)      498 |

[  13 -   13]ms ( 99.90%)      517 |

[  14 -   14]ms ( 99.90%)      534 |

[  15 -   15]ms ( 99.91%)      531 |

[  16 -   16]ms ( 99.92%)      601 |

[  17 -   17]ms ( 99.92%)      522 |

[  18 -   18]ms ( 99.93%)      460 |

[  19 -   19]ms ( 99.93%)      450 |

[  20 -   20]ms ( 99.94%)      458 |

[  21 -   21]ms ( 99.95%)      467 |

[  22 -   22]ms ( 99.95%)      429 |

[  23 -   23]ms ( 99.96%)      393 |

[  24 -   24]ms ( 99.96%)      432 |

[  25 -   25]ms ( 99.97%)      351 |

[  26 -   26]ms ( 99.97%)      358 |

[  27 -   27]ms ( 99.97%)      342 |

[  28 -   28]ms ( 99.98%)      333 |

[  29 -   29]ms ( 99.98%)      329 |

[  30 -   30]ms ( 99.99%)      290 |

[  31 -   31]ms ( 99.99%)      345 |

[  32 -   32]ms ( 99.99%)      280 |

[  33 -   33]ms (100.00%)      205 |

[  34 -   34]ms (100.00%)      129 |

[  35 -   35]ms (100.00%)       64 |

[  36 -   36]ms (100.00%)       37 |

[  37 -   37]ms (100.00%)       27 |

[  38 -   38]ms (100.00%)       22 |

[  39 -   39]ms (100.00%)       13 |

[  40 -   40]ms (100.00%)        6 |

[  41 -   41]ms (100.00%)       10 |

[  42 -   42]ms (100.00%)       10 |

[  43 -   43]ms (100.00%)       14 |

[  44 -   44]ms (100.00%)        4 |

[  45 -   45]ms (100.00%)        1 |

[  46 -   46]ms (100.00%)        4 |

[  47 -   47]ms (100.00%)        1 |

[  48 -   48]ms (100.00%)        1 |

[  49 -   49]ms (100.00%)        3 |

[  50 -  499]ms (100.00%)       39 |

Total: 8287401 operations

Latency on *.108 (swapped out):

The following data is collected for "GET"

[   0 -    9]us ( 53.71%)  2547731 | ############################################

[  10 -   19]us ( 90.02%)  1722588 | #############################

[  20 -   29]us ( 91.19%)    55079 |

[  30 -   39]us ( 91.26%)     3705 |

[  40 -   49]us ( 91.27%)      292 |

[  50 -   59]us ( 91.27%)       40 |

[  60 -   69]us ( 91.27%)       23 |

[  70 -   79]us ( 91.27%)       30 |

[  80 -   89]us ( 91.27%)       35 |

[  90 -   99]us ( 91.27%)       22 |

[ 100 -  109]us ( 91.27%)       16 |

[ 110 -  119]us ( 91.27%)       45 |

[ 120 -  129]us ( 91.28%)       68 |

[ 130 -  139]us ( 91.28%)       96 |

[ 140 -  149]us ( 91.28%)      149 |

[ 150 -  159]us ( 91.28%)      190 |

[ 160 -  169]us ( 91.29%)      247 |

[ 170 -  179]us ( 91.30%)      326 |

[ 180 -  189]us ( 91.31%)      408 |

[ 190 -  199]us ( 91.32%)      498 |

[ 200 -  209]us ( 91.33%)      604 |

[ 210 -  219]us ( 91.34%)      698 |

[ 220 -  229]us ( 91.36%)      838 |

[ 230 -  239]us ( 91.38%)     1009 |

[ 240 -  249]us ( 91.41%)     1279 |

[ 250 -  259]us ( 91.44%)     1374 |

[ 260 -  269]us ( 91.47%)     1329 |

[ 270 -  279]us ( 91.49%)     1272 |

[ 280 -  289]us ( 91.52%)     1225 |

[ 290 -  299]us ( 91.54%)     1151 |

[ 300 -  309]us ( 91.57%)     1105 |

[ 310 -  319]us ( 91.59%)     1128 |

[ 320 -  329]us ( 91.61%)     1137 |

[ 330 -  339]us ( 91.64%)     1235 |

[ 340 -  349]us ( 91.67%)     1257 |

[ 350 -  359]us ( 91.70%)     1503 |

[ 360 -  369]us ( 91.73%)     1641 |

[ 370 -  379]us ( 91.77%)     1702 |

[ 380 -  389]us ( 91.81%)     2099 |

[ 390 -  399]us ( 91.86%)     2341 |

[ 400 -  409]us ( 91.92%)     2646 |

[ 410 -  419]us ( 91.98%)     2986 |

[ 420 -  429]us ( 92.05%)     3334 |

[ 430 -  439]us ( 92.12%)     3482 |

[ 440 -  449]us ( 92.20%)     3769 |

[ 450 -  459]us ( 92.29%)     3847 |

[ 460 -  469]us ( 92.37%)     3958 |

[ 470 -  479]us ( 92.45%)     3958 |

[ 480 -  489]us ( 92.54%)     3971 |

[ 490 -  499]us ( 92.62%)     3985 |

[ 500 -  509]us ( 92.71%)     4200 |

[ 510 -  519]us ( 92.80%)     4189 |

[ 520 -  529]us ( 92.89%)     4368 |

[ 530 -  539]us ( 92.99%)     4817 |

[ 540 -  549]us ( 93.10%)     5023 |

[ 550 -  559]us ( 93.21%)     5320 |

[ 560 -  569]us ( 93.33%)     5623 |

[ 570 -  579]us ( 93.45%)     6012 |

[ 580 -  589]us ( 93.59%)     6255 |

[ 590 -  599]us ( 93.72%)     6545 |

[ 600 -  609]us ( 93.86%)     6645 |

[ 610 -  619]us ( 94.01%)     6786 |

[ 620 -  629]us ( 94.15%)     6617 |

[ 630 -  639]us ( 94.28%)     6308 |

[ 640 -  649]us ( 94.41%)     6260 |

[ 650 -  659]us ( 94.54%)     5943 |

[ 660 -  669]us ( 94.66%)     5793 |

[ 670 -  679]us ( 94.78%)     5752 |

[ 680 -  689]us ( 94.90%)     5532 |

[ 690 -  699]us ( 95.01%)     5550 |

[ 700 -  709]us ( 95.13%)     5480 |

[ 710 -  719]us ( 95.24%)     5461 |

[ 720 -  729]us ( 95.36%)     5682 |

[ 730 -  739]us ( 95.48%)     5541 |

[ 740 -  749]us ( 95.60%)     5600 |

[ 750 -  759]us ( 95.72%)     5602 |

[ 760 -  769]us ( 95.84%)     5609 |

[ 770 -  779]us ( 95.95%)     5516 |

[ 780 -  789]us ( 96.07%)     5614 |

[ 790 -  799]us ( 96.19%)     5480 |

[ 800 -  809]us ( 96.30%)     5231 |

[ 810 -  819]us ( 96.40%)     5054 |

[ 820 -  829]us ( 96.50%)     4835 |

[ 830 -  839]us ( 96.60%)     4697 |

[ 840 -  849]us ( 96.70%)     4469 |

[ 850 -  859]us ( 96.79%)     4296 |

[ 860 -  869]us ( 96.88%)     4182 |

[ 870 -  879]us ( 96.96%)     3762 |

[ 880 -  889]us ( 97.03%)     3712 |

[ 890 -  899]us ( 97.11%)     3583 |

[ 900 -  909]us ( 97.18%)     3480 |

[ 910 -  919]us ( 97.26%)     3483 |

[ 920 -  929]us ( 97.33%)     3428 |

[ 930 -  939]us ( 97.40%)     3284 |

[ 940 -  949]us ( 97.47%)     3265 |

[ 950 -  959]us ( 97.53%)     3122 |

[ 960 -  969]us ( 97.60%)     3078 |

[ 970 -  979]us ( 97.66%)     2982 |

[ 980 -  989]us ( 97.72%)     2945 |

[ 990 -  999]us ( 97.78%)     2684 |

[   1 -    1]ms ( 99.21%)    67919 | #

[   2 -    2]ms ( 99.44%)    10721 |

[   3 -    3]ms ( 99.54%)     4728 |

[   4 -    4]ms ( 99.58%)     2083 |

[   5 -    5]ms ( 99.60%)     1035 |

[   6 -    6]ms ( 99.62%)      747 |

[   7 -    7]ms ( 99.63%)      696 |

[   8 -    8]ms ( 99.65%)      695 |

[   9 -    9]ms ( 99.66%)      684 |

[  10 -   10]ms ( 99.68%)      673 |

[  11 -   11]ms ( 99.69%)      638 |

[  12 -   12]ms ( 99.70%)      651 |

[  13 -   13]ms ( 99.72%)      664 |

[  14 -   14]ms ( 99.73%)      670 |

[  15 -   15]ms ( 99.74%)      644 |

[  16 -   16]ms ( 99.76%)      709 |

[  17 -   17]ms ( 99.77%)      711 |

[  18 -   18]ms ( 99.79%)      678 |

[  19 -   19]ms ( 99.80%)      596 |

[  20 -   20]ms ( 99.81%)      580 |

[  21 -   21]ms ( 99.83%)      624 |

[  22 -   22]ms ( 99.84%)      597 |

[  23 -   23]ms ( 99.85%)      606 |

[  24 -   24]ms ( 99.86%)      624 |

[  25 -   25]ms ( 99.88%)      610 |

[  26 -   26]ms ( 99.89%)      594 |

[  27 -   27]ms ( 99.90%)      594 |

[  28 -   28]ms ( 99.92%)      596 |

[  29 -   29]ms ( 99.93%)      560 |

[  30 -   30]ms ( 99.94%)      634 |

[  31 -   31]ms ( 99.95%)      572 |

[  32 -   32]ms ( 99.96%)      526 |

[  33 -   33]ms ( 99.97%)      431 |

[  34 -   34]ms ( 99.98%)      284 |

[  35 -   35]ms ( 99.98%)      156 |

[  36 -   36]ms ( 99.98%)       84 |

[  37 -   37]ms ( 99.99%)       58 |

[  38 -   38]ms ( 99.99%)       45 |

[  39 -   39]ms ( 99.99%)       30 |

[  40 -   40]ms ( 99.99%)       27 |

[  41 -   41]ms ( 99.99%)       22 |

[  42 -   42]ms ( 99.99%)       23 |

[  43 -   43]ms ( 99.99%)       16 |

[  44 -   44]ms ( 99.99%)       16 |

[  45 -   45]ms ( 99.99%)        8 |

[  46 -   46]ms ( 99.99%)       10 |

[  47 -   47]ms ( 99.99%)        4 |

[  48 -   48]ms ( 99.99%)        4 |

[  49 -   49]ms ( 99.99%)        8 |

[  50 -  499]ms (100.00%)      497 |

Total: 4743528 operations

Latency on *.109 (swapped in):

The following data is collected for "GET"

[   0 -    9]us ( 29.91%)  1036660 | ############################################

[  10 -   19]us ( 57.45%)   954644 | ########################################

[  20 -   29]us ( 58.47%)    35256 | #

[  30 -   39]us ( 58.54%)     2519 |

[  40 -   49]us ( 58.55%)      221 |

[  50 -   59]us ( 58.55%)       60 |

[  60 -   69]us ( 58.55%)       23 |

[  70 -   79]us ( 58.55%)       15 |

[  80 -   89]us ( 58.55%)       28 |

[  90 -   99]us ( 58.55%)       70 |

[ 100 -  109]us ( 58.56%)      148 |

[ 110 -  119]us ( 58.56%)      292 |

[ 120 -  129]us ( 58.58%)      487 |

[ 130 -  139]us ( 58.60%)      724 |

[ 140 -  149]us ( 58.63%)     1118 |

[ 150 -  159]us ( 58.67%)     1472 |

[ 160 -  169]us ( 58.73%)     1958 |

[ 170 -  179]us ( 58.80%)     2349 |

[ 180 -  189]us ( 58.87%)     2631 |

[ 190 -  199]us ( 58.96%)     2863 |

[ 200 -  209]us ( 59.04%)     2960 |

[ 210 -  219]us ( 59.13%)     3078 |

[ 220 -  229]us ( 59.22%)     3116 |

[ 230 -  239]us ( 59.31%)     3054 |

[ 240 -  249]us ( 59.39%)     2845 |

[ 250 -  259]us ( 59.47%)     2665 |

[ 260 -  269]us ( 59.55%)     2775 |

[ 270 -  279]us ( 59.64%)     3069 |

[ 280 -  289]us ( 59.74%)     3711 |

[ 290 -  299]us ( 59.88%)     4786 |

[ 300 -  309]us ( 60.06%)     6216 |

[ 310 -  319]us ( 60.29%)     7869 |

[ 320 -  329]us ( 60.56%)     9592 |

[ 330 -  339]us ( 60.91%)    11869 |

[ 340 -  349]us ( 61.30%)    13529 |

[ 350 -  359]us ( 61.74%)    15403 |

[ 360 -  369]us ( 62.22%)    16759 |

[ 370 -  379]us ( 62.76%)    18653 |

[ 380 -  389]us ( 63.33%)    19615 |

[ 390 -  399]us ( 63.93%)    20708 |

[ 400 -  409]us ( 64.54%)    21352 |

[ 410 -  419]us ( 65.15%)    21226 |

[ 420 -  429]us ( 65.75%)    20783 |

[ 430 -  439]us ( 66.32%)    19671 |

[ 440 -  449]us ( 66.85%)    18361 |

[ 450 -  459]us ( 67.34%)    17019 |

[ 460 -  469]us ( 67.80%)    15692 |

[ 470 -  479]us ( 68.22%)    14834 |

[ 480 -  489]us ( 68.63%)    14067 |

[ 490 -  499]us ( 69.03%)    14015 |

[ 500 -  509]us ( 69.44%)    14013 |

[ 510 -  519]us ( 69.86%)    14525 |

[ 520 -  529]us ( 70.29%)    14934 |

[ 530 -  539]us ( 70.73%)    15283 |

[ 540 -  549]us ( 71.18%)    15605 |

[ 550 -  559]us ( 71.65%)    16396 |

[ 560 -  569]us ( 72.13%)    16712 |

[ 570 -  579]us ( 72.62%)    16966 |

[ 580 -  589]us ( 73.12%)    17303 |

[ 590 -  599]us ( 73.62%)    17283 |

[ 600 -  609]us ( 74.12%)    17353 |

[ 610 -  619]us ( 74.61%)    16984 |

[ 620 -  629]us ( 75.09%)    16515 |

[ 630 -  639]us ( 75.56%)    16445 |

[ 640 -  649]us ( 76.01%)    15625 |

[ 650 -  659]us ( 76.46%)    15387 |

[ 660 -  669]us ( 76.89%)    15120 |

[ 670 -  679]us ( 77.32%)    14637 |

[ 680 -  689]us ( 77.74%)    14558 |

[ 690 -  699]us ( 78.15%)    14445 |

[ 700 -  709]us ( 78.55%)    13883 |

[ 710 -  719]us ( 78.95%)    13916 |

[ 720 -  729]us ( 79.35%)    13541 |

[ 730 -  739]us ( 79.74%)    13656 |

[ 740 -  749]us ( 80.13%)    13405 |

[ 750 -  759]us ( 80.50%)    13045 |

[ 760 -  769]us ( 80.87%)    12705 |

[ 770 -  779]us ( 81.23%)    12548 |

[ 780 -  789]us ( 81.58%)    12087 |

[ 790 -  799]us ( 81.93%)    12011 |

[ 800 -  809]us ( 82.25%)    11280 |

[ 810 -  819]us ( 82.58%)    11356 |

[ 820 -  829]us ( 82.89%)    10803 |

[ 830 -  839]us ( 83.20%)    10675 |

[ 840 -  849]us ( 83.49%)    10256 |

[ 850 -  859]us ( 83.78%)     9869 |

[ 860 -  869]us ( 84.06%)     9577 |

[ 870 -  879]us ( 84.33%)     9374 |

[ 880 -  889]us ( 84.59%)     9090 |

[ 890 -  899]us ( 84.84%)     8869 |

[ 900 -  909]us ( 85.09%)     8663 |

[ 910 -  919]us ( 85.34%)     8618 |

[ 920 -  929]us ( 85.58%)     8312 |

[ 930 -  939]us ( 85.81%)     7896 |

[ 940 -  949]us ( 86.04%)     7800 |

[ 950 -  959]us ( 86.25%)     7589 |

[ 960 -  969]us ( 86.46%)     7166 |

[ 970 -  979]us ( 86.67%)     7188 |

[ 980 -  989]us ( 86.87%)     6860 |

[ 990 -  999]us ( 87.06%)     6700 |

[   1 -    1]ms ( 94.14%)   245341 | ##########

[   2 -    2]ms ( 96.04%)    65889 | ##

[   3 -    3]ms ( 96.87%)    28935 | #

[   4 -    4]ms ( 97.26%)    13519 |

[   5 -    5]ms ( 97.44%)     6296 |

[   6 -    6]ms ( 97.55%)     3518 |

[   7 -    7]ms ( 97.61%)     2228 |

[   8 -    8]ms ( 97.66%)     1816 |

[   9 -    9]ms ( 97.70%)     1405 |

[  10 -   10]ms ( 97.74%)     1190 |

[  11 -   11]ms ( 97.77%)     1077 |

[  12 -   12]ms ( 97.80%)      971 |

[  13 -   13]ms ( 97.82%)      884 |

[  14 -   14]ms ( 97.84%)      734 |

[  15 -   15]ms ( 97.87%)      748 |

[  16 -   16]ms ( 97.89%)      744 |

[  17 -   17]ms ( 97.91%)      719 |

[  18 -   18]ms ( 97.93%)      734 |

[  19 -   19]ms ( 97.95%)      699 |

[  20 -   20]ms ( 97.97%)      668 |

[  21 -   21]ms ( 97.99%)      615 |

[  22 -   22]ms ( 98.00%)      637 |

[  23 -   23]ms ( 98.02%)      599 |

[  24 -   24]ms ( 98.04%)      590 |

[  25 -   25]ms ( 98.05%)      561 |

[  26 -   26]ms ( 98.07%)      586 |

[  27 -   27]ms ( 98.09%)      501 |

[  28 -   28]ms ( 98.10%)      478 |

[  29 -   29]ms ( 98.11%)      517 |

[  30 -   30]ms ( 98.13%)      496 |

[  31 -   31]ms ( 98.14%)      521 |

[  32 -   32]ms ( 98.16%)      467 |

[  33 -   33]ms ( 98.17%)      492 |

[  34 -   34]ms ( 98.19%)      500 |

[  35 -   35]ms ( 98.20%)      491 |

[  36 -   36]ms ( 98.21%)      438 |

[  37 -   37]ms ( 98.23%)      493 |

[  38 -   38]ms ( 98.24%)      412 |

[  39 -   39]ms ( 98.25%)      429 |

[  40 -   40]ms ( 98.26%)      416 |

[  41 -   41]ms ( 98.28%)      416 |

[  42 -   42]ms ( 98.29%)      459 |

[  43 -   43]ms ( 98.30%)      435 |

[  44 -   44]ms ( 98.31%)      453 |

[  45 -   45]ms ( 98.33%)      438 |

[  46 -   46]ms ( 98.34%)      400 |

[  47 -   47]ms ( 98.35%)      378 |

[  48 -   48]ms ( 98.36%)      393 |

[  49 -   49]ms ( 98.37%)      400 |

[  50 -  499]ms ( 99.67%)    44996 | #

[ 500 -  999]ms ( 99.79%)     4180 |

[1000 - 1499]ms ( 99.81%)      800 |

[1500 - 1999]ms ( 99.82%)      289 |

[2000 - 2499]ms ( 99.83%)      126 |

[2500 - 2999]ms ( 99.83%)      135 |

[3000 - 3499]ms ( 99.83%)      118 |

[3500 - 3999]ms ( 99.84%)       95 |

[4000 - 4499]ms ( 99.84%)      106 |

[4500 - 4999]ms ( 99.84%)      110 |

[   5 -    9]s  ( 99.86%)      779 |

[  10 -   19]s  ( 99.89%)      897 |

[  20 -   39]s  ( 99.93%)     1248 |

[  40 -   79]s  ( 99.96%)     1167 |

[  80 - inf.]s  (100.00%)     1373 |

Total: 3466202 operations

This issue doesn't make a lot of sense actually because a similar test (~~MB-26738~~) passes with a reasonably low latency. I would blame the environment if performance didn't recover at the end of rebalance.

Some graphs:
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_510-1368_rebalance_c509

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

bg_wait.png
444 kB
11/Nov/17 8:04 AM
cur_items.png
431 kB
11/Nov/17 8:08 AM
ops.png
419 kB
11/Nov/17 8:06 AM
Screen Shot 2017-11-14 at 11.13.42.png
41 kB
14/Nov/17 3:14 AM

Issue Links

relates to

MB-29217 9 node rebalance is 5-8x slower in vulcan

Closed

MB-29046 Throughput of YCSB Workload D dropped by 10% due to connection buffer recycling

Closed

MB-28708 Ops/sec suffers during rebalance of data service

Closed

MB-29682 Rebalance exited with reason {child_died,bad_replicas}

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Latency of GET operations exceeds minutes during swap rebalance (9 nodes, 1TB)

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty