Details
-
Bug
-
Resolution: Duplicate
-
Major
-
6.5.0
-
6.5.0-4169
-
Untriaged
-
-
Unknown
Description
It is observed that the doc ops rate is going down quite drastically on rebalance out 1 node from a 4 node cluster given number of clients connections and parallel data loading threads remains same.
1 default bucket with replicas=2, data loading is done with durability=MAJORITY throughout.
1. 4 Node Cluster:
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.220 | [u'kv'] | Cluster node |
|
| 172.23.105.221 | None | <--- IN --- |
|
| 172.23.105.223 | None | <--- IN --- |
|
| 172.23.105.225 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
Create 250k docs, Ops Rate:
Ops rate for 'default': 0.000000
|
Ops rate for 'default': 0.000000
|
Ops rate for 'default': 1402.000000
|
Ops rate for 'default': 4349.650350
|
Ops rate for 'default': 4426.426426
|
Ops rate for 'default': 6935.064935
|
Ops rate for 'default': 5419.000000
|
Ops rate for 'default': 6333.333333
|
Ops rate for 'default': 6602.000000
|
Ops rate for 'default': 5754.000000
|
Ops rate for 'default': 6175.000000
|
Ops rate for 'default': 5428.000000
|
Ops rate for 'default': 6986.000000
|
Ops rate for 'default': 6448.000000
|
Ops rate for 'default': 4748.251748
|
Ops rate for 'default': 6550.550551
|
Ops rate for 'default': 5544.000000
|
Ops rate for 'default': 6727.544910
|
Ops rate for 'default': 5139.278557
|
Ops rate for 'default': 5948.051948
|
Ops rate for 'default': 5541.541542
|
Ops rate for 'default': 6012.000000
|
Ops rate for 'default': 6002.000000
|
Ops rate for 'default': 5052.000000
|
Ops rate for 'default': 6637.000000
|
Ops rate for 'default': 5725.000000
|
Ops rate for 'default': 6190.000000
|
Ops rate for 'default': 4936.936937
|
Ops rate for 'default': 6724.000000
|
Ops rate for 'default': 5446.000000
|
Ops rate for 'default': 6323.353293
|
Ops rate for 'default': 6513.026052
|
Ops rate for 'default': 5308.000000
|
Ops rate for 'default': 6863.000000
|
Ops rate for 'default': 5767.232767
|
Ops rate for 'default': 6436.436436
|
Ops rate for 'default': 5122.244489
|
Ops rate for 'default': 6393.000000
|
2. 3 node cluster:
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.225 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.220 | [u'kv'] | Cluster node |
|
| 172.23.105.223 | [u'kv'] | Cluster node |
|
| 172.23.105.221 | [u'kv'] | Cluster node |
|
+----------------+----------+--------------+
|
Rebalance completed and then data loading is started.
Update 250k docs, Ops Rate:
Ops rate for 'default': 679.000000
|
Ops rate for 'default': 321.000000
|
Ops rate for 'default': 1255.000000
|
Ops rate for 'default': 1714.000000
|
Ops rate for 'default': 1532.000000
|
Ops rate for 'default': 2092.907093
|
Ops rate for 'default': 2043.043043
|
Ops rate for 'default': 2078.000000
|
Ops rate for 'default': 2060.939061
|
Ops rate for 'default': 2002.000000
|
Ops rate for 'default': 2209.000000
|
Ops rate for 'default': 2199.000000
|
Ops rate for 'default': 2094.000000
|
Ops rate for 'default': 708.000000
|
Ops rate for 'default': 566.000000
|
Ops rate for 'default': 1830.000000
|
Ops rate for 'default': 2017.000000
|
Ops rate for 'default': 1658.000000
|
Ops rate for 'default': 1988.000000
|
Ops rate for 'default': 1971.000000
|
Ops rate for 'default': 2102.897103
|
Ops rate for 'default': 2210.210210
|
Ops rate for 'default': 1562.000000
|
Ops rate for 'default': 1947.000000
|
Ops rate for 'default': 1914.000000
|
Ops rate for 'default': 1829.000000
|
Ops rate for 'default': 1452.000000
|
Ops rate for 'default': 1235.000000
|
Ops rate for 'default': 1871.128871
|
Ops rate for 'default': 1807.807808
|
Ops rate for 'default': 1839.000000
|
Ops rate for 'default': 1345.000000
|
Ops rate for 'default': 1831.000000
|
Ops rate for 'default': 1992.000000
|
Ops rate for 'default': 1992.000000
|
Ops rate for 'default': 2018.981019
|
Ops rate for 'default': 1635.635636
|
Ops rate for 'default': 2030.000000
|
Ops rate for 'default': 1847.152847
|
Ops rate for 'default': 2089.089089
|
Ops rate for 'default': 1541.916168
|
Ops rate for 'default': 2118.236473
|
Ops rate for 'default': 2058.000000
|
Ops rate for 'default': 2087.000000
|
Ops rate for 'default': 2034.000000
|
Ops rate for 'default': 1673.000000
|
Ops rate for 'default': 2017.000000
|
Ops rate for 'default': 2029.850746
|
Ops rate for 'default': 2171.859296
|
Ops rate for 'default': 2184.000000
|
Ops rate for 'default': 1505.000000
|
Ops rate for 'default': 1733.000000
|
Ops rate for 'default': 1972.000000
|
Ops rate for 'default': 1995.000000
|
Ops rate for 'default': 1421.000000
|
Ops rate for 'default': 1792.000000
|
Ops rate for 'default': 1808.383234
|
Ops rate for 'default': 1761.523046
|
Ops rate for 'default': 1894.000000
|
Ops rate for 'default': 1427.000000
|
Ops rate for 'default': 1814.000000
|
Ops rate for 'default': 1927.000000
|
Ops rate for 'default': 1941.000000
|
Ops rate for 'default': 2048.000000
|
Ops rate for 'default': 2017.000000
|
Ops rate for 'default': 1731.000000
|
Ops rate for 'default': 2012.000000
|
Ops rate for 'default': 2241.000000
|
Ops rate for 'default': 1561.000000
|
Ops rate for 'default': 2045.954046
|
Ops rate for 'default': 2088.088088
|
Ops rate for 'default': 2136.000000
|
Ops rate for 'default': 2212.000000
|
Ops rate for 'default': 1681.954138
|
Ops rate for 'default': 1988.966901
|
Ops rate for 'default': 2218.000000
|
Ops rate for 'default': 2152.000000
|
Ops rate for 'default': 1563.000000
|
Ops rate for 'default': 1443.000000
|
Ops rate for 'default': 1877.000000
|
Ops rate for 'default': 1878.000000
|
Ops rate for 'default': 1873.000000
|
Ops rate for 'default': 1407.000000
|
Ops rate for 'default': 1861.000000
|
Ops rate for 'default': 1845.154845
|
Ops rate for 'default': 1829.829830
|
Ops rate for 'default': 1919.080919
|
Ops rate for 'default': 1451.548452
|
Ops rate for 'default': 1975.000000
|
Ops rate for 'default': 2019.000000
|
Ops rate for 'default': 2140.859141
|
Ops rate for 'default': 1646.646647
|
Ops rate for 'default': 1593.000000
|
Ops rate for 'default': 2134.000000
|
Ops rate for 'default': 2115.000000
|
Ops rate for 'default': 2110.000000
|
Ops rate for 'default': 1651.348651
|
Ops rate for 'default': 2052.052052
|
Ops rate for 'default': 1986.000000
|
Ops rate for 'default': 2094.715852
|
Ops rate for 'default': 1575.424575
|
Ops rate for 'default': 2136.726547
|
Ops rate for 'default': 2153.460381
|
Ops rate for 'default': 2030.969031
|
Ops rate for 'default': 1933.933934
|
Ops rate for 'default': 1555.000000
|
Ops rate for 'default': 1948.000000
|
Ops rate for 'default': 2015.984016
|
Ops rate for 'default': 1836.836837
|
Ops rate for 'default': 1902.000000
|
Ops rate for 'default': 1395.000000
|
Ops rate for 'default': 1675.000000
|
Ops rate for 'default': 1782.000000
|
Ops rate for 'default': 1984.000000
|
Ops rate for 'default': 1427.000000
|
Ops rate for 'default': 1782.000000
|
Ops rate for 'default': 2004.000000
|
Ops rate for 'default': 1984.000000
|
Ops rate for 'default': 1640.000000
|
Ops rate for 'default': 1339.000000
|
Ops rate for 'default': 1843.000000
|
Ops rate for 'default': 1991.000000
|
3. 2 node cluster:
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.220 | [u'kv'] | Cluster node |
|
| 172.23.105.223 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.221 | [u'kv'] | Cluster node |
|
+----------------+----------+--------------+
|
Rebalance is completed.
Update same 250k doc, ops rate:
Ops rate for 'default': 0.000000
|
Ops rate for 'default': 938.000000
|
Ops rate for 'default': 319.000000
|
Ops rate for 'default': 743.000000
|
Ops rate for 'default': 1105.000000
|
Ops rate for 'default': 1142.000000
|
Ops rate for 'default': 1066.000000
|
Ops rate for 'default': 793.000000
|
Ops rate for 'default': 1134.865135
|
Ops rate for 'default': 1059.000000
|
Ops rate for 'default': 1108.108108
|
Ops rate for 'default': 1132.000000
|
Ops rate for 'default': 901.000000
|
Ops rate for 'default': 1083.000000
|
Ops rate for 'default': 1057.000000
|
Ops rate for 'default': 1091.000000
|
Ops rate for 'default': 1104.000000
|
Ops rate for 'default': 941.058941
|
Ops rate for 'default': 1129.129129
|
Ops rate for 'default': 1101.000000
|
Ops rate for 'default': 1119.880120
|
Ops rate for 'default': 698.698699
|
Ops rate for 'default': 977.000000
|
Ops rate for 'default': 928.000000
|
Ops rate for 'default': 842.157842
|
Ops rate for 'default': 760.760761
|
Ops rate for 'default': 718.562874
|
Ops rate for 'default': 860.721443
|
Ops rate for 'default': 883.000000
|
Ops rate for 'default': 1005.000000
|
Ops rate for 'default': 954.000000
|
Ops rate for 'default': 823.000000
|
Ops rate for 'default': 1018.000000
|
Ops rate for 'default': 1116.000000
|
Ops rate for 'default': 1089.000000
|
Ops rate for 'default': 1096.903097
|
Ops rate for 'default': 1147.000000
|
Ops rate for 'default': 1128.000000
|
Ops rate for 'default': 1111.000000
|
Ops rate for 'default': 1117.000000
|
Ops rate for 'default': 947.000000
|
Ops rate for 'default': 1104.000000
|
Ops rate for 'default': 1041.000000
|
Ops rate for 'default': 1089.000000
|
Ops rate for 'default': 1068.000000
|
Ops rate for 'default': 867.000000
|
Ops rate for 'default': 1126.000000
|
Ops rate for 'default': 1118.000000
|
Ops rate for 'default': 1095.904096
|
Ops rate for 'default': 1141.141141
|
Ops rate for 'default': 794.000000
|
Ops rate for 'default': 921.000000
|
Ops rate for 'default': 927.000000
|
Ops rate for 'default': 860.000000
|
Ops rate for 'default': 816.000000
|
Ops rate for 'default': 898.203593
|
Ops rate for 'default': 853.707415
|
Ops rate for 'default': 960.000000
|
Ops rate for 'default': 892.000000
|
Ops rate for 'default': 807.000000
|
Ops rate for 'default': 1103.000000
|
Ops rate for 'default': 1183.000000
|
Ops rate for 'default': 1061.000000
|
Ops rate for 'default': 1098.000000
|
Ops rate for 'default': 999.000000
|
Ops rate for 'default': 1093.000000
|
Ops rate for 'default': 1070.929071
|
Ops rate for 'default': 1106.106106
|
Ops rate for 'default': 782.000000
|
Ops rate for 'default': 1071.000000
|
Ops rate for 'default': 1050.000000
|
Ops rate for 'default': 1056.000000
|
Ops rate for 'default': 1101.000000
|
Ops rate for 'default': 756.000000
|
Ops rate for 'default': 1099.800399
|
Ops rate for 'default': 1034.068136
|
Ops rate for 'default': 1031.000000
|
Ops rate for 'default': 1059.000000
|
Ops rate for 'default': 642.000000
|
Ops rate for 'default': 893.106893
|
Ops rate for 'default': 863.863864
|
Ops rate for 'default': 838.000000
|
Ops rate for 'default': 756.000000
|
Ops rate for 'default': 851.000000
|
Ops rate for 'default': 901.000000
|
Ops rate for 'default': 939.000000
|
Ops rate for 'default': 977.022977
|
Ops rate for 'default': 794.794795
|
Ops rate for 'default': 973.000000
|
Ops rate for 'default': 1024.000000
|
Ops rate for 'default': 1086.000000
|
Ops rate for 'default': 1127.000000
|
Ops rate for 'default': 817.000000
|
Ops rate for 'default': 1125.000000
|
Ops rate for 'default': 1132.000000
|
Ops rate for 'default': 1146.000000
|
Ops rate for 'default': 764.000000
|
Ops rate for 'default': 1061.000000
|
Ops rate for 'default': 1106.000000
|
Ops rate for 'default': 1068.000000
|
Ops rate for 'default': 1063.000000
|
Ops rate for 'default': 822.177822
|
Ops rate for 'default': 1068.068068
|
Ops rate for 'default': 1104.000000
|
Ops rate for 'default': 1067.000000
|
Ops rate for 'default': 1061.000000
|
Ops rate for 'default': 773.000000
|
Ops rate for 'default': 926.000000
|
Ops rate for 'default': 865.000000
|
Ops rate for 'default': 858.000000
|
Ops rate for 'default': 653.692615
|
Ops rate for 'default': 719.438878
|
Ops rate for 'default': 882.000000
|
Ops rate for 'default': 1031.000000
|
Ops rate for 'default': 1047.000000
|
Ops rate for 'default': 789.000000
|
Ops rate for 'default': 1104.000000
|
Ops rate for 'default': 1099.000000
|
Ops rate for 'default': 1082.000000
|
Ops rate for 'default': 1113.000000
|
Ops rate for 'default': 815.000000
|
Ops rate for 'default': 1071.000000
|
Ops rate for 'default': 1128.000000
|
Ops rate for 'default': 1075.000000
|
Ops rate for 'default': 1095.000000
|
Ops rate for 'default': 1135.000000
|
Ops rate for 'default': 1107.000000
|
Ops rate for 'default': 1062.000000
|
Ops rate for 'default': 1099.000000
|
Ops rate for 'default': 801.000000
|
Ops rate for 'default': 1099.000000
|
Ops rate for 'default': 1082.000000
|
Ops rate for 'default': 931.000000
|
Ops rate for 'default': 1119.000000
|
Ops rate for 'default': 764.000000
|
Ops rate for 'default': 993.000000
|
Ops rate for 'default': 961.038961
|
Ops rate for 'default': 855.855856
|
Ops rate for 'default': 873.000000
|
Ops rate for 'default': 681.000000
|
Ops rate for 'default': 885.000000
|
Ops rate for 'default': 971.000000
|
Ops rate for 'default': 1029.000000
|
Ops rate for 'default': 861.000000
|
Ops rate for 'default': 1086.000000
|
Ops rate for 'default': 1094.000000
|
Ops rate for 'default': 1051.000000
|
Ops rate for 'default': 1099.000000
|
Ops rate for 'default': 914.000000
|
Ops rate for 'default': 1155.000000
|
Ops rate for 'default': 1059.000000
|
Ops rate for 'default': 1068.000000
|
Ops rate for 'default': 1103.000000
|
Ops rate for 'default': 982.000000
|
Ops rate for 'default': 952.000000
|
Ops rate for 'default': 1077.000000
|
Ops rate for 'default': 1103.000000
|
Ops rate for 'default': 808.000000
|
Ops rate for 'default': 1049.000000
|
Ops rate for 'default': 1113.000000
|
Ops rate for 'default': 1124.000000
|
Ops rate for 'default': 1062.000000
|
Ops rate for 'default': 739.000000
|
Ops rate for 'default': 935.129741
|
Ops rate for 'default': 898.797595
|
Ops rate for 'default': 857.000000
|
Ops rate for 'default': 626.000000
|
Ops rate for 'default': 859.000000
|
Ops rate for 'default': 926.000000
|
Ops rate for 'default': 978.000000
|
Ops rate for 'default': 1102.000000
|
Ops rate for 'default': 959.000000
|
Ops rate for 'default': 1067.000000
|
Ops rate for 'default': 1054.000000
|
Ops rate for 'default': 892.000000
|
Ops rate for 'default': 1108.000000
|
Ops rate for 'default': 961.000000
|
Ops rate for 'default': 1041.958042
|
Ops rate for 'default': 1121.121121
|
Ops rate for 'default': 1098.000000
|
Ops rate for 'default': 953.046953
|
Ops rate for 'default': 1124.124124
|
Ops rate for 'default': 1047.000000
|
Ops rate for 'default': 1104.000000
|
Ops rate for 'default': 1075.000000
|
Ops rate for 'default': 886.000000
|
Ops rate for 'default': 1107.000000
|
Ops rate for 'default': 1069.000000
|
Ops rate for 'default': 1096.903097
|
Ops rate for 'default': 1052.052052
|
Ops rate for 'default': 861.000000
|
Ops rate for 'default': 974.000000
|
Ops rate for 'default': 932.000000
|
Ops rate for 'default': 882.000000
|
Ops rate for 'default': 657.000000
|
Ops rate for 'default': 867.132867
|
Ops rate for 'default': 908.091908
|
Ops rate for 'default': 885.771543
|
Ops rate for 'default': 983.000000
|
Ops rate for 'default': 827.000000
|
Ops rate for 'default': 1092.000000
|
Ops rate for 'default': 1106.000000
|
Ops rate for 'default': 1032.000000
|
Ops rate for 'default': 1138.000000
|
Ops rate for 'default': 851.000000
|
Ops rate for 'default': 1021.000000
|
Ops rate for 'default': 1053.000000
|
Ops rate for 'default': 1078.000000
|
Ops rate for 'default': 788.000000
|
Ops rate for 'default': 1099.000000
|
Ops rate for 'default': 938.000000
|
Ops rate for 'default': 1091.000000
|
Ops rate for 'default': 1142.000000
|
Ops rate for 'default': 742.000000
|
Ops rate for 'default': 1108.000000
|
Ops rate for 'default': 1138.861139
|
Ops rate for 'default': 1101.101101
|
Ops rate for 'default': 1048.000000
|
Ops rate for 'default': 909.000000
|
Ops rate for 'default': 968.000000
|
Ops rate for 'default': 892.000000
|
Ops rate for 'default': 885.000000
|
Ops rate for 'default': 889.110889
|
Ops rate for 'default': 829.829830
|
Ops rate for 'default': 896.207585
|
Ops rate for 'default': 960.921844
|
Ops rate for 'default': 1027.000000
|
Ops rate for 'default': 818.000000
|
Ops rate for 'default': 874.000000
|
Ops rate for 'default': 1079.000000
|
Ops rate for 'default': 1079.000000
|
Ops rate for 'default': 1095.000000
|
Ops rate for 'default': 861.000000
|
Ops rate for 'default': 1081.000000
|
Ops rate for 'default': 1034.000000
|
Ops rate for 'default': 1062.937063
|
Ops rate for 'default': 1082.000000
|
Ops rate for 'default': 954.954955
|
I am not sure on the expected result about how much it should affect the data loading doc ops rate, but is this expected?
Attachments
Issue Links
- duplicates
-
-
- Closed
-