Details
Description
Using UPR, centos-64 bit, Version:: 3.0.0-584
1. Add 5 nodes to cluster
2. Create default bucket with replica=1
3. Add 50000 items
4. Rebalance out 1 node
Cluster Nodes
10.6.2.144
10.6.2.147
10.6.2.148
10.6.2.149
10.6.2.150
Rebalanced-out Node:: 10.6.2.150
Expected UUIDs of buckets is the same after we rebalance out
Observed Difference in UUIDs which are a part of 10.6.2.150
vb_28 :
{'uuid': 'Expected 170724551552615 :: Actual 10627939076719'}
vb_29 :
{'uuid': 'Expected 45359036031519 :: Actual 59640339796757'}
vb_26 :
{'uuid': 'Expected 133471459893767 :: Actual 114252468159585''}
vb_27 :
{'uuid': 'Expected 104126006114437 :: Actual 86374568011280'}
vb_6 :
{'uuid': 'Expected 175130199501286 :: Actual 23856384718339'}
vb_13 :
{'uuid': 'Expected 84680084219325 :: Actual 233007425883197'}
vb_31 :
{'uuid': 'Expected 170201395300887 :: Actual 174005664911105'}
vb_30 :
{'uuid': 'Expected 83362933025898 :: Actual 229555825112404'}
Dump of Bucket, Node, Vbucket Information (uuid, high_seqno, purge_seqno)
Before we rebalanced-out
______________________
----- Bucket default -----
------------Node 10.6.2.147-----------
for vbucket vb_9
:: for key high_seqno = 1558
:: for key uuid = 224689180220009
:: for key purge_seqno = 0
for vbucket vb_8
:: for key purge_seqno = 0
:: for key uuid = 24173416366515
:: for key high_seqno = 1569
for vbucket vb_1
:: for key purge_seqno = 0
:: for key uuid = 3382665152986
:: for key high_seqno = 1569
for vbucket vb_0
:: for key purge_seqno = 0
:: for key uuid = 252550884809196
:: for key high_seqno = 1551
for vbucket vb_7
:: for key high_seqno = 1551
:: for key uuid = 160369294390021
:: for key purge_seqno = 0
for vbucket vb_11
:: for key purge_seqno = 0
:: for key uuid = 91922888028629
:: for key high_seqno = 1560
for vbucket vb_10
:: for key high_seqno = 1568
:: for key uuid = 98217872738904
:: for key purge_seqno = 0
for vbucket vb_13
:: for key purge_seqno = 0
:: for key uuid = 84680084219325
:: for key high_seqno = 1568
for vbucket vb_12
:: for key high_seqno = 1560
:: for key uuid = 75150401842511
:: for key purge_seqno = 0
for vbucket vb_15
:: for key purge_seqno = 0
:: for key uuid = 21343486345542
:: for key high_seqno = 1569
for vbucket vb_16
:: for key high_seqno = 1560
:: for key uuid = 113002939605931
:: for key purge_seqno = 0
for vbucket vb_21
:: for key purge_seqno = 0
:: for key uuid = 131979757081143
:: for key high_seqno = 1559
for vbucket vb_28
:: for key high_seqno = 1553
:: for key uuid = 170724551552615
:: for key purge_seqno = 0
------------Node 10.6.2.144-----------
for vbucket vb_8
:: for key purge_seqno = 0
:: for key uuid = 24173416366515
:: for key high_seqno = 1569
for vbucket vb_1
:: for key purge_seqno = 0
:: for key uuid = 3382665152986
:: for key high_seqno = 1569
for vbucket vb_0
:: for key purge_seqno = 0
:: for key uuid = 252550884809196
:: for key high_seqno = 1551
for vbucket vb_3
:: for key high_seqno = 1570
:: for key uuid = 189410128542953
:: for key purge_seqno = 0
for vbucket vb_2
:: for key high_seqno = 1552
:: for key uuid = 202175152188264
:: for key purge_seqno = 0
for vbucket vb_5
:: for key high_seqno = 1552
:: for key uuid = 272505847148341
:: for key purge_seqno = 0
for vbucket vb_4
:: for key purge_seqno = 0
:: for key uuid = 170782289546886
:: for key high_seqno = 1570
for vbucket vb_7
:: for key high_seqno = 1551
:: for key uuid = 160369294390021
:: for key purge_seqno = 0
for vbucket vb_6
:: for key purge_seqno = 0
:: for key uuid = 175130199501286
:: for key high_seqno = 1569
for vbucket vb_14
:: for key purge_seqno = 0
:: for key uuid = 9666779467087
:: for key high_seqno = 1558
for vbucket vb_20
:: for key high_seqno = 1569
:: for key uuid = 226337594275581
:: for key purge_seqno = 0
for vbucket vb_26
:: for key purge_seqno = 0
:: for key uuid = 133471459893767
:: for key high_seqno = 1569
for vbucket vb_27
:: for key purge_seqno = 0
:: for key uuid = 104126006114437
:: for key high_seqno = 1553
------------Node 10.6.2.150-----------
for vbucket vb_5
:: for key high_seqno = 1552
:: for key uuid = 272505847148341
:: for key purge_seqno = 0
for vbucket vb_6
:: for key purge_seqno = 0
:: for key uuid = 175130199501286
:: for key high_seqno = 1569
for vbucket vb_28
:: for key high_seqno = 1553
:: for key uuid = 170724551552615
:: for key purge_seqno = 0
for vbucket vb_29
:: for key purge_seqno = 0
:: for key uuid = 45359036031519
:: for key high_seqno = 1569
for vbucket vb_13
:: for key purge_seqno = 0
:: for key uuid = 84680084219325
:: for key high_seqno = 1568
for vbucket vb_26
:: for key purge_seqno = 0
:: for key uuid = 133471459893767
:: for key high_seqno = 1569
for vbucket vb_19
:: for key purge_seqno = 0
:: for key uuid = 251498636389290
:: for key high_seqno = 1569
for vbucket vb_27
:: for key purge_seqno = 0
:: for key uuid = 104126006114437
:: for key high_seqno = 1553
for vbucket vb_24
:: for key high_seqno = 1571
:: for key uuid = 184565815683126
:: for key purge_seqno = 0
for vbucket vb_25
:: for key high_seqno = 1552
:: for key uuid = 266765133502323
:: for key purge_seqno = 0
for vbucket vb_31
:: for key purge_seqno = 0
:: for key uuid = 170201395300887
:: for key high_seqno = 1571
for vbucket vb_30
:: for key high_seqno = 1552
:: for key uuid = 83362933025898
:: for key purge_seqno = 0
------------Node 10.6.2.148-----------
for vbucket vb_9
:: for key high_seqno = 1558
:: for key uuid = 224689180220009
:: for key purge_seqno = 0
for vbucket vb_10
:: for key high_seqno = 1568
:: for key uuid = 98217872738904
:: for key purge_seqno = 0
for vbucket vb_3
:: for key high_seqno = 1570
:: for key uuid = 189410128542953
:: for key purge_seqno = 0
for vbucket vb_2
:: for key high_seqno = 1552
:: for key uuid = 202175152188264
:: for key purge_seqno = 0
for vbucket vb_29
:: for key purge_seqno = 0
:: for key uuid = 45359036031519
:: for key high_seqno = 1569
for vbucket vb_15
:: for key purge_seqno = 0
:: for key uuid = 21343486345542
:: for key high_seqno = 1569
for vbucket vb_14
:: for key purge_seqno = 0
:: for key uuid = 9666779467087
:: for key high_seqno = 1558
for vbucket vb_17
:: for key purge_seqno = 0
:: for key uuid = 50245200692736
:: for key high_seqno = 1570
for vbucket vb_16
:: for key high_seqno = 1560
:: for key uuid = 113002939605931
:: for key purge_seqno = 0
for vbucket vb_19
:: for key purge_seqno = 0
:: for key uuid = 251498636389290
:: for key high_seqno = 1569
for vbucket vb_18
:: for key high_seqno = 1559
:: for key uuid = 5501229932749
:: for key purge_seqno = 0
for vbucket vb_22
:: for key high_seqno = 1570
:: for key uuid = 247498759341926
:: for key purge_seqno = 0
for vbucket vb_23
:: for key purge_seqno = 0
:: for key uuid = 74620381341917
:: for key high_seqno = 1560
------------Node 10.6.2.149-----------
for vbucket vb_4
:: for key purge_seqno = 0
:: for key uuid = 170782289546886
:: for key high_seqno = 1570
for vbucket vb_11
:: for key purge_seqno = 0
:: for key uuid = 91922888028629
:: for key high_seqno = 1560
for vbucket vb_12
:: for key purge_seqno = 0
:: for key uuid = 75150401842511
:: for key high_seqno = 1560
for vbucket vb_17
:: for key purge_seqno = 0
:: for key uuid = 50245200692736
:: for key high_seqno = 1570
for vbucket vb_18
:: for key high_seqno = 1559
:: for key uuid = 5501229932749
:: for key purge_seqno = 0
for vbucket vb_20
:: for key high_seqno = 1569
:: for key uuid = 226337594275581
:: for key purge_seqno = 0
for vbucket vb_21
:: for key purge_seqno = 0
:: for key uuid = 131979757081143
:: for key high_seqno = 1559
for vbucket vb_22
:: for key high_seqno = 1570
:: for key uuid = 247498759341926
:: for key purge_seqno = 0
for vbucket vb_23
:: for key purge_seqno = 0
:: for key uuid = 74620381341917
:: for key high_seqno = 1560
for vbucket vb_24
:: for key high_seqno = 1571
:: for key uuid = 184565815683126
:: for key purge_seqno = 0
for vbucket vb_25
:: for key high_seqno = 1552
:: for key uuid = 266765133502323
:: for key purge_seqno = 0
for vbucket vb_31
:: for key purge_seqno = 0
:: for key uuid = 170201395300887
:: for key high_seqno = 1571
for vbucket vb_30
:: for key high_seqno = 1552
:: for key uuid = 83362933025898
:: for key purge_seqno = 0
After we rebalanced-out
____________________
----- Bucket default -----
------------Node 10.6.2.147-----------
for vbucket vb_9
:: for key high_seqno = 2337
:: for key uuid = 224689180220009
:: for key purge_seqno = 0
for vbucket vb_8
:: for key purge_seqno = 0
:: for key uuid = 24173416366515
:: for key high_seqno = 2354
for vbucket vb_1
:: for key purge_seqno = 0
:: for key uuid = 3382665152986
:: for key high_seqno = 2355
for vbucket vb_0
:: for key purge_seqno = 0
:: for key uuid = 252550884809196
:: for key high_seqno = 2326
for vbucket vb_7
:: for key high_seqno = 2326
:: for key uuid = 160369294390021
:: for key purge_seqno = 0
for vbucket vb_6
:: for key purge_seqno = 0
:: for key uuid = 23856384718339
:: for key high_seqno = 2355
for vbucket vb_11
:: for key purge_seqno = 0
:: for key uuid = 91922888028629
:: for key high_seqno = 2339
for vbucket vb_10
:: for key high_seqno = 2353
:: for key uuid = 98217872738904
:: for key purge_seqno = 0
for vbucket vb_12
:: for key high_seqno = 2339
:: for key uuid = 75150401842511
:: for key purge_seqno = 0
for vbucket vb_15
:: for key purge_seqno = 0
:: for key uuid = 21343486345542
:: for key high_seqno = 2354
for vbucket vb_26
:: for key purge_seqno = 0
:: for key uuid = 114252468159585
:: for key high_seqno = 2355
for vbucket vb_16
:: for key high_seqno = 2339
:: for key uuid = 113002939605931
:: for key purge_seqno = 0
for vbucket vb_21
:: for key purge_seqno = 0
:: for key uuid = 131979757081143
:: for key high_seqno = 2338
for vbucket vb_24
:: for key high_seqno = 2357
:: for key uuid = 184565815683126
:: for key purge_seqno = 0
for vbucket vb_31
:: for key purge_seqno = 0
:: for key uuid = 174005664911105
:: for key high_seqno = 2357
for vbucket vb_28
:: for key high_seqno = 2328
:: for key uuid = 10627939076719
:: for key purge_seqno = 0
------------Node 10.6.2.144-----------
for vbucket vb_8
:: for key purge_seqno = 0
:: for key uuid = 24173416366515
:: for key high_seqno = 2354
for vbucket vb_1
:: for key purge_seqno = 0
:: for key uuid = 3382665152986
:: for key high_seqno = 2355
for vbucket vb_0
:: for key purge_seqno = 0
:: for key uuid = 252550884809196
:: for key high_seqno = 2326
for vbucket vb_3
:: for key high_seqno = 2356
:: for key uuid = 189410128542953
:: for key purge_seqno = 0
for vbucket vb_2
:: for key high_seqno = 2327
:: for key uuid = 202175152188264
:: for key purge_seqno = 0
for vbucket vb_5
:: for key high_seqno = 2327
:: for key uuid = 272505847148341
:: for key purge_seqno = 0
for vbucket vb_4
:: for key purge_seqno = 0
:: for key uuid = 170782289546886
:: for key high_seqno = 2356
for vbucket vb_7
:: for key high_seqno = 2326
:: for key uuid = 160369294390021
:: for key purge_seqno = 0
for vbucket vb_28
:: for key high_seqno = 2328
:: for key uuid = 10627939076719
:: for key purge_seqno = 0
for vbucket vb_29
:: for key purge_seqno = 0
:: for key uuid = 59640339796757
:: for key high_seqno = 2355
for vbucket vb_13
:: for key purge_seqno = 0
:: for key uuid = 233007425883197
:: for key high_seqno = 2353
for vbucket vb_14
:: for key purge_seqno = 0
:: for key uuid = 9666779467087
:: for key high_seqno = 2337
for vbucket vb_20
:: for key high_seqno = 2354
:: for key uuid = 226337594275581
:: for key purge_seqno = 0
for vbucket vb_30
:: for key high_seqno = 2327
:: for key uuid = 229555825112404
:: for key purge_seqno = 0
for vbucket vb_26
:: for key purge_seqno = 0
:: for key uuid = 114252468159585
:: for key high_seqno = 2355
for vbucket vb_27
:: for key purge_seqno = 0
:: for key uuid = 86374568011280
:: for key high_seqno = 2328
------------Node 10.6.2.148-----------
for vbucket vb_9
:: for key high_seqno = 2337
:: for key uuid = 224689180220009
:: for key purge_seqno = 0
for vbucket vb_10
:: for key high_seqno = 2353
:: for key uuid = 98217872738904
:: for key purge_seqno = 0
for vbucket vb_3
:: for key high_seqno = 2356
:: for key uuid = 189410128542953
:: for key purge_seqno = 0
for vbucket vb_2
:: for key high_seqno = 2327
:: for key uuid = 202175152188264
:: for key purge_seqno = 0
for vbucket vb_29
:: for key purge_seqno = 0
:: for key uuid = 59640339796757
:: for key high_seqno = 2355
for vbucket vb_13
:: for key purge_seqno = 0
:: for key uuid = 233007425883197
:: for key high_seqno = 2353
for vbucket vb_15
:: for key purge_seqno = 0
:: for key uuid = 21343486345542
:: for key high_seqno = 2354
for vbucket vb_14
:: for key purge_seqno = 0
:: for key uuid = 9666779467087
:: for key high_seqno = 2337
for vbucket vb_17
:: for key purge_seqno = 0
:: for key uuid = 50245200692736
:: for key high_seqno = 2355
for vbucket vb_16
:: for key high_seqno = 2339
:: for key uuid = 113002939605931
:: for key purge_seqno = 0
for vbucket vb_19
:: for key purge_seqno = 0
:: for key uuid = 251498636389290
:: for key high_seqno = 2354
for vbucket vb_18
:: for key high_seqno = 2338
:: for key uuid = 5501229932749
:: for key purge_seqno = 0
for vbucket vb_22
:: for key high_seqno = 2355
:: for key uuid = 247498759341926
:: for key purge_seqno = 0
for vbucket vb_23
:: for key purge_seqno = 0
:: for key uuid = 74620381341917
:: for key high_seqno = 2338
for vbucket vb_25
:: for key high_seqno = 2327
:: for key uuid = 266765133502323
:: for key purge_seqno = 0
for vbucket vb_27
:: for key purge_seqno = 0
:: for key uuid = 86374568011280
:: for key high_seqno = 2328
------------Node 10.6.2.149-----------
for vbucket vb_5
:: for key high_seqno = 2327
:: for key uuid = 272505847148341
:: for key purge_seqno = 0
for vbucket vb_4
:: for key purge_seqno = 0
:: for key uuid = 170782289546886
:: for key high_seqno = 2356
for vbucket vb_6
:: for key purge_seqno = 0
:: for key uuid = 23856384718339
:: for key high_seqno = 2355
for vbucket vb_11
:: for key purge_seqno = 0
:: for key uuid = 91922888028629
:: for key high_seqno = 2339
for vbucket vb_19
:: for key purge_seqno = 0
:: for key uuid = 251498636389290
:: for key high_seqno = 2354
for vbucket vb_12
:: for key purge_seqno = 0
:: for key uuid = 75150401842511
:: for key high_seqno = 2339
for vbucket vb_17
:: for key purge_seqno = 0
:: for key uuid = 50245200692736
:: for key high_seqno = 2355
for vbucket vb_18
:: for key high_seqno = 2338
:: for key uuid = 5501229932749
:: for key purge_seqno = 0
for vbucket vb_20
:: for key high_seqno = 2354
:: for key uuid = 226337594275581
:: for key purge_seqno = 0
for vbucket vb_21
:: for key purge_seqno = 0
:: for key uuid = 131979757081143
:: for key high_seqno = 2338
for vbucket vb_22
:: for key high_seqno = 2355
:: for key uuid = 247498759341926
:: for key purge_seqno = 0
for vbucket vb_23
:: for key purge_seqno = 0
:: for key uuid = 74620381341917
:: for key high_seqno = 2338
for vbucket vb_24
:: for key high_seqno = 2357
:: for key uuid = 184565815683126
:: for key purge_seqno = 0
for vbucket vb_25
:: for key purge_seqno = 0
:: for key uuid = 266765133502323
:: for key high_seqno = 2327
for vbucket vb_31
:: for key purge_seqno = 0
:: for key uuid = 174005664911105
:: for key high_seqno = 2357
for vbucket vb_30
:: for key high_seqno = 2327
:: for key uuid = 229555825112404
:: for key purge_seqno = 0