Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
5.5.0
-
None
-
Untriaged
-
Unknown
Description
Latest show-fast results indicate that rebalance is taking longer on DGM with latest Vulcan build.
e.g. Swap rebalance with previous build took 58.9 mins. With 5.5.0-2884, it is 74.9 mins.
http://showfast.sc.couchbase.com/#/timeline/Linux/reb/kv/DGM
I compared logs from following two runs:
- Old: http://showfast.sc.couchbase.com/#/runs/reb_swap_1B_4nodes_kv_titan/5.5.0-2845
- New: http://showfast.sc.couchbase.com/#/runs/reb_swap_1B_4nodes_kv_titan/5.5.0-2884
More vBuckets took longer to persist high seq #. * Total 512 vBuckets are being moved.
- In the old build, 149 vBuckets took longer than 10 mins to move.
- In the new build, 284 vBuckets took longer than 10 mins to move. Most of this was spent in the persistence of seq #s.
In the old build, there are 211 instances of slow seqno persistence in memcached log of node being swaped-in (node 104).
Where as in the new build, there are 2971 of these instances.
/Users/poonam/Logs/perf/vulcan_regre/old/cbcollect_info_ns_1@172.23.96.104_20180605-080634$ grep SEQNO_PERSISTENCE memcached.log | grep Slow | wc -l
211
/Users/poonam/Logs/perf/vulcan_regre/new/cbcollect_info_ns_1@172.23.96.104_20180612-030355$ grep SEQNO_PERSISTENCE memcached.log | grep Slow | wc -l
2971
The new build also has more OOM errors.
