System test : observed slow swap-rebalance in 2.0.2 ( 15 hours+)

1. Setup a 6 node cluster with 2 buckets.
2. Each bucket has 1 ddoc and 2 views.
3. Load 60M items on both buckets, data-size 512 bytes consistent
4. Start mutations on the cluster for 2 hours.
5. Wait for initial indexing to complete.
6. Swap rebalance 1-1 node.

• Seeing very high swap ~ 5G on most of the nodes.
• Further inspection of process using swap shows memcached is using most of swap space.
• The memory stats show very high fragmentation
• Views are highly fragmented ( 91 percent) across the cluster.

]# pgrep memcached
9551

1. grep --color VmSwap /proc/9551/status
   VmSwap: 5219956 kB

[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 14060977616 (13409.6 MiB) Bytes in use by application
MALLOC: + 263405568 ( 251.2 MiB) Bytes in page heap freelist
MALLOC: + 5191139360 ( 4950.7 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 2662928 ( 2.5 MiB) Bytes in thread cache freelists
MALLOC: + 137932952 ( 131.5 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 19656118424 (18745.5 MiB) Actual memory used (physical + swap)
MALLOC: + 126885888 ( 121.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 19783004312 (18866.5 MiB) Virtual address space used
MALLOC:
MALLOC: 2348103 Spans in use
MALLOC: 20 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 838 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 2856430 objs; 43.6 MiB; 43.6 cum MiB
class 3 [ 32 bytes ] : 684331 objs; 20.9 MiB; 64.5 cum MiB
class 4 [ 48 bytes ] : 10987 objs; 0.5 MiB; 65.0 cum MiB
class 5 [ 64 bytes ] : 21645 objs; 1.3 MiB; 66.3 cum MiB
class 6 [ 80 bytes ] : 5347446 objs; 408.0 MiB; 474.3 cum MiB
class 7 [ 96 bytes ] : 905 objs; 0.1 MiB; 474.4 cum MiB
class 8 [ 112 bytes ] : 130 objs; 0.0 MiB; 474.4 cum MiB
class 9 [ 128 bytes ] : 337 objs; 0.0 MiB; 474.4 cum MiB
class 10 [ 144 bytes ] : 224 objs; 0.0 MiB; 474.4 cum MiB
class 11 [ 160 bytes ] : 260 objs; 0.0 MiB; 474.5 cum MiB
class 12 [ 176 bytes ] : 43 objs; 0.0 MiB; 474.5 cum MiB
class 13 [ 192 bytes ] : 83 objs; 0.0 MiB; 474.5 cum MiB
class 14 [ 208 bytes ] : 64 objs; 0.0 MiB; 474.5 cum MiB
class 15 [ 224 bytes ] : 70 objs; 0.0 MiB; 474.5 cum MiB
class 16 [ 240 bytes ] : 66 objs; 0.0 MiB; 474.6 cum MiB
class 17 [ 256 bytes ] : 127 objs; 0.0 MiB; 474.6 cum MiB
class 18 [ 288 bytes ] : 41 objs; 0.0 MiB; 474.6 cum MiB
class 19 [ 320 bytes ] : 76 objs; 0.0 MiB; 474.6 cum MiB
class 20 [ 352 bytes ] : 119 objs; 0.0 MiB; 474.7 cum MiB
class 21 [ 384 bytes ] : 57 objs; 0.0 MiB; 474.7 cum MiB
class 22 [ 416 bytes ] : 28 objs; 0.0 MiB; 474.7 cum MiB
class 23 [ 448 bytes ] : 52 objs; 0.0 MiB; 474.7 cum MiB
class 24 [ 480 bytes ] : 36 objs; 0.0 MiB; 474.7 cum MiB
class 25 [ 512 bytes ] : 907 objs; 0.4 MiB; 475.2 cum MiB
class 26 [ 576 bytes ] : 165 objs; 0.1 MiB; 475.3 cum MiB
class 27 [ 640 bytes ] : 18 objs; 0.0 MiB; 475.3 cum MiB
class 28 [ 704 bytes ] : 60 objs; 0.0 MiB; 475.3 cum MiB
class 29 [ 768 bytes ] : 4872097 objs; 3568.4 MiB; 4043.7 cum MiB
class 30 [ 896 bytes ] : 42 objs; 0.0 MiB; 4043.8 cum MiB
class 31 [ 1024 bytes ] : 96 objs; 0.1 MiB; 4043.9 cum MiB
class 32 [ 1152 bytes ] : 50 objs; 0.1 MiB; 4043.9 cum MiB
class 33 [ 1280 bytes ] : 38 objs; 0.0 MiB; 4044.0 cum MiB
class 34 [ 1408 bytes ] : 33 objs; 0.0 MiB; 4044.0 cum MiB
class 35 [ 1536 bytes ] : 20 objs; 0.0 MiB; 4044.0 cum MiB
class 36 [ 1792 bytes ] : 78 objs; 0.1 MiB; 4044.2 cum MiB
class 37 [ 2048 bytes ] : 122 objs; 0.2 MiB; 4044.4 cum MiB
class 38 [ 2304 bytes ] : 12 objs; 0.0 MiB; 4044.4 cum MiB
class 39 [ 2560 bytes ] : 5 objs; 0.0 MiB; 4044.5 cum MiB
class 40 [ 2816 bytes ] : 16 objs; 0.0 MiB; 4044.5 cum MiB
class 42 [ 3328 bytes ] : 52 objs; 0.2 MiB; 4044.7 cum MiB
class 43 [ 4096 bytes ] : 29 objs; 0.1 MiB; 4044.8 cum MiB
class 47 [ 6656 bytes ] : 64 objs; 0.4 MiB; 4045.2 cum MiB
class 48 [ 8192 bytes ] : 14 objs; 0.1 MiB; 4045.3 cum MiB
class 49 [ 9216 bytes ] : 27 objs; 0.2 MiB; 4045.5 cum MiB
class 53 [ 16384 bytes ] : 7 objs; 0.1 MiB; 4045.6 cum MiB
class 54 [ 20480 bytes ] : 1 objs; 0.0 MiB; 4045.7 cum MiB
class 57 [ 32768 bytes ] : 13 objs; 0.4 MiB; 4046.1 cum MiB
class 61 [ 65536 bytes ] : 1 objs; 0.1 MiB; 4046.1 cum MiB
class 85 [ 262144 bytes ] : 1 objs; 0.2 MiB; 4046.4 cum MiB
------------------------------------------------
PageHeap: 26 sizes; 251.2 MiB free; 121.0 MiB unmapped
------------------------------------------------
1 pages * 27211 spans ~ 212.6 MiB; 212.6 MiB cum; unmapped: 5.4 MiB; 5.4 MiB cum
2 pages * 3062 spans ~ 47.8 MiB; 260.4 MiB cum; unmapped: 12.3 MiB; 17.7 MiB cum
3 pages * 1018 spans ~ 23.9 MiB; 284.3 MiB cum; unmapped: 18.4 MiB; 36.1 MiB cum
4 pages * 721 spans ~ 22.5 MiB; 306.8 MiB cum; unmapped: 20.1 MiB; 56.2 MiB cum
5 pages * 175 spans ~ 6.8 MiB; 313.7 MiB cum; unmapped: 6.8 MiB; 63.0 MiB cum
6 pages * 113 spans ~ 5.3 MiB; 319.0 MiB cum; unmapped: 5.3 MiB; 68.3 MiB cum
7 pages * 50 spans ~ 2.7 MiB; 321.7 MiB cum; unmapped: 2.7 MiB; 71.0 MiB cum
8 pages * 101 spans ~ 6.3 MiB; 328.0 MiB cum; unmapped: 6.1 MiB; 77.1 MiB cum
9 pages * 23 spans ~ 1.6 MiB; 329.6 MiB cum; unmapped: 1.6 MiB; 78.7 MiB cum
10 pages * 18 spans ~ 1.4 MiB; 331.0 MiB cum; unmapped: 1.4 MiB; 80.1 MiB cum
11 pages * 10 spans ~ 0.9 MiB; 331.9 MiB cum; unmapped: 0.9 MiB; 80.9 MiB cum
12 pages * 11 spans ~ 1.0 MiB; 332.9 MiB cum; unmapped: 1.0 MiB; 82.0 MiB cum
13 pages * 1 spans ~ 0.1 MiB; 333.0 MiB cum; unmapped: 0.1 MiB; 82.1 MiB cum
14 pages * 2 spans ~ 0.2 MiB; 333.2 MiB cum; unmapped: 0.2 MiB; 82.3 MiB cum
15 pages * 3 spans ~ 0.4 MiB; 333.6 MiB cum; unmapped: 0.4 MiB; 82.6 MiB cum
16 pages * 3 spans ~ 0.4 MiB; 334.0 MiB cum; unmapped: 0.2 MiB; 82.9 MiB cum
17 pages * 1 spans ~ 0.1 MiB; 334.1 MiB cum; unmapped: 0.0 MiB; 82.9 MiB cum
20 pages * 1 spans ~ 0.2 MiB; 334.2 MiB cum; unmapped: 0.2 MiB; 83.0 MiB cum
21 pages * 1 spans ~ 0.2 MiB; 334.4 MiB cum; unmapped: 0.2 MiB; 83.2 MiB cum
29 pages * 1 spans ~ 0.2 MiB; 334.6 MiB cum; unmapped: 0.2 MiB; 83.4 MiB cum
49 pages * 18 spans ~ 6.9 MiB; 341.5 MiB cum; unmapped: 6.9 MiB; 90.3 MiB cum
58 pages * 1 spans ~ 0.5 MiB; 342.0 MiB cum; unmapped: 0.5 MiB; 90.8 MiB cum
75 pages * 1 spans ~ 0.6 MiB; 342.6 MiB cum; unmapped: 0.6 MiB; 91.4 MiB cum
98 pages * 8 spans ~ 6.1 MiB; 348.7 MiB cum; unmapped: 6.1 MiB; 97.5 MiB cum
100 pages * 1 spans ~ 0.8 MiB; 349.5 MiB cum; unmapped: 0.8 MiB; 98.3 MiB cum
122 pages * 1 spans ~ 1.0 MiB; 350.4 MiB cum; unmapped: 1.0 MiB; 99.2 MiB cum
>255 large * 13 spans ~ 21.8 MiB; 372.2 MiB cum; unmapped: 21.8 MiB; 121.0 MiB cum

[root@orange-11601 ~]#
[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default memory
bytes: 8468018888
ep_kv_size: 8190890236
ep_max_data_size: 13212057600
ep_mem_high_wat: 11230248960
ep_mem_low_wat: 9909043200
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 128884714
ep_tmp_oom_errors: 0
ep_value_size: 6628874716
mem_used: 8468018888
tcmalloc_current_thread_cache_bytes: 2581856
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 126844928
total_allocated_bytes: 14068670960
total_fragmentation_bytes: 5313338896
total_free_bytes: 263061504
total_heap_bytes: 19645071360

added screenshot of the cluster