Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
3.0
-
Security Level: Public
-
None
-
centOS 64 bit each node - 5 cores 15GB
-
Untriaged
-
Unknown
-
June 30 - July 18
Description
Scenario
-------------
1. Load on both clusters till vb_active_resident_items_ratio < 50 on standardbucket, <70 on standardbucket1, 20M on sasl.
2. Access phase with 100% gets runs for 3 hours with 50% gets and 50% deletes
3. Rebalance-out 1 node at cluster1
4. Rebalance-in 1 node at cluster1
5. Failover and remove node at cluster1
6. Failover and add-back node at cluster1
7. Rebalance-out 1 node at cluster2
8. Rebalance-in 1 node at cluster2
9. Failover and remove node at cluster2
10. Failover and add-back node at cluster2
11. Soft restart all nodes in cluster1 one by one
12. Soft restart all nodes in cluster2 one by one
During phase 11 it was seen that cluster 2 had uneven resident ratio(one node .55 had 0.2% while others ranging from 8-25%) . During phase 12, active resident ratio suddenly dropped to 0 on all nodes. Test log shows cluster warmup was issued, however, no such activity is apparent from GUI logs. KIndly check if warmup resulted in this state.
Please see from the screenshot that no nodes have reached 5GB memory usage (allotted bucket capacity) yet active resident ratio for all nodes is 0. I've previously loaded upto 150M docs(same size) onto 5GB buckets without hitting this issue. Looks like we have some memory bloating happening.
Attaching cbcollect info. Live cluster : http://172.23.105.54:8091/index.html