Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
3.0
-
Security Level: Public
-
CentOS 6.x 8*8 clusters. Each node : 15GB RAM, 450Gb HDD
-
Untriaged
-
Unknown
-
June 30 - July 18
Description
Continuation on MB-11448 [This system test was modified to maintain 25% active resident item ratio compared to the previous run which was <1% ]
Build
--------
3.0.0-840(xdcr on upr, internal replication on upr)
Clusters
-----------
Source : http://172.23.105.44:8091/
Destination : http://172.23.105.54:8091/
The clusters are available to investigate.
Steps
--------
1. Load on both clusters till vb_active_resident_items_ratio < 50.
2. Access phase with 50% gets, 50%deletes runs for 3 hours
Bucket under discussion: standardbucket (has uni-xdcr)
Another bucket standardbucket1 also has uni-xdcr but it's memory usage is comparable to that of other nodes. So it may not be due to XDCR.
Bucket capacity:5GB but ep_value_size is >10gb while the node is at 0% dgm.
Rest of the nodes are at 75% dgm with 3.8GB.
Cluster C1: http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/
See attached screenshots and logs. Marking this as a test blocker because rebalance in this scenario will hang.
Attachments
Issue Links
- relates to
-
MB-11536 [system test] [kv unix] rebalance hang due to uneven memmory distribution between nodes
- Closed
For Gerrit Dashboard: MB-11488 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
38758,3 | MB-11488: Check memory usage before upr-backfill | master | ep-engine | Status: MERGED | +2 | +1 |
38818,1 | MB-11488: Throttling UPR-BACKFILL | master | ep-engine | Status: ABANDONED | 0 | 0 |
38941,3 | MB-11488 Set the thread local to NULL before logging. | master | ep-engine | Status: MERGED | +2 | +1 |
38942,4 | MB-11488 Set the thread local to NULL before calling thread pool API | master | ep-engine | Status: MERGED | +2 | +1 |
38948,2 | MB-11488 Fix to the memory leak in PassiveStream. | master | ep-engine | Status: MERGED | +2 | +1 |
38950,2 | MB-11488 Fix to the memory leak in Stream instance. | master | ep-engine | Status: MERGED | +2 | +1 |
38979,3 | MB-11488 more cases for thread local to NULL, avoid repeated calls | master | ep-engine | Status: ABANDONED | 0 | 0 |