Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
3.0
-
Security Level: Public
-
Build 3.0.0-928
Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2680 v2 (40 vCPU)
Memory = 256 GB
Disk = RAID 10 SSD
-
Untriaged
-
Centos 64-bit
-
Yes
-
June 30 - July 18
Description
2 nodes, 5 buckets, 200K x 1KB docs per bucket, 2K updates per bucket.
You can see that mem_used for bucket-1 increased from ~600M to ~1250MB after 5 hours.
It doesn't look like a fragmentation issue, at least allocator stats don't indicate that:
MALLOC: 1575414200 ( 1502.4 MiB) Bytes in use by application
MALLOC: + 24248320 ( 23.1 MiB) Bytes in page heap freelist
MALLOC: + 77763952 ( 74.2 MiB) Bytes in central cache freelist
MALLOC: + 3931648 ( 3.7 MiB) Bytes in transfer cache freelist
MALLOC: + 27337432 ( 26.1 MiB) Bytes in thread cache freelists
MALLOC: + 7663776 ( 7.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 1716359328 ( 1636.8 MiB) Actual memory used (physical + swap)
MALLOC: + 1581056 ( 1.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 1717940384 ( 1638.4 MiB) Virtual address space used
MALLOC:
MALLOC: 94773 Spans in use
MALLOC: 36 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
Please notice that actual RAM usage (RSS) is pretty stable.
Another issue is dropping mem_used for bucket-2 and bucket-4 and these errors:
Mon Jul 7 10:24:59.559952 PDT 3: (bucket-2) Total memory in memoryDeallocated() >= GIGANTOR !!! Disable the memory tracker...
Mon Jul 7 10:54:58.109779 PDT 3: (bucket-4) Total memory in memoryDeallocated() >= GIGANTOR !!! Disable the memory tracker...
Mon Jul 7 10:54:58.109779 PDT 3: (bucket-4) Total memory in memoryDeallocated() >= GIGANTOR !!! Disable the memory tracker...