Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
This was seen while simulating an expiry workload :
a) by setting TTL at a bucket level (600s), and creating an index, OR
b) by inserting documents having TTL and using META.expiration() as the index key.
After all the documents expire, it was observed using PlasmaDiag that lot of pages towards the beginning of the index have deltaLen=1 & number of records as zero. On one such test, 40K pages out of total 48K pages had zero records. They seem to have either SwapOut delta (eviction) or Reloc delta only. These pages seem to take ~ 100 bytes.
curl -o - 127.0.0.1:9102/plasmaDiag -u Administrator:asdasd -X GET -d '{"Cmd":"memoryUsage", "Args":["2:6"]}'
|
[0] "" deltaLen:1 numItems:0 memused:120
|
[1] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3135353138373334323232393515000100000000000000 deltaLen:1 numItems:0 memused:120
|
[2] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3231313831323839333636333415000100000000000000 deltaLen:1 numItems:0 memused:120
|
[3] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3236373631353033393435333915000100000000000000 deltaLen:1 numItems:0 memused:120
|
[4] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3331303638363837343534373915000100000000000000 deltaLen:1 numItems:0 memused:120
|
[5] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3335373437303336343038373615000100000000000000 de
|
curl -o - 127.0.0.1:9102/plasmaDiag -u Administrator:asdasd -X GET -d '{"Cmd":"dumpPage", "Args":["2:6", "2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3231313831323839333636333415000100000000000000"]}'
|
----swapin-----
|
-------swapout------ max:item key:<ud>>>>2101635887949-airline_2676150394539)</ud> val:<ud>((nil))</ud> sn:1 insert: false, offset:14016543814
|
-------delta------ op:compress[false]purge[false]op[opMetaDelta]
|
curl -o - 127.0.0.1:9102/plasmaDiag -u Administrator:asdasd -X GET -d '{"Cmd":"dumpPage", "Args":["1:2", "2300000008053e3e34313431382d00006169726c696e655f3937363335343433343836383115000100000000000000"]}'
------reloc----- max:item key:<ud>>>41675-airline_9474920163539)</ud> val:<ud>((nil))</ud> sn:1 insert: false, offset:6276721099
curl -o - 127.0.0.1:9102/plasmaDiag -u Administrator:asdasd -X GET -d '{"Cmd":"dumpPagesPurgeRatio", "Args":["2:6"]}'
|
|
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[0] "" deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[1] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3135353138373334323232393515000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[2] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3231313831323839333636333415000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[3] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3236373631353033393435333915000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[4] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3331303638363837343534373915000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[5] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3335373437303336343038373615000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[6] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3430303032323538353834323015000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
[7] 2c00000008053e3e3e323130313633353838373934392d00006169726c696e655f3434313937383339363532373215000100000000000000 deltaLen:1 numItems:0 itemsCount:0 recCount:0 purgeRatio:0
|
|
This is seen because during compactFull marshal, we may purge stale records but we do not check if those pages are merge candidates. As a fix, we can probably merge these pages in the background.
Also a full index scan with all items expired took ~600ms. I am not sure how much of the total query time was contributed by traversal of the first 40K pages which did not have any records.(avg page size 4k, item size : 55).
Overall we can save some memory as well as CPU during scans by cleaning up the empty pages.
Attachments
Issue Links
- relates to
-
MB-51759 Low scan throughput for expiry workload
- Open
For Gerrit Dashboard: MB-49325 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
165128,9 | MB-49325: Merge Empty Pages during LogCleaning | unstable | plasma | Status: NEW | 0 | +1 |
166984,6 | MB-49325: Merge Empty Pages during Persist | unstable | plasma | Status: NEW | 0 | -1 |
167308,10 | MB-49325: Add diag command to list holes in the index | unstable | plasma | Status: NEW | 0 | 0 |
166983,9 | MB-49325: Add method for merging empty pages | unstable | plasma | Status: MERGED | +2 | +1 |
166985,13 | MB-49325: Merge empty pages by HoleCleaner | unstable | plasma | Status: MERGED | +2 | +1 |
167882,6 | MB-49325: Add configs for hole cleaning | unstable | indexing | Status: MERGED | +2 | +1 |