Details
-
Bug
-
Resolution: Duplicate
-
Major
-
Cheshire-Cat
-
CC 7.0 build 5017
-
Untriaged
-
1
-
Unknown
Description
Repro steps:
- Create hundreds of indexes on an empty bucket with deferred build. (Note: I never issued the build commend, just the create commands with deferred builds. Also, I was trying to create 3,000 empty indexes using cbindex but was not monitoring it, and at some point before ~1,700 indexes the create calls started returning errors. Unfortunately I did not capture those as I'm used to such attempts on my laptop eventually starting to fail, probably due to too high a demand for resources, and don't expect them to be due to a bug.)
- Drop the bucket and start a rebalance of all nodes immediately. (Rebalance command: curl -u Administrator:asdasd -X POST "http://localhost:9000/controller/rebalance" -d "knownNodes=n_0%40127.0.0.1%2Cn_1%40127.0.0.1%2Cn_2%40127.0.0.1%2Cn_3%40127.0.0.1")
- Rebalance succeeds. (If it fails stuck in PrepareTopology change, it might be repro of
MB-45919, which is what I was trying to reproduce at the time but have not been able to so far.) - plasma consumes lots of CPU for several minutes afterwards. Profile (attached) indicates this is doing lots of work under swapperWorker.
I will attach the profile I captured:
- 100-second CPU sample: pprof.indexer.samples.cpu.001.pb.gz
- heap mem profile: pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
- allocs mem profile: pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
Will also attach logs.