Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46035

Plasma high CPU consumption for several minutes after drop bucket with many empty indexes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.0.0
    • Cheshire-Cat
    • storage-engine
    • CC 7.0 build 5017
    • Untriaged
    • 1
    • Unknown

    Description

      Repro steps:

      1. Create hundreds of indexes on an empty bucket with deferred build. (Note: I never issued the build commend, just the create commands with deferred builds. Also, I was trying to create 3,000 empty indexes using cbindex but was not monitoring it, and at some point before ~1,700 indexes the create calls started returning errors. Unfortunately I did not capture those as I'm used to such attempts on my laptop eventually starting to fail, probably due to too high a demand for resources, and don't expect them to be due to a bug.)
      2. Drop the bucket and start a rebalance of all nodes immediately. (Rebalance command: curl -u Administrator:asdasd -X POST "http://localhost:9000/controller/rebalance" -d "knownNodes=n_0%40127.0.0.1%2Cn_1%40127.0.0.1%2Cn_2%40127.0.0.1%2Cn_3%40127.0.0.1")
      3. Rebalance succeeds. (If it fails stuck in PrepareTopology change, it might be repro of MB-45919, which is what I was trying to reproduce at the time but have not been able to so far.)
      4. plasma consumes lots of CPU for several minutes afterwards. Profile (attached) indicates this is doing lots of work under swapperWorker.

      I will attach the profile I captured:

      1. 100-second CPU sample: pprof.indexer.samples.cpu.001.pb.gz
      2. heap mem profile: pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
      3. allocs mem profile: pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz

      Will also attach logs.

      Attachments

        1. cpu_profile_top50.txt
          4 kB
        2. cpu_profile_web.png
          cpu_profile_web.png
          440 kB
        3. createCollDefer
          1 kB
        4. n_0.tar.gz
          38.15 MB
        5. n_1.tar.gz
          18.80 MB
        6. n_2.tar.gz
          18.64 MB
        7. n_3.tar.gz
          19.52 MB
        8. pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
          118 kB
        9. pprof.indexer.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
          118 kB
        10. pprof.indexer.samples.cpu.001.pb.gz
          98 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            kevin.cherkauer Kevin Cherkauer (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty