Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23347

Very low rate of insert operations due to lock contention

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • memcached
    • CentOS 7.3, E5-2630 v4 (40 vCPU), 64GB RAM
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Rate of insert operations on a single node dropped from 600-700K to 50K after these changes in build 2296:

      CHANGELOG for ep-engine
       
       * Commit: 3311f7af9e5dec52391dd3c4b3e4a73c957dab12 (in build: 2296)
         Author: Trond Norbye
         MB-22163: Optimize get_if for value eviction
         
         If we're using value eviction we should be able to call
         the filter method with the meta-data stored in the hashtable
         without having to fetch the value off disk.
         
         Change-Id: I9850ee1679094983d22b0af0c374ac02d73fdb8a
         Reviewed-on: http://review.couchbase.org/74903
         Reviewed-by: Dave Rigby 
         Tested-by: Build Bot 
       
       
      CHANGELOG for memcached
       
       * Commit: aef7929f01907105d4044f3e6915faae54fbbb14 (in build: 2296)
         Author: Trond Norbye
         MB-22163: Use get_if to fetch items for mutations
         
         Reintroduce the get_if patch for mutations. It had to be
         reverted due to ep-engine returning KEY_EEXISTS for locked
         items instead of ENGINE_LOCKED (I thought I had fixed that,
         but it turned out that was pending).
         
         commit badccafaa2e670719c181a9ff39ffa2add5e19d6
         Author: Trond Norbye 
         Date:   Thu Mar 9 07:48:36 2017 +0100
         
             MB-22163: Use get_if to fetch items for mutations
         
             The mutations _might_ need to preserve the xattrs from
             the previous version of a document so we want to retrieve
             only the documents which already contains xattrs.
         
             The motivation for moving to get_if instead of get is
             that the underlying engine may provide the metadata to
             the callback without having the actual value for the
             document in memory (whereas get() would require the
             engine to fetch the document off disk). This would
             result in a significant performance improvement in
             DGM situations where we're running with value eviction.
         
             For full eviction we must go to disk anyway.
         
         Change-Id: I4c726e717b0351afaa6bb7a22c3ee8ce2d6e1629
         Reviewed-on: http://review.couchbase.org/75143
         Tested-by: Build Bot 
         Reviewed-by: Dave Rigby 
      

      Steps:

      cbc-pillowfight --spec couchbase://172.23.132.17/bucket-1 --password password --batch-size 1000 --num-items 25000000 --num-threads 24 --min-size 1024 --max-size 1024 --populate-only
      

      perf indicates obvious issues with locking. Updates are fine.

      Attachments

        1. 172.23.132.17.zip
          22.45 MB
        2. ops.png
          ops.png
          554 kB
        3. perf.txt
          2.79 MB
        4. pstack.txt
          207 kB

        Issue Links

          For Gerrit Dashboard: MB-23347
          # Subject Branch Project Status CR V

          Activity

            People

              trond Trond Norbye
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty