Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23347

Very low rate of insert operations due to lock contention



    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • memcached
    • CentOS 7.3, E5-2630 v4 (40 vCPU), 64GB RAM
    • Untriaged
    • Centos 64-bit
    • Yes


      Rate of insert operations on a single node dropped from 600-700K to 50K after these changes in build 2296:

      CHANGELOG for ep-engine
       * Commit: 3311f7af9e5dec52391dd3c4b3e4a73c957dab12 (in build: 2296)
         Author: Trond Norbye
         MB-22163: Optimize get_if for value eviction
         If we're using value eviction we should be able to call
         the filter method with the meta-data stored in the hashtable
         without having to fetch the value off disk.
         Change-Id: I9850ee1679094983d22b0af0c374ac02d73fdb8a
         Reviewed-on: http://review.couchbase.org/74903
         Reviewed-by: Dave Rigby 
         Tested-by: Build Bot 
      CHANGELOG for memcached
       * Commit: aef7929f01907105d4044f3e6915faae54fbbb14 (in build: 2296)
         Author: Trond Norbye
         MB-22163: Use get_if to fetch items for mutations
         Reintroduce the get_if patch for mutations. It had to be
         reverted due to ep-engine returning KEY_EEXISTS for locked
         items instead of ENGINE_LOCKED (I thought I had fixed that,
         but it turned out that was pending).
         commit badccafaa2e670719c181a9ff39ffa2add5e19d6
         Author: Trond Norbye 
         Date:   Thu Mar 9 07:48:36 2017 +0100
             MB-22163: Use get_if to fetch items for mutations
             The mutations _might_ need to preserve the xattrs from
             the previous version of a document so we want to retrieve
             only the documents which already contains xattrs.
             The motivation for moving to get_if instead of get is
             that the underlying engine may provide the metadata to
             the callback without having the actual value for the
             document in memory (whereas get() would require the
             engine to fetch the document off disk). This would
             result in a significant performance improvement in
             DGM situations where we're running with value eviction.
             For full eviction we must go to disk anyway.
         Change-Id: I4c726e717b0351afaa6bb7a22c3ee8ce2d6e1629
         Reviewed-on: http://review.couchbase.org/75143
         Tested-by: Build Bot 
         Reviewed-by: Dave Rigby 


      cbc-pillowfight --spec couchbase:// --password password --batch-size 1000 --num-items 25000000 --num-threads 24 --min-size 1024 --max-size 1024 --populate-only

      perf indicates obvious issues with locking. Updates are fine.


          22.45 MB
        2. ops.png
          554 kB
        3. perf.txt
          2.79 MB
        4. pstack.txt
          207 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              trond Trond Norbye
              pavelpaulau Pavel Paulau (Inactive)
              0 Vote for this issue
              2 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes