Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4292

ep-engine does not flush expired replica items if expiry time is 2-3 seconds (memory leak)

    Details

      Description

      to reproduce :

      add 2k items with expiry = 0
      then add 10k items each 50k where expiry is set to 2 seconds
      wait until expiry pager runs( you can modify the expiry pager time to 120 seconds instead of the default which is 60 minutes)

      then look at the active and replica items:

      in my case : active items : 2000 , replica items on node 1: 6700 and node2: 7800 items

      i used mbstats hash to get the number of items in acrive and replica vbucket :

      vb_38 is active on node 10.1.5.38 and replica in 10.1.5.54

      vb_38:counted is 1 for the active node versus 16 on the replica

      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.54:11210 hash detail | grep vb_38:
      vb_38:counted: 16
      vb_38:histo_0,1: 3063
      vb_38:histo_1,2: 16
      vb_38:locks: 5
      vb_38:max_depth: 1
      vb_38:mem_size: 2288
      vb_38:mem_size_counted: 2288
      vb_38:min_depth: 0
      vb_38:reported: 16
      vb_38:resized: 0
      vb_38:size: 3079
      vb_38:state: replica
      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.38:11210 hash detail | grep vb_38:
      vb_38:counted: 1
      vb_38:histo_0,1: 3078
      vb_38:histo_1,2: 1
      vb_38:locks: 5
      vb_38:max_depth: 1
      vb_38:mem_size: 143
      vb_38:mem_size_counted: 143
      vb_38:min_depth: 0
      vb_38:reported: 1
      vb_38:resized: 0
      vb_38:size: 3079
      vb_38:state: active
      root@ubuntu1004-64:~#

      because of this the memory usage also remains higher than it should be :

      32613 membase 20 0 1248m 1.1g 3088 S 2 45.8 2:43.53 memcached

      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.54:11210 all | grep mem_
      ep_mem_high_wat: 943718400
      ep_mem_low_wat: 754974720
      mem_used: 27999326

      mem_used is 279MB
      if i restart the node mem_used drops to 60 meg and memcached resident memory also becomes 70 MB ( down from 1.1 GB )

      after restart active and replica items count will also drop to 1000

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        karan Karan Kumar (Inactive) added a comment -

        Was able to reproduce this using simple workload, which loaded about 60K keys, with expiration and deletes.

        Show
        karan Karan Kumar (Inactive) added a comment - Was able to reproduce this using simple workload, which loaded about 60K keys, with expiration and deletes.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #131 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/131/)

        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #131 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/131/ )
        Hide
        mikew Mike Wiederhold added a comment -

        This issue appears to be fixed in the master branch of ep-engine. I looked over the place where the race condition was and one of Chiyoung's other changes appears to have fixed this problem. Sending back to QA for testing.

        Show
        mikew Mike Wiederhold added a comment - This issue appears to be fixed in the master branch of ep-engine. I looked over the place where the race condition was and one of Chiyoung's other changes appears to have fixed this problem. Sending back to QA for testing.
        Hide
        steve Steve Yen added a comment -

        needs a backport to 1.8.1 (if not already there) - please resolve/close once 1.8.1 has the fix.

        Show
        steve Steve Yen added a comment - needs a backport to 1.8.1 (if not already there) - please resolve/close once 1.8.1 has the fix.
        Hide
        chiyoung Chiyoung Seo added a comment -

        Three fixes were already backported to 1.8.1 branch

        Show
        chiyoung Chiyoung Seo added a comment - Three fixes were already backported to 1.8.1 branch

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes