Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7729

memcached consumes 100% CPU with latest ep-engine changes

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      PCI Team - Sprint 6

      Description

      To reproduce:

      • start cluster
      • initialize it

      Memcached immediately starts consuming 100% CPU. Reverting f6b583f3760cc1e7df85b5bf3abbc2e016a270fc in ep-engine returns everything back to normal.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        pavelpaulau Pavel Paulau added a comment - - edited

        For some reason disk queue doesn't become zero during load phase (even after a couple of hours). It's slightly blocking issue, reproduced twice.

        Otherwise it fails because of:

        Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting. Messages:
        Tue May 7 08:09:05.181138 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.17 - Suspend for 5.00 secs
        Tue May 7 08:09:07.370985 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs
        Tue May 7 08:09:13.346550 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs

        Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139. Restarting. Messages: Tue May 7 09:15:17.398602 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs
        Tue May 7 09:15:19.547858 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.16 - Suspend for 5.00 secs
        Tue May 7 09:15:23.401947 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs
        Tue May 7 09:15:27.917558 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.16 - Suspend for 5.00 secs
        Tue May 7 09:15:29.439715 PDT 3: (default) TAP (Producer)

        and so on.

        Sorry, this build doesn't look good.

        Show
        pavelpaulau Pavel Paulau added a comment - - edited For some reason disk queue doesn't become zero during load phase (even after a couple of hours). It's slightly blocking issue, reproduced twice. Otherwise it fails because of: Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting. Messages: Tue May 7 08:09:05.181138 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.17 - Suspend for 5.00 secs Tue May 7 08:09:07.370985 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs Tue May 7 08:09:13.346550 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139. Restarting. Messages: Tue May 7 09:15:17.398602 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs Tue May 7 09:15:19.547858 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.16 - Suspend for 5.00 secs Tue May 7 09:15:23.401947 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.15 - Suspend for 5.00 secs Tue May 7 09:15:27.917558 PDT 3: (default) TAP (Producer) eq_tapq:replication_ns_1@172.23.96.16 - Suspend for 5.00 secs Tue May 7 09:15:29.439715 PDT 3: (default) TAP (Producer) and so on. Sorry, this build doesn't look good.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        per bug scrub, mike, pls take a look.

        Show
        maria Maria McDuff (Inactive) added a comment - per bug scrub, mike, pls take a look.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        andrei, pls verify / close.

        Show
        maria Maria McDuff (Inactive) added a comment - andrei, pls verify / close.
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        don't see it onn 2.0.2-803

        Show
        andreibaranouski Andrei Baranouski added a comment - don't see it onn 2.0.2-803
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #488 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/488/)
        MB-7729: Fix 100% CPU Issue (Revision 4098eb08a2b1a7ba1d4bc70448a56776c03dcba9)

        Result = SUCCESS
        Mike Wiederhold :
        Files :

        • src/bgfetcher.cc
        • src/vbucket.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #488 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/488/ ) MB-7729 : Fix 100% CPU Issue (Revision 4098eb08a2b1a7ba1d4bc70448a56776c03dcba9) Result = SUCCESS Mike Wiederhold : Files : src/bgfetcher.cc src/vbucket.cc

          People

          • Assignee:
            andreibaranouski Andrei Baranouski
            Reporter:
            Aliaksey Artamonau Aliaksey Artamonau
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes