Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10125

Memcached ops/sec drop if a UPR consumer is running on the cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 3.0
    • 3.0
    • None
    • Security Level: Public
    • couchbase-server-enterprise_x86_64_3.0.0-350-rel.deb
      ubuntu 12.04, amazon m1.xlarge, 15G RAM, 4 ECU
    • Untriaged
    • Ubuntu 64-bit

    Description

      Memcached ops/sec can drop to 500 ops/sec from 10K ops/sec if a UPR consumer is subscribing to the cluster. This is lot of free RAM, CPU etc while this is happening.

      Steps to reproduce:

      1. Install couchbase-server-enterprise_x86_64_3.0.0-350-rel.deb on single node (amazon m1.xlarge, 15G RAM, 4 ecu)

      2. Create default bucket

      3. Start a UPR consumer which subscribes for all 1024 vbuckets for mutations.

      4. Start front-end load using 2 mcsoda clients from 2 separate EC2 nodes (500K items, 30% sets, 70% gets)
      client1:
      ./lib/perf_engines/mcsoda.py membase://10.147.4.31:8091 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=0.3 min-value-size=256,512 max-items=500000 exit-after-creates=1 prefix=70one

      client2:
      ./lib/perf_engines/mcsoda.py membase://10.147.4.31:8091 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=0.3 min-value-size=256,512 max-items=500000 exit-after-creates=1 prefix=70two

      This generates roughly 18k ops/sec as seen in screenshot normal_cluster.png

      5. Repeat Step4 after changing the prefix 2-3 times. One of the clients starts getting very slow response as can be seen in througput_drop.png. The ops/sec drop to 500 from 9-10k.

      6. Kill the UPR consumer while client is experiencing low ops/sec. Instantly the ops/sec surge back to 10k ops/sec as can be seen in after_killing_upr_consumer.png.

      The UPR consumer is only taking 30% cpu and no memory. Its not doing any network activity as well(its running on the KV node).

      Attachments

        1. after_killing_upr_consumer.png
          after_killing_upr_consumer.png
          699 kB
        2. normal_cluster.png
          normal_cluster.png
          729 kB
        3. througput_drop.png
          througput_drop.png
          724 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            deepkaran.salooja Deepkaran Salooja
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty