Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7972

Performance dip during rebalance on bucket not involved in rebalance

    XMLWordPrintable

Details

    • PCI Team - Sprint 3, PCI Team - Sprint 4, PCI Team - Sprint 5, PCI Team - Sprint 6, CB Bucket - 08/12 - 08/29, 12/Aug - 30/Aug

    Description

      Environment:
      -memcache-test running through client-side Moxi.
      -memcache-test workload: /root/mctest/Users/ingenthr/tmp/memcachetest-centos_x86 -h localhost -i 1000 -t 4 -K test -M 10 -F -S -l

      -Moxi config: CLI: /opt/moxi/bin/moxi -u nobody -Z usr=Administrator,pwd=password http://10.197.1.147:8091/pools/default/bucketsStreaming/beer-sample -vv -O /var/log/moxi,
      config:
      port_listen=11211,
      default_bucket_name=default,
      downstream_max=1024,
      downstream_conn_max=4,
      downstream_conn_queue_timeout=200,
      downstream_timeout=5000,
      wait_queue_timeout=200,
      connect_max_errors=5,
      connect_retry_interval=30000,
      connect_timeout=400,
      auth_timeout=100,
      cycle=200

      -4 nodes of Couchbase Server 2.0.1. c1.medium EC2 instances, beer-sample database bucket
      -2 nodes of same config added to cluster and rebalanced

      Observation:
      -Workload very steady around 8k ops/sec before rebalance
      -Workload drops to around 4k ops/sec during rebalance. Both gets and sets affected, compaction not running when load initially drops (kicks in eventually). Workload very choppy during rebalance.
      -Workload returns to 8k ops/sec after rebalance

      Client memcachetest and moxi logs:
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client1.zip
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client2.zip

      Cluster logs:
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node1.zip
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node2.zip
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node3.zip
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node4.zip
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node5.zip (aded to first 4)
      https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node6.zip (added to first 4)

      Attachments

        1. aot_wrath_200_beginning.png
          aot_wrath_200_beginning.png
          553 kB
        2. aot_wrath_200_end.png
          aot_wrath_200_end.png
          527 kB
        3. aot_wrath_200_middle.png
          aot_wrath_200_middle.png
          598 kB
        4. aot_wrath_201_beginning.png
          aot_wrath_201_beginning.png
          521 kB
        5. aot_wrath_201_end.png
          aot_wrath_201_end.png
          498 kB
        6. aot_wrath_201_middle.png
          aot_wrath_201_middle.png
          556 kB
        7. aot_wrath_202_beginning.png
          aot_wrath_202_beginning.png
          502 kB
        8. aot_wrath_202_end.png
          aot_wrath_202_end.png
          471 kB
        9. aot_wrath_202_middle.png
          aot_wrath_202_middle.png
          499 kB
        10. Couchbase2Test.properties
          1 kB
        11. mb-7972-sergey-pillowfight-test.png
          mb-7972-sergey-pillowfight-test.png
          136 kB
        12. pillowfight-libcouchbase-load.png
          pillowfight-libcouchbase-load.png
          136 kB
        13. pillowfight-libcouchbase-rebalance-after-remove-performance-drop.png
          pillowfight-libcouchbase-rebalance-after-remove-performance-drop.png
          555 kB
        14. pillowfight-libcouchbase-with-moxi.png
          pillowfight-libcouchbase-with-moxi.png
          525 kB
        15. pillowfight-libcouchbase-with-moxi-timings.txt
          133 kB
        16. pillowfight-libcouchbase-without-moxi.png
          pillowfight-libcouchbase-without-moxi.png
          518 kB
        17. pillowfight-libcouchbase-without-moxi-timings.txt
          2.37 MB
        18. roadrunner_200_beginning.png
          roadrunner_200_beginning.png
          582 kB
        19. roadrunner_200_end.png
          roadrunner_200_end.png
          542 kB
        20. roadrunner_200_midlle.png
          roadrunner_200_midlle.png
          605 kB
        21. roadrunner_201_beginning.png
          roadrunner_201_beginning.png
          574 kB
        22. roadrunner_201_end.png
          roadrunner_201_end.png
          540 kB
        23. roadrunner_201_midlle.png
          roadrunner_201_midlle.png
          607 kB
        24. RoadRunner-0.3-jar-with-dependencies.jar
          4.94 MB
        25. Screen Shot 2013-04-26 at 11.59.39 AM.png
          Screen Shot 2013-04-26 at 11.59.39 AM.png
          41 kB
        26. Screen Shot 2013-04-26 at 12.02.08 PM.png
          Screen Shot 2013-04-26 at 12.02.08 PM.png
          42 kB
        27. stats.tar
          226 kB
        28. wrath.jar
          4.30 MB
        29. ycsb_01.png
          ycsb_01.png
          576 kB
        30. ycsb_02.png
          ycsb_02.png
          573 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            perry Perry Krug
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty