Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6726

Rebalance is slow when indexing/compaction and query load are going on in parallel

    Details

    • Sprint:
      PCI Team - Sprint 1, PCI Team - Sprint 5

      Description

      Cluster information:

      • 8 centos 6.2 64bit server with 4 cores CPU
      • Each server has 32 GB RAM and 400 GB SSD disk.
      • SSD disk format ext4 on /data
      • Each server has its own drive, no disk sharing with other server.
      • Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system.
      • Load 12 million items to saslbucket and 45 million items to default bucket. Each key has size from 512 bytes to 1500 bytes
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
      • Create cluster with 4 nodes installed couchbase server 2.0.0-1746

      10.6.2.37
      10.6.2.38
      10.6.2.39
      10.6.2.40

      • Data path /data
      • View path /data
      • Let load running at 6K ops, query 400 to 500 per second
      • Add 4 nodes to cluster and rebalance
        10.6.2.42
        10.6.2.43
        10.6.2.44
        10.6.2.45
      • rebalance hang. Filed bug MB-6706
      • Try rebalance again. Cluster rebalances saslbucket first and took 9 hours to complete rebalance saslbucket
      • default bucket took 32 hours to complete

      Link to collect_info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-slow-constn-enable-20120925-141314.tgz

      Each atop file size from 1.5 GB to 2.5 GB. If atop file needed, I will upload them as request

      1. master_events.log.svg
        246 kB
        Aleksey Kondratenko
      2. master_events.tar.gz
        1.05 MB
        Pavel Paulau
      1. rebalance_progress.png
        37 kB

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          thuan Thuan Nguyen created issue -
          chiyoung Chiyoung Seo made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          ketaki Ketaki Gangal made changes -
          Resolution Duplicate [ 3 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Sprint Status Current Sprint
          Assignee Chiyoung Seo [ chiyoung ] Aleksey Kondratenko [ alkondratenko ]
          ketaki Ketaki Gangal made changes -
          Link This issue is duplicated by MB-6769 [ MB-6769 ]
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Ketaki Gangal [ ketaki ]
          ketaki Ketaki Gangal made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          ketaki Ketaki Gangal made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          chiyoung Chiyoung Seo made changes -
          Sprint Status Current Sprint
          karan Karan Kumar (Inactive) made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Ketaki Gangal [ ketaki ] Aleksey Kondratenko [ alkondratenko ]
          farshid Farshid Ghods (Inactive) made changes -
          Fix Version/s 2.0-beta-2 [ 10385 ]
          Fix Version/s 2.0 [ 10114 ]
          farshid Farshid Ghods (Inactive) made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          pavelpaulau Pavel Paulau made changes -
          Labels system-test pblock system-test
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Assignee Aleksey Kondratenko [ alkondratenko ] Thuan Nguyen [ thuan ]
          Resolution Duplicate [ 3 ]
          karan Karan Kumar (Inactive) made changes -
          Resolution Duplicate [ 3 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          karan Karan Kumar (Inactive) made changes -
          Summary [system test] rebalance is slow with consistent view enable [system test] rebalance is slow with consistent view enabled, indexing/compaction and query load
          thuan Thuan Nguyen made changes -
          Assignee Thuan Nguyen [ thuan ] Chiyoung Seo [ chiyoung ]
          chiyoung Chiyoung Seo made changes -
          Assignee Chiyoung Seo [ chiyoung ] Farshid Ghods [ farshid ]
          farshid Farshid Ghods (Inactive) made changes -
          Assignee Farshid Ghods [ farshid ] Sriram Melkote [ siri ]
          steve Steve Yen made changes -
          Assignee Sriram Melkote [ siri ] Mike Wiederhold [ mikew ]
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Assignee Mike Wiederhold [ mikew ] Aleksey Kondratenko [ alkondratenko ]
          dipti Dipti Borkar made changes -
          Fix Version/s 2.0 [ 10114 ]
          Fix Version/s 2.0-beta-2 [ 10385 ]
          steve Steve Yen made changes -
          Fix Version/s 2.0.1 [ 10399 ]
          Fix Version/s 2.0 [ 10114 ]
          dipti Dipti Borkar made changes -
          Summary [system test] rebalance is slow with consistent view enabled, indexing/compaction and query load [system test] rebalance is slow when index-aware, when indexing/compaction and query load are going on in parallel
          mikew Mike Wiederhold made changes -
          Component/s ns_server [ 10019 ]
          Component/s view-engine [ 10060 ]
          Component/s couchbase-bucket [ 10173 ]
          farshid Farshid Ghods (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Ketaki Gangal [ ketaki ]
          farshid Farshid Ghods (Inactive) made changes -
          Assignee Ketaki Gangal [ ketaki ] Aleksey Kondratenko [ alkondratenko ]
          farshid Farshid Ghods (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Paulau [ pavelpaulau ]
          Sprint Status Current Sprint
          Sprint Priority 1
          pavelpaulau Pavel Paulau made changes -
          Assignee Pavel Paulau [ pavelpaulau ] Farshid Ghods [ farshid ]
          farshid Farshid Ghods (Inactive) made changes -
          Assignee Farshid Ghods [ farshid ] Aleksey Kondratenko [ alkondratenko ]
          kzeller kzeller made changes -
          Labels pblock system-test 2.0.1-release-notes pblock system-test
          pavelpaulau Pavel Paulau made changes -
          Attachment master_events.tar.gz [ 16322 ]
          jin Jin Lim made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Paulau [ pavelpaulau ]
          pavelpaulau Pavel Paulau made changes -
          Assignee Pavel Paulau [ pavelpaulau ] Aleksey Kondratenko [ alkondratenko ]
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Paulau [ pavelpaulau ]
          pavelpaulau Pavel Paulau made changes -
          Assignee Pavel Paulau [ pavelpaulau ] Aleksey Kondratenko [ alkondratenko ]
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Attachment master_events.log.svg [ 16731 ]
          pavelpaulau Pavel Paulau made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Paulau [ pavelpaulau ]
          pavelpaulau Pavel Paulau made changes -
          Fix Version/s 2.1 [ 10414 ]
          Fix Version/s 2.0.1 [ 10399 ]
          pavelpaulau Pavel Paulau made changes -
          Summary [system test] rebalance is slow when index-aware, when indexing/compaction and query load are going on in parallel Rebalance is slow when indexing/compaction and query load are going on in parallel
          pavelpaulau Pavel Paulau made changes -
          Fix Version/s 2.0.2 [ 10418 ]
          pavelpaulau Pavel Paulau made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          pavelpaulau Pavel Paulau made changes -
          Rank Ranked higher
          pavelpaulau Pavel Paulau made changes -
          Rank Ranked higher
          pavelpaulau Pavel Paulau made changes -
          Sprint Sprint 1 [ 1 ]
          pavelpaulau Pavel Paulau made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          pavelpaulau Pavel Paulau made changes -
          Assignee Pavel Paulau [ pavelpaulau ] Aleksey Kondratenko [ alkondratenko ]
          pavelpaulau Pavel Paulau made changes -
          Planned End (re-schedule end date based on new assignee)
          pavelpaulau Pavel Paulau made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Paulau [ pavelpaulau ]
          pavelpaulau Pavel Paulau made changes -
          Planned End (re-schedule end date based on new assignee)
          pavelpaulau Pavel Paulau made changes -
          Attachment rebalance_progress.png [ 16803 ]
          pavelpaulau Pavel Paulau made changes -
          Rank Ranked higher
          dipti Dipti Borkar made changes -
          Sprint PCI Team - Sprint 1 [ 1 ] PCI Team - Sprint 1, PCI Team - Sprint 2 [ 1, 2 ]
          dipti Dipti Borkar made changes -
          Component/s performance [ 10222 ]
          Component/s ns_server [ 10019 ]
          Component/s view-engine [ 10060 ]
          pavelpaulau Pavel Paulau made changes -
          Link This issue depends on CBD-899 [ CBD-899 ]
          pavelpaulau Pavel Paulau made changes -
          Assignee Pavel Paulau [ pavelpaulau ] Aleksey Kondratenko [ alkondratenko ]
          pavelpaulau Pavel Paulau made changes -
          Planned End (re-schedule end date based on new assignee)
          pavelpaulau Pavel Paulau made changes -
          Sprint PCI Team - Sprint 1, PCI Team - Sprint 2 [ 1, 2 ] PCI Team - Sprint 1 [ 1 ]
          steve Steve Yen made changes -
          Rank Ranked lower
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s 2.0.2 [ 10418 ]
          anil Anil Kumar made changes -
          Sprint PCI Team - Sprint 1 [ 1 ] PCI Team - Sprint 1, PCI Team - Sprint 4 [ 1, 7 ]
          anil Anil Kumar made changes -
          Rank Ranked higher
          anil Anil Kumar made changes -
          Rank Ranked higher
          anil Anil Kumar made changes -
          Rank Ranked higher
          anil Anil Kumar made changes -
          Rank Ranked lower
          anil Anil Kumar made changes -
          Sprint PCI Team - Sprint 1, PCI Team - Sprint 4 [ 1, 7 ] PCI Team - Sprint 1 [ 1 ]
          pavelpaulau Pavel Paulau made changes -
          Sprint PCI Team - Sprint 1 [ 1 ] PCI Team - Sprint 1, PCI Team - Sprint 5 [ 1, 9 ]
          anil Anil Kumar made changes -
          Rank Ranked lower
          anil Anil Kumar made changes -
          Rank Ranked higher
          pavelpaulau Pavel Paulau made changes -
          Rank Ranked higher
          pavelpaulau Pavel Paulau made changes -
          Sprint PCI Team - Sprint 1, PCI Team - Sprint 5 [ 1, 9 ] PCI Team - Sprint 1, PCI Team - Sprint 5, PCI Team - Sprint 6 [ 1, 9, 11 ]
          anil Anil Kumar made changes -
          Sprint PCI Team - Sprint 1, PCI Team - Sprint 5, PCI Team - Sprint 6 [ 1, 9, 11 ] PCI Team - Sprint 1, PCI Team - Sprint 5 [ 1, 9 ]
          maria Maria McDuff (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Thuan Nguyen [ thuan ]
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s 2.5.0 [ 11200 ]
          Fix Version/s 3.0 [ 10414 ]
          maria Maria McDuff (Inactive) made changes -
          Status In Progress [ 3 ] Closed [ 6 ]
          Resolution Won't Fix [ 2 ]

            People

            • Assignee:
              thuan Thuan Nguyen
              Reporter:
              thuan Thuan Nguyen
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile