Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19432

[FTS] Rebalance out during indexing with 5 indexes+ replicas is too slow with FDB

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • bug-backlog
    • 4.5.0
    • fts

    Description

      Build
      4.5.0-2417

      Testcase
      ./testrunner -i INI_FILE.ini skip-cleanup=True,get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.moving_topology_fts.MovingTopFTS.rebalance_out_during_index_building,items=30000,cluster=D,F,F,index_replicas=1,standard_buckets=2,sasl_buckets=2,GROUP=P1

      Rebalance is painfully slow and this test always failed. Earlier MB-19295, now with memory optimizations, just slow rebalance. Test times out after no progress in rebalance % for 3 mins.

      2016-04-28 14:44:22 | INFO | MainProcess | test_thread | [fts_base.__async_rebalance_out] Starting rebalance-out nodes:[ip:172.23.106.176 port:8091 ssh_username:root] at C1 cluster 172.23.106.139
      		 
      2016-04-28 14:44:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.176&user=Administrator&knownNodes=ns_1%40172.23.106.176%2Cns_1%40172.23.106.175%2Cns_1%40172.23.106.139
      		 
      2016-04-28 14:44:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
      		 
      2016-04-28 14:44:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
      		 
      2016-04-28 14:44:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 16.67 %
      		 
      2016-04-28 14:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 16.67 %
      		 
      2016-04-28 14:44:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 17.65 %
      		 
      2016-04-28 14:45:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:45:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:45:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:45:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:45:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:45:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:46:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:47:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:48:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %
      		 
      2016-04-28 14:48:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 19.61 %

      Test log - https://gist.github.com/arunapiravi/530b26e69fbc4ea53453f53eeddb0b5f

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty