Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48422

Greedy planner is not creating index on least loaded node(empty indexer)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.0.2
    • secondary-index
    • CBS-7.0.2-6653
    • Untriaged
    • 1
    • No

    Description

      Greedy planner is not placing equivalent index on an empty indexer node.

      Steps to reproduce:- 

      1. Create 5 indexes with 2 replicas so that all the index nodes have equal no. of replicas
      2. Drop all the replica instance from on index node (10.112.205.102) to make skewed index node.
      3. Add an index (equivalent index) with one replica
      4. one replica would go to skewed indexer node and other to one of the two indexer node.

       

      Index replicas are created on non-skewed indexer node, ignoring empty indexer node.

      least loaded node as per cbindex_plan - 

       

       

      hemantrajput@LFC testrunner % less index_plan.log | grep 'usageRatio\|nodeId'
      		 
      "nodeId": "10.112.205.102:8091",
      		 
      "usageRatio": 0
      		 
      "nodeId": "10.112.205.103:8091",
      		 
      "usageRatio": 1.100489084874193
      		 
      "nodeId": "10.112.205.101:8091",
      		 
      "usageRatio": 0.6995109151258069

       

      hemantrajput@LFC testrunner % 

      ['10.112.205.102', '10.112.205.101']

      Indexer metadata

      [{'defnId': 6794570716059611895, 'instId': 9355553456284584391, 'name': 'idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6794570716059611895, 'instId': 13076976066364619977, 'name': 'idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6043748238486343072, 'instId': 181406721272622509, 'name': 'idx_1 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`city`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`city`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 14708979313040993898, 'instId': 785011925906681646, 'name': 'idx_2 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`country`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`country`)', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 7912375296931637555, 'instId': 17752739916677953116, 'name': 'idx_3', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`title`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`title`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 14212517112665457718, 'name': 'idx_4', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 15515767151450973914, 'name': 'idx_4 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 15403016386308928468, 'name': 'new_idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 7682157202707397131, 'name': 'new_idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}]

       

      Attachments

        1. index_plan.log
          16 kB
        2. node1-cb660-centos7.vagrants.zip
          10.46 MB
        3. node2-cb660-centos7.vagrants.zip
          2.29 MB
        4. node3-cb660-centos7.vagrants.zip
          2.30 MB
        5. test.log
          115 kB

        Issue Links

          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-48459 [Backport MB-48422 to 7.0.2] Greedy planner is not creating index on least loaded node(empty indexer)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              amit.kulkarni Amit Kulkarni
              hemant.rajput Hemant Rajput
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty