Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48459

[Backport MB-48422 to 7.0.2] Greedy planner is not creating index on least loaded node(empty indexer)

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      Greedy planner is not placing equivalent index on an empty indexer node.

      Steps to reproduce:- 

      1. Create 5 indexes with 2 replicas so that all the index nodes have equal no. of replicas
      2. Drop all the replica instance from on index node (10.112.205.102) to make skewed index node.
      3. Add an index (equivalent index) with one replica
      4. one replica would go to skewed indexer node and other to one of the two indexer node.

       

      Index replicas are created on non-skewed indexer node, ignoring empty indexer node.

      least loaded node as per cbindex_plan - 

       

       

      hemantrajput@LFC testrunner % less index_plan.log | grep 'usageRatio\|nodeId'
      		 
      "nodeId": "10.112.205.102:8091",
      		 
      "usageRatio": 0
      		 
      "nodeId": "10.112.205.103:8091",
      		 
      "usageRatio": 1.100489084874193
      		 
      "nodeId": "10.112.205.101:8091",
      		 
      "usageRatio": 0.6995109151258069

       

      hemantrajput@LFC testrunner % 

      ['10.112.205.102', '10.112.205.101']

      Indexer metadata

      [{'defnId': 6794570716059611895, 'instId': 9355553456284584391, 'name': 'idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6794570716059611895, 'instId': 13076976066364619977, 'name': 'idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6043748238486343072, 'instId': 181406721272622509, 'name': 'idx_1 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`city`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`city`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 14708979313040993898, 'instId': 785011925906681646, 'name': 'idx_2 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`country`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`country`)', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 7912375296931637555, 'instId': 17752739916677953116, 'name': 'idx_3', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`title`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`title`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 14212517112665457718, 'name': 'idx_4', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 15515767151450973914, 'name': 'idx_4 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 15403016386308928468, 'name': 'new_idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 7682157202707397131, 'name': 'new_idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}]

       

      Attachments

        Issue Links

          Clones

          Bug - A problem which impairs or prevents the functions of the product. MB-48422 Greedy planner is not creating index on least loaded node(empty indexer)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-48459
          # Subject Branch Project Status CR V
          161554,2 MB-48459: Fix book-keeping for equivalent indexes in greedy planner cheshire-cat indexing Status: MERGED +2 +1

          Activity

            People

              hemant.rajput Hemant Rajput
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty