Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48459

[Backport MB-48422 to 7.0.2] Greedy planner is not creating index on least loaded node(empty indexer)

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      Greedy planner is not placing equivalent index on an empty indexer node.

      Steps to reproduce:- 

      1. Create 5 indexes with 2 replicas so that all the index nodes have equal no. of replicas
      2. Drop all the replica instance from on index node (10.112.205.102) to make skewed index node.
      3. Add an index (equivalent index) with one replica
      4. one replica would go to skewed indexer node and other to one of the two indexer node.

       

      Index replicas are created on non-skewed indexer node, ignoring empty indexer node.

      least loaded node as per cbindex_plan - 

       

       

      hemantrajput@LFC testrunner % less index_plan.log | grep 'usageRatio\|nodeId'
      "nodeId": "10.112.205.102:8091",
      "usageRatio": 0
      "nodeId": "10.112.205.103:8091",
      "usageRatio": 1.100489084874193
      "nodeId": "10.112.205.101:8091",
      "usageRatio": 0.6995109151258069
      

       

      hemantrajput@LFC testrunner % 

      ['10.112.205.102', '10.112.205.101']

      Indexer metadata

      [{'defnId': 6794570716059611895, 'instId': 9355553456284584391, 'name': 'idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6794570716059611895, 'instId': 13076976066364619977, 'name': 'idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6043748238486343072, 'instId': 181406721272622509, 'name': 'idx_1 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`city`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`city`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 14708979313040993898, 'instId': 785011925906681646, 'name': 'idx_2 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`country`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`country`)', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 7912375296931637555, 'instId': 17752739916677953116, 'name': 'idx_3', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`title`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`title`)', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 14212517112665457718, 'name': 'idx_4', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1553416589640592643, 'instId': 15515767151450973914, 'name': 'idx_4 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`firstName`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'idx_4', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 15403016386308928468, 'name': 'new_idx_0', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.103:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.103:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4023184642012500649, 'instId': 7682157202707397131, 'name': 'new_idx_0 (replica 1)', 'bucket': 'test_bucket', 'scope': 'test_scope_1', 'collection': 'test_collection_1', 'secExprs': ['`age`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `new_idx_0` ON `test_bucket`.`test_scope_1`.`test_collection_1`(`age`) WITH {  "nodes":[ "10.112.205.101:8091","10.112.205.103:8091" ], "num_replica":1 }', 'hosts': ['10.112.205.101:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'10.112.205.101:8091': [0]}, 'numReplica': 1, 'indexName': 'new_idx_0', 'replicaId': 1, 'stale': False, 'lastScanTime': 'NA'}]

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            amit.kulkarni Amit Kulkarni added a comment - - edited

            Hi Jeelan Poola, Mihir Kamdar, Hemant Rajput, I think this needs to be back ported to 7.0.2, as it can lead to uneven load distribution in the cluster in presence of equivalent indexes.

            Note: on greedy planner failure, in-product work around uses the round robin approach which ensures replica and server group HA but does not ensure load distribution.

            amit.kulkarni Amit Kulkarni added a comment - - edited Hi Jeelan Poola , Mihir Kamdar , Hemant Rajput , I think this needs to be back ported to 7.0.2, as it can lead to uneven load distribution in the cluster in presence of equivalent indexes. Note: on greedy planner failure, in-product work around uses the round robin approach which ensures replica and server group HA but does not ensure load distribution.

            I am fine taking this in 7.0.2

            mihir.kamdar Mihir Kamdar added a comment - I am fine taking this in 7.0.2
            jeelan.poola Jeelan Poola added a comment -

            Amit Kulkarni Added to master ticket.

            jeelan.poola Jeelan Poola added a comment - Amit Kulkarni Added to master ticket.

            Build couchbase-server-7.0.2-6677 contains indexing commit cca5f1b with commit message:
            MB-48459: Fix book-keeping for equivalent indexes in greedy planner

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6677 contains indexing commit cca5f1b with commit message: MB-48459 : Fix book-keeping for equivalent indexes in greedy planner

            Validated on 7.0.2-6683

            hemant.rajput Hemant Rajput added a comment - Validated on 7.0.2-6683

            People

              hemant.rajput Hemant Rajput
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty