Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43908

Rest call to getIndexStatus doesn't return results for all the indexes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Not a Bug
    • Affects Version/s: Cheshire-Cat
    • Fix Version/s: 7.0.0
    • Component/s: qe, test-execution
    • Labels:
    • Environment:
      CBS-7.0.0-4291
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Rest call to getIndexStatus doesn't return information about all the available indexes even after waiting for over a min after index creation. However, All index information is available in query to system:indexes 

       

      Results form getIndexStatus:

      sleep for 60 secs. Wait till indexes available in stats ...
      2021-01-27 00:31:03 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_duplicate_scheduled_indexes] getIndexStatus result: {'code': 'success', 'status': [{'defnId': 16785926088077324474, 'instId': 18034077185607946155, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Error', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'error': 'Index idx_0 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.', 'completion': 0, 'progress': 0, 'scheduled': True, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1601970787610290293, 'instId': 15997506821159671007, 'name': 'idx_3', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}]}

      Results from query to system:indexes

      2021-01-27 00:31:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select * from system:indexes
      2021-01-27 00:31:03 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+%2A+from+system%3Aindexes
      2021-01-27 00:31:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 11.921209ms
      2021-01-27 00:31:03 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_duplicate_scheduled_indexes] Query result: [{'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'a966cf5830e50217', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_6', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '851aa6a4cc2a72cd', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_8', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '5ea9a9b5563c9af4', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_4', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'a97af5c8587e34c1', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_1', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '4aede527c6ac020d', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_9', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'faacc7914f8a03f3', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_7', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '163b57f198106875', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_3', 'namespace_id': 'default', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'e8f396d769cfb0ba', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'message': 'Index idx_0 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.\n', 'name': 'idx_0', 'namespace_id': 'default', 'state': 'pending', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'f1a151e82a965805', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_5', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '9e9cdc7a7fbc561b', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_2', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}]

       

       

       

       

       

       

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          dfinlay Dave Finlay added a comment -

          Abhi - thanks for investigating. Hemant - looks like the config files are getting deleted somehow. We've seen issues like this in the past and it's unfortunate when it happens as it costs a bunch of time for folks to file, investigate and triage tickets. I'm going to set this one as a qe issue as it seems there must be something up with the environment.

          Show
          dfinlay Dave Finlay added a comment - Abhi - thanks for investigating. Hemant - looks like the config files are getting deleted somehow. We've seen issues like this in the past and it's unfortunate when it happens as it costs a bunch of time for folks to file, investigate and triage tickets. I'm going to set this one as a qe issue as it seems there must be something up with the environment.
          Hide
          raju Raju Suravarjjala added a comment -

          Hemant Rajput Any update on this?

          Show
          raju Raju Suravarjjala added a comment - Hemant Rajput Any update on this?
          Hide
          hemant.rajput Hemant Rajput added a comment -

          Raju Suravarjjala, I tried identifying the issue in an isolated and fresh environment, but the issue can't be produced that way. Issue only happens when the tests are run in suite (one after another). I can see the issue consistently on Jenkins run. 

          >> What exactly do the tests do in-between restart of ns_server?

          Test doesn't do any Index related operation during any of server restart. I'll work with Amit Kulkarni on identifying the issue.

          Show
          hemant.rajput Hemant Rajput added a comment - Raju Suravarjjala , I tried identifying the issue in an isolated and fresh environment, but the issue can't be produced that way. Issue only happens when the tests are run in suite (one after another). I can see the issue consistently on Jenkins run.  >> What exactly do the tests do in-between restart of ns_server? Test doesn't do any Index related operation during any of server restart. I'll work with Amit Kulkarni  on identifying the issue.
          Hide
          mihir.kamdar Mihir Kamdar added a comment -

          The issue is with tests that do reset_services=True and the tests wipe off the data and config directories. This leads to some inconsistencies in the configuration after the server is brought up. After modifying the tests to not do reset_services=True, the issue does not show.

          QE will investigate the inconsistencies created by resetting services.

          Show
          mihir.kamdar Mihir Kamdar added a comment - The issue is with tests that do reset_services=True and the tests wipe off the data and config directories. This leads to some inconsistencies in the configuration after the server is brought up. After modifying the tests to not do reset_services=True, the issue does not show. QE will investigate the inconsistencies created by resetting services.
          Hide
          mihir.kamdar Mihir Kamdar added a comment -

          Bulk closing non-fixed defects. Feel free to reopen if necessary.

          Show
          mihir.kamdar Mihir Kamdar added a comment - Bulk closing non-fixed defects. Feel free to reopen if necessary.

            People

            Assignee:
            hemant.rajput Hemant Rajput
            Reporter:
            hemant.rajput Hemant Rajput
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There is 1 open Gerrit change

                  PagerDuty