Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50692

Internal error is observed while trying to run CBAS queries when CBAS service is starting

    XMLWordPrintable

Details

    • Bug
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • 7.1.0
    • Morpheus
    • analytics
    • Enterprise Edition 7.1.0 build 2183

    Description

      Test case - 
      test_cbas_node_crash_while_rebalancing_IN_cbas_nodes_does_not_change_actual_number_of_replicas
       
      Steps to reproduce - 

      1. Create 4 node cluster with 2 cbas and 2 KV nodes.
      2. Set cbas replica to 3.
      3. create cbas infra like datasets, dataverses and indexes.
      4. Actual number of replica will be 1, as there are only 2 cbas nodes.
      5. Now rebalance-In 2 more CBAS nodes.
      6. while rebalance is happening, stop couchbase server on one of the existing cbas nodes.
      7. rebalance fails as expected. Verify that no data loss happened on cbas side and the actual replica number is still 1.
      8. start the couchbase server that was stopped in step 6.
      9. Try running following query -

      2022-01-30 23:08:10,437 | infra | ERROR   | MainThread | [Rest_Connection:_http_request:286] POST http://172.23.104.163:8095/analytics/service body: {"pretty": "true", "client_context_id": null, "timeout": "120s", "statement": "set `import-private-functions` `true`;ping()"} headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 503 reason: unknown {
      	"requestID": "0b454580-c2ac-4cf7-9208-0b7eeb22832f",
      	"clientContextID": "null",
      	"errors": [{ 
      		"code": 25000,		"msg": "Internal error"	} 
      	],
      	"status": "failed",
      	"metrics": {
      		"elapsedTime": "400.167051ms",
      		"executionTime": "397.788588ms",
      		"resultCount": 0,
      		"resultSize": 0,
      		"processedObjects": 0,
      		"errorCount": 1
      	}
      } auth: Administrator:password

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          michael.blow Michael Blow added a comment -

          When this query is issued, one or more nodes is in the process of booting but has not yet been failed by the analytics cluster from when it had been running before. While a 503 could arguably be more of an ideal response, I do not think we should change anything on this behavior in the Neo timeframe.

          If you feel strongly that we should be reporting a 503 in this circumstance, please re-open the issue targeted for Morpheus.

          michael.blow Michael Blow added a comment - When this query is issued, one or more nodes is in the process of booting but has not yet been failed by the analytics cluster from when it had been running before. While a 503 could arguably be more of an ideal response, I do not think we should change anything on this behavior in the Neo timeframe. If you feel strongly that we should be reporting a 503 in this circumstance, please re-open the issue targeted for Morpheus.
          umang.agrawal Umang added a comment -

          Michael Blow In my opinion we should be reporting a 503 error with a better error message than "Internal Error" as this will help everyone to better debug the issue. Hence I am reopening it and making Fix version as Morpheus.

          umang.agrawal Umang added a comment - Michael Blow In my opinion we should be reporting a 503 error with a better error message than "Internal Error" as this will help everyone to better debug the issue. Hence I am reopening it and making Fix version as Morpheus.

          People

            umang.agrawal Umang
            umang.agrawal Umang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty