Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29027

System test: NC node is down and cluster is in unusable state. 503 service unavailable.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • analytics
    • Enterprise Edition 5.5.0 build 2314

    Description

      • 4 nodes in cluster: 2 KV, 2CBAS
      • 4 buckets in CB and 4 corresponding buckets in CBAS shadowing 4 datasets.
      • Secondary indexes present on 2 datasets.

        create index idx_result1 on ds1(result:string);
        create index idx_result2 on ds2(result:string);
        

      • 12 queries/sec running on all 4 datasets makes it 48 queries/sec.
        2 query types are such that they use secondary indexes while other 2 use primary index.
        select COUNT(result) from ds1 where result = "SUCCESS" limit 50
        select claim from ds2 where result != "SUCCESS" limit 50
        select COUNT(result) from ds3 where result = "SUCCESS" limit 50
        select * from ds4 where result != "SUCCESS" limit 50
      • On CC:
        Failed to get bucket insights.
        Analytics service returned status: 503
        Try refreshing the bucket insights.

      On workbench(CC):

      [
        {
          "code": 1,
          "msg": "Exception: Cannot execute request, cluster is UNUSABLE",
          "query_from_user": "select count(*) from ds3"
        }
      ]
      

      Cluster state:

      {
      "ccNodeId": "63eb1ffeea7ba0677822f667345cdafd",
      "nodes": [
      {
      "configUri": "http://172.23.108.232:8095/analytics/node/config",
      "diagnosticsUri": "http://172.23.108.232:8095/analytics/node/diagnostics",
      "nodeId": "63eb1ffeea7ba0677822f667345cdafd",
      "nodeName": "172.23.108.232:8091"
      }
      ],
      "partitions": [
      {
      "active": true,
      "activeNodeId": "63eb1ffeea7ba0677822f667345cdafd",
      "iodeviceNum": 0,
      "nodeId": "63eb1ffeea7ba0677822f667345cdafd",
      "partitionId": 0,
      "pendingActivation": false
      },
      {
      "active": false,
      "activeNodeId": "b69d22b93b2b07b0e0f6fea98d1456b8",
      "iodeviceNum": 0,
      "nodeId": "b69d22b93b2b07b0e0f6fea98d1456b8",
      "partitionId": 1,
      "pendingActivation": false
      }
      ],
      "restartClusterUri": "http://172.23.108.232:8095/analytics/cluster/restart",
      "state": "UNUSABLE",
      "versionUri": "http://172.23.108.232:8095/analytics/version"
      }
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty