Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33192

[System Test] Index rebalance hung since 20+ hrs

    XMLWordPrintable

Details

    Description

      Build : 5.5.4-4324
      Test : -test tests/integration/test_allFeaturesWithGSI_vulcan.yml -scope tests/integration/scope_Xattrs_GSI_Vulcan.yml
      Scale : 3
      Iteration : 3rd

      Rebalance in operation for an indexer node is stuck at 62% for 20+ hrs. This is because building of default_result_ratings_claims_pindex (replica 1) index on 172.23.96.56 is stalled since 2019-02-26T18:45 (no progress at all).

      Examining the indexer logs around this time, there are errors like : 

      indexer.log.5.gz:2019-02-26T18:34:30.828-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.108.103:9999 Topic MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e IndexInst [indexInstance:<instId:2348234235873728
      		 
      17 state:IndexLoading  definition:<defnID:5028851138657533047 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`rating` `result` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn
      		 
      :<nil>> indexInstance:<instId:10502546875077776841 state:IndexActive  definition:<defnID:5028851138657533047 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`rating` `result` `claim`])</ud> partitionSchem
      		 
      e:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]. Err feed.invalidBucket
      		 
      indexer.log.5.gz:2019-02-26T18:37:44.568-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.108.103:9999 Topic MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e IndexInst [indexInstance:<instId:2348234235873728
      		 
      17 state:IndexLoading  definition:<defnID:5028851138657533047 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`rating` `result` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn
      		 
      :<nil>> indexInstance:<instId:10502546875077776841 state:IndexActive  definition:<defnID:5028851138657533047 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`rating` `result` `claim`])</ud> partitionSchem
      		 
      e:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]. Err feed.invalidBucket

      The corresponding projector logs around this time from 172.23.108.103:

      2019-02-26T18:37:44.561-08:00 [Info] pram[:9999] Request "/adminport/addInstancesRequest"
      		 
      2019-02-26T18:37:44.562-08:00 [Info] PROJ[:9999] ##1180 doAddInstances() "MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e"
      		 
      2019-02-26T18:37:44.564-08:00 [Info] FEED[<=>MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e(127.0.0.1:8091)] ##1180 endpoint "172.23.96.56:9105" active ...
      		 
      2019-02-26T18:37:44.564-08:00 [Info] FEED[<=>MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e(127.0.0.1:8091)] ##1180 endpoint "172.23.96.56:9105" active ...
      		 
      2019-02-26T18:37:44.564-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e(127.0.0.1:8091)] ##1180 addInstances() invalid-bucket "default"
      		 
      2019-02-26T18:37:44.564-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_a6a6546ea084f355e4abe5d0cbedd64e(127.0.0.1:8091)] ##1180 addInstances() invalid-bucket "default"
      		 
      2019-02-26T18:37:44.564-08:00 [Info] PROJ[:9999] ##1180 doAddInstances() returns ...

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            deepkaran.salooja Deepkaran Salooja
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty