Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37140

[System Test] : index building for one index is stuck

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-4908
      Test : -test tests/integration/test_allFeatures_madhatter.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Day : 4th

      Seeing that index building for index o1_claims has been stuck for several hours. The logs on the indexer node 172.23.97.242 shows the following errors :

      2019-12-03T21:48:23.670-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.99.20:9999 Topic MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8 IndexInst [indexInstance:<instId:380757036783636610 state:IndexLoading  definition:<defnID:13760507631792423205 bucket:CUSTOMER isPrimary:false name:o1_claims using:0 exprType:N1QL secExpressions:<ud>([`claim`])</ud> partitionScheme:SINGLE whereExpression:<ud>()</ud> > singlePartn:endpoints:"172.23.97.242:9105" > ]. Err feed.invalidBucket
      		 
      2019-12-03T21:48:23.663-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.119:9999
      		 
      2019-12-03T21:48:23.669-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.239:9999
      		 
      2019-12-03T21:48:23.670-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.99.20:9999
      		 
      2019-12-03T21:48:24.675-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error from Projector ErrPartialVbStart

      The projector logs also shows errors like the following :

      2019-12-03T21:46:39.974-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8(127.0.0.1:8091)] ##86e addInstances() invalid-bucket "CUSTOMER"

      (Logs attached are from an hour back but similar errors can be seen in these logs as well)

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-37474 [System Test] : index building for one index is stuck - o2_result_rating - DISTRICT bucket

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-37140
          # Subject Branch Project Status CR V
          119060,2 MB-37140 skip repair for stream in recovery unstable indexing Status: MERGED +2 +1
          119130,2 MB-37140 [Backport to mad-hatter] skip repair for stream in recovery mad-hatter indexing Status: MERGED +2 +1

          Activity

            People

              girish.benakappa Girish Benakappa
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty