Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37140

[System Test] : index building for one index is stuck

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-4908
      Test : -test tests/integration/test_allFeatures_madhatter.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Day : 4th

      Seeing that index building for index o1_claims has been stuck for several hours. The logs on the indexer node 172.23.97.242 shows the following errors :

      2019-12-03T21:48:23.670-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.99.20:9999 Topic MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8 IndexInst [indexInstance:<instId:380757036783636610 state:IndexLoading  definition:<defnID:13760507631792423205 bucket:CUSTOMER isPrimary:false name:o1_claims using:0 exprType:N1QL secExpressions:<ud>([`claim`])</ud> partitionScheme:SINGLE whereExpression:<ud>()</ud> > singlePartn:endpoints:"172.23.97.242:9105" > ]. Err feed.invalidBucket
      2019-12-03T21:48:23.663-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.119:9999
      2019-12-03T21:48:23.669-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.239:9999
      2019-12-03T21:48:23.670-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.99.20:9999
      2019-12-03T21:48:24.675-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error from Projector ErrPartialVbStart
      

      The projector logs also shows errors like the following :

      2019-12-03T21:46:39.974-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8(127.0.0.1:8091)] ##86e addInstances() invalid-bucket "CUSTOMER"
      

      (Logs attached are from an hour back but similar errors can be seen in these logs as well)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            http://review.couchbase.org/119060 MB-37140 skip repair for stream in recovery

            deepkaran.salooja Deepkaran Salooja added a comment - http://review.couchbase.org/119060 MB-37140 skip repair for stream in recovery

            Build couchbase-server-7.0.0-1118 contains indexing commit 2d51e4d with commit message:
            MB-37140 skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1118 contains indexing commit 2d51e4d with commit message: MB-37140 skip repair for stream in recovery

            Build couchbase-server-6.5.0-4934 contains indexing commit fa962ce with commit message:
            MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4934 contains indexing commit fa962ce with commit message: MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            Build couchbase-server-6.5.1-6004 contains indexing commit fa962ce with commit message:
            MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6004 contains indexing commit fa962ce with commit message: MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            Closing this issue as we did not see this is longevity run with 6.5.0-4947

            girish.benakappa Girish Benakappa added a comment - Closing this issue as we did not see this is longevity run with 6.5.0-4947

            People

              girish.benakappa Girish Benakappa
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty