Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37140

[System Test] : index building for one index is stuck

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-4908
      Test : -test tests/integration/test_allFeatures_madhatter.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Day : 4th

      Seeing that index building for index o1_claims has been stuck for several hours. The logs on the indexer node 172.23.97.242 shows the following errors :

      2019-12-03T21:48:23.670-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.99.20:9999 Topic MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8 IndexInst [indexInstance:<instId:380757036783636610 state:IndexLoading  definition:<defnID:13760507631792423205 bucket:CUSTOMER isPrimary:false name:o1_claims using:0 exprType:N1QL secExpressions:<ud>([`claim`])</ud> partitionScheme:SINGLE whereExpression:<ud>()</ud> > singlePartn:endpoints:"172.23.97.242:9105" > ]. Err feed.invalidBucket
      2019-12-03T21:48:23.663-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.119:9999
      2019-12-03T21:48:23.669-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.97.239:9999
      2019-12-03T21:48:23.670-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error Received feed.invalidBucket from 172.23.99.20:9999
      2019-12-03T21:48:24.675-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM CUSTOMER Error from Projector ErrPartialVbStart
      

      The projector logs also shows errors like the following :

      2019-12-03T21:46:39.974-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_651fec2aa25727547fbc0302d37f47d8(127.0.0.1:8091)] ##86e addInstances() invalid-bucket "CUSTOMER"
      

      (Logs attached are from an hour back but similar errors can be seen in these logs as well)

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-37140
          # Subject Branch Project Status CR V

          Activity

            http://review.couchbase.org/119060 MB-37140 skip repair for stream in recovery

            deepkaran.salooja Deepkaran Salooja added a comment - http://review.couchbase.org/119060 MB-37140 skip repair for stream in recovery

            Build couchbase-server-7.0.0-1118 contains indexing commit 2d51e4d with commit message:
            MB-37140 skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1118 contains indexing commit 2d51e4d with commit message: MB-37140 skip repair for stream in recovery

            Build couchbase-server-6.5.0-4934 contains indexing commit fa962ce with commit message:
            MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4934 contains indexing commit fa962ce with commit message: MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            Build couchbase-server-6.5.1-6004 contains indexing commit fa962ce with commit message:
            MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6004 contains indexing commit fa962ce with commit message: MB-37140 [Backport to mad-hatter] skip repair for stream in recovery

            Closing this issue as we did not see this is longevity run with 6.5.0-4947

            girish.benakappa Girish Benakappa added a comment - Closing this issue as we did not see this is longevity run with 6.5.0-4947

            People

              girish.benakappa Girish Benakappa
              mihir.kamdar Mihir Kamdar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty