Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19794

Indexer in recovery

    XMLWordPrintable

Details

    Description

      Longevity test tried to recreate an index but getting error:
      "code": 5000,
      "msg": "GSI CreateIndex() - cause: Indexer In Recovery",
      "query_from_user": "create index default_claim on `default`(claim);"

      Buckets may have been deleted and recreated as in CBSE-2008, Arunkumar Senthilnathan can verify. (we needed to rehab the cluster from different issue, hence the index recreate)

      Seeing these types of errors/warnings in indexer log. 

      16-05-31T08:19:07.128-07:00 [Error] SCAN##3992344 RESPONSE rows:4322, waitTime:9.413µs, totalTime:31.463883ms, status:(error = Client requested cancel), requestId:a76c4c99-9b85-42cb-8f17-0c4c49eef7f8
      		 
      2016-05-31T08:19:07.385-07:00 [Info] Indexer::ReadMemstats Time Taken 71.918114ms
      		 
      2016-05-31T08:19:07.551-07:00 [Error] SCAN##3992516 RESPONSE rows:3459, waitTime:5.572µs, totalTime:26.184419ms, status:(error = Client requested cancel), requestId:78371e6f-d248-41ff-8605-84ba2a40c783
      		 
      2016-05-31T08:19:07.656-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 297000 Milliseconds Bucket default Vbucket 714
      		 
      2016-05-31T08:19:10.656-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 300000 Milliseconds Bucket default Vbucket 714
      		 
      2016-05-31T08:19:10.686-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Bucket default Vbucket 714
      		 
      2016-05-31T08:19:10.960-07:00 [Error] SCAN##3993953 RESPONSE rows:3028, waitTime:6.647µs, totalTime:21.687427ms, status:(error = Client requested cancel), requestId:7429c6cf-ea78-4959-be70-5b465dd97ea0

      collecting full cbinfo

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. MB-19685 4.5.0 GA Release

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-19794
          # Subject Branch Project Status CR V
          64612,2 MB-19794 Handle flush abort for prepareRecovery state unstable indexing Status: MERGED +2 +1
          64625,1 MB-19794 Handle flush abort for prepareRecovery state watson indexing Status: MERGED +2 +1

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty