Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19145

indexer received a snapshot with end seq 0xFFFFFFFFFFFFFFFF

    XMLWordPrintable

Details

    Description

      4.1.1-5911 Seeing "Forcing Alloc. Vbucket 839" for +36hrs now.

      So I stopped all data loading and querying to verify items are eventually processed, but looks like the indexer stat for Total Items remaining is not dropping. (see charts) 

      indexer.log.5.gz:2016-*04-09T22*:22:57.175-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Vbucket 839
      		 
      indexer.log.5.gz:2016-04-09T22:50:04.63-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Vbucket 839
      		 
      indexer.log.5.gz:2016-04-09T23:16:29.357-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Vbucket 839
      		 
      ....
      		 
      indexer.log:2016-04-*11T06*:22:03.241-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Vbucket 839
      		 
      indexer.log:2016-04-11T06:35:26.11-07:00 [Warn] Indexer::MutationQueue Max Wait Period for Node Alloc Expired 300030. Forcing Alloc. Vbucket 839


      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137275,"default:num_mutations_queued":30293931
      		 
      default:mutation_queue_size":137276,"default:num_mutations_queued":30293932
      		 
      default:mutation_queue_size":137276,"default:num_mutations_queued":30293932

      This is related to MB-18912 where Force alloc is added to help avoid hanging. Where,
      > "The proper fix has been done in watson."
      Deep, I believe we need same fix here - please confirm.

      Attachments

        1. indexer_pprof.rtf
          34 kB
          Tommie McAfee
        2. patch-sherlock-log-snapshot.diff
          0.7 kB
          Pratap Chakravarthy
        3. Screen Shot 2016-04-11 at 10.01.07 AM.png
          138 kB
          Tommie McAfee
        4. Screen Shot 2016-04-11 at 10.01.30 AM.png
          131 kB
          Tommie McAfee

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-19678 CLONE [3.1.6 and 4.1.2]: indexer received a snapshot with end seq 0xFFFFFFFFFFFFFFFF

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-19145
          # Subject Branch Project Status CR V
          62781,2 MB-19145 Add more logging to diagnose flusher hang sherlock indexing Status: ABANDONED +2 +1
          62837,2 MB-19145 toy manifest for adding debug logs master manifest Status: MERGED +2 +1
          62839,2 MB-19145 rename toy manifest master manifest Status: MERGED +2 +1
          62840,2 MB-19145 add remote to toy-deep-411.xml master manifest Status: MERGED +2 +1
          63037,5 MB-19145: add trap code to check for open snapshots. unstable indexing Status: MERGED +2 +1
          63155,1 MB-19145: Add debugging messages for tracking snapshot master indexing Status: ABANDONED 0 0
          63156,2 MB-19145: Add debugging messages for tracking snapshot usage unstable indexing Status: MERGED +2 +1
          63157,2 MB-19145: Ensure serialization when broadcasting timestamp unstable indexing Status: MERGED +2 +1
          63160,2 MB-19145: Ensure flushTS is monotonically increasing unstable indexing Status: MERGED +2 +1
          63254,4 MB-19145 Add more logging to diagnose flusher hang unstable indexing Status: MERGED +2 +1
          63287,1 MB-19145: Reduce noise in debug log for tracking TS seqno unstable indexing Status: MERGED +2 +1
          63313,2 MB-19145 Disable flush monitor by default unstable indexing Status: MERGED +2 +1
          63321,1 MB-19145: Add debug log for out-of-bound seqno unstable indexing Status: MERGED +2 +1
          63610,3 MB-19145: Check StreamBegin before updating snapshot filter on arriving snapshot markter unstable indexing Status: MERGED +2 +1
          63683,1 MB-19145 Log projector addr with rollbackTs unstable indexing Status: MERGED +2 +1
          64095,4 MB-19145: Merge backfill and in-memory snapshots correctly on replica vb watson ep-engine Status: MERGED +2 +1
          64332,1 Merge remote-tracking branch 'couchbase/watson' into master master ep-engine Status: MERGED +2 +1

          Activity

            People

              tommie Tommie McAfee (Inactive)
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty