Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11768

movement of 27 empty replica vbuckets gets stuck in seqnoWaiting

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • Build 3.0.0-988

      Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630 (24 vCPU)
      Memory = 64 GB
      Disk = RAID 10 HDD

    Description

      Rebalance of 10 empty buckets never finishes.

      Rebalance of first bucket ("bucket-10") completed successfully.

      Movement of 27 vbuckets in bucket "bucket-9" started but obviously makes no progress.

      451, 455, 459, 463, 467, 471, 475, 479, 483, 487, 491, 495, 499, 503, 507, 511, 811, 815, 819, 823, 827, 831, 835, 839, 843, 847, 851

      According to master events the last step in all cases is seqnoWaitingStarted:

      {u'node': u'ns_1@172.23.96.13', u'bucket': u'bucket-9', u'pid': u'<0.19962.2>', u'ts': 1405841984.068374, u'chainBefore': [u'172.23.96.13:11209', u'172.23.96.11:11209'], u'chainAfter': [u'172.23.96.14:11209', u'172.23.96.11:11209'], u'vbucket': 851, u'type': u'vbucketMoveStart'} {u'bucket': u'bucket-9', u'state': u'replica', u'ts': 1405841984.069871, u'host': u'172.23.96.11:11209', u'vbucket': 851, u'type': u'vbucketStateChange'} {u'bucket': u'bucket-9', u'state': u'replica', u'ts': 1405841984.070029, u'host': u'172.23.96.14:11209', u'vbucket': 851, u'type': u'vbucketStateChange'} {u'node': u'172.23.96.14:11209', u'bucket': u'bucket-9', u'vbucket': 851, u'type': u'indexingInitiated', u'ts': 1405841984.073599} {u'bucket': u'bucket-9', u'vbucket': 851, u'type': u'backfillPhaseEnded', u'ts': 1405841984.074577} {u'node': u'172.23.96.14:11209', u'seqno': 0, u'bucket': u'bucket-9', u'ts': 1405841984.075081, u'vbucket': 851, u'type': u'seqnoWaitingStarted'} {u'node': u'172.23.96.11:11209', u'seqno': 0, u'bucket': u'bucket-9', u'ts': 1405841984.075081, u'vbucket': 851, u'type': u'seqnoWaitingStarted'}

      Not surprisingly 1051 replica vubckets are reported.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sriram Sriram Ganesan (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty