Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
3.0
-
Security Level: Public
-
Build 3.0.0-988
Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630 (24 vCPU)
Memory = 64 GB
Disk = RAID 10 HDD
-
Untriaged
-
Centos 64-bit
-
-
Yes
-
June 30 - July 18
Description
Rebalance of 10 empty buckets never finishes.
Rebalance of first bucket ("bucket-10") completed successfully.
Movement of 27 vbuckets in bucket "bucket-9" started but obviously makes no progress.
451, 455, 459, 463, 467, 471, 475, 479, 483, 487, 491, 495, 499, 503, 507, 511, 811, 815, 819, 823, 827, 831, 835, 839, 843, 847, 851
According to master events the last step in all cases is seqnoWaitingStarted:
{u'node': u'ns_1@172.23.96.13', u'bucket': u'bucket-9', u'pid': u'<0.19962.2>', u'ts': 1405841984.068374, u'chainBefore': [u'172.23.96.13:11209', u'172.23.96.11:11209'], u'chainAfter': [u'172.23.96.14:11209', u'172.23.96.11:11209'], u'vbucket': 851, u'type': u'vbucketMoveStart'} {u'bucket': u'bucket-9', u'state': u'replica', u'ts': 1405841984.069871, u'host': u'172.23.96.11:11209', u'vbucket': 851, u'type': u'vbucketStateChange'} {u'bucket': u'bucket-9', u'state': u'replica', u'ts': 1405841984.070029, u'host': u'172.23.96.14:11209', u'vbucket': 851, u'type': u'vbucketStateChange'} {u'node': u'172.23.96.14:11209', u'bucket': u'bucket-9', u'vbucket': 851, u'type': u'indexingInitiated', u'ts': 1405841984.073599} {u'bucket': u'bucket-9', u'vbucket': 851, u'type': u'backfillPhaseEnded', u'ts': 1405841984.074577} {u'node': u'172.23.96.14:11209', u'seqno': 0, u'bucket': u'bucket-9', u'ts': 1405841984.075081, u'vbucket': 851, u'type': u'seqnoWaitingStarted'} {u'node': u'172.23.96.11:11209', u'seqno': 0, u'bucket': u'bucket-9', u'ts': 1405841984.075081, u'vbucket': 851, u'type': u'seqnoWaitingStarted'}Not surprisingly 1051 replica vubckets are reported.
Attachments
Issue Links
- is duplicated by
-
MB-11796 Rebalance after manual failover hangs (delta recovery)
- Closed