Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
6.5.0
-
Triaged
-
Unknown
Description
Build 6.5.0-2802
While doing high bucket density tests with 30 buckets we observed message "Service ‘goxdcr’ exited with status 137" in logs when failed over a data node and rebalance started without adding that node back in cluster.
Here is message in log-
Service 'goxdcr' exited with status 137. Restarting. Messages:
|
2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) pipelineMgr.pipeline_factory.NewPipeline(5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets., current status=name={5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12}, status={Pending}, errors={[{"time":"2019-04-10T08:17:43.218085539-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:32.002366535-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:21.896272358-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:11.793103972-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:01.671011744-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:51.357903537-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:41.166658167-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:30.95037004-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:25.444335775-07:00","errMsg":"dcp_5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12_172.23.97.13:11210_1 : DCP upr feed has been closed."}]}, oldProgress={Start pipeline construction}, progress={Start pipeline construction}
|
|
2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Replication 5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12's status experienced changes or errors (<nil>). However, last update resulted in failure, so will reschedule a future update
|
2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled to update in 10s
|
Logs-
Failed over node is 172.23.97.13
https://s3.amazonaws.com/bugdb/jira/hbd_3/collectinfo-2019-04-11T041252-ns_1%40172.23.97.12.zip
https://s3.amazonaws.com/bugdb/jira/hbd_3/collectinfo-2019-04-11T041252-ns_1%40172.23.97.14.zip
Attachments
Issue Links
- duplicates
-
MB-31855 [System test] Rebalancing a node out or failing over a node in longevity cluster causes service crashes
- Closed