Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33752

Service ‘goxdcr’ exited with status 137 after failover

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 6.5.0
    • XDCR
    • Triaged
    • Unknown

    Description

      Build 6.5.0-2802

      While doing high bucket density tests with 30 buckets we observed message "Service ‘goxdcr’ exited with status 137" in logs when failed over a data node and rebalance started without adding that node back in cluster.

      Here is message in log-

       

      Service 'goxdcr' exited with status 137. Restarting. Messages:
      2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) pipelineMgr.pipeline_factory.NewPipeline(5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets., current status=name={5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12}, status={Pending}, errors={[{"time":"2019-04-10T08:17:43.218085539-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:32.002366535-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:21.896272358-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:11.793103972-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:17:01.671011744-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:51.357903537-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:41.166658167-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:30.95037004-07:00","errMsg":"Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets."},{"time":"2019-04-10T08:16:25.444335775-07:00","errMsg":"dcp_5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12_172.23.97.13:11210_1 : DCP upr feed has been closed."}]}, oldProgress={Start pipeline construction}, progress={Start pipeline construction}
       
      2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Replication 5c36fc1d11a75877c4d9e15d61b490ca/bucket-12/bucket-12's status experienced changes or errors (<nil>). However, last update resulted in failure, so will reschedule a future update
      2019-04-10T08:17:43.218-07:00 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled to update in 10s
      

       

      Logs-

      Failed over node is 172.23.97.13

      https://s3.amazonaws.com/bugdb/jira/hbd_3/collectinfo-2019-04-11T041252-ns_1%40172.23.97.12.zip

      https://s3.amazonaws.com/bugdb/jira/hbd_3/collectinfo-2019-04-11T041252-ns_1%40172.23.97.14.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              yu Yu Sui (Inactive)
              mahesh.mandhare Mahesh Mandhare (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty