Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-16113

[system tests windows] Rebalance exited with reason {unexpected_exit, {'EXIT',, bulk_set_vbucket_state_failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.1.0
    • 4.0.0
    • XDCR
    • Security Level: Public
    • 4.0.0-4047
    • Untriaged
    • Unknown

    Description

      rebelance in 3+2 nodes on src cluster

      Service 'goxdcr' exited with status 1. Restarting. Messages: github.com/couchbase/goxdcr/simple_utils.funcĀ·001(0xc082be8600)
      C:/Jenkins/workspace/sherlock-windows/couchbase/goproj/src/github.com/couchbase/goxdcr/simple_utils/simple_utils.go:38 +0x60
      created by github.com/couchbase/goxdcr/simple_utils.ExecWithTimeout
      C:/Jenkins/workspace/sherlock-windows/couchbase/goproj/src/github.com/couchbase/goxdcr/simple_utils/simple_utils.go:40 +0x189
      [goport] 2015/08/19 14:49:16 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 2 ns_log000 ns_1@172.23.105.39 14:49:16 - Wed Aug 19, 2015
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.16496.64>,
      {bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.39',
      {'EXIT',
      {{timeout,
      {gen_server,call,
      [<19384.29431.25>,

      {register,#Ref<19384.0.30.197786>, <19384.2738.27>}]}},
      {gen_server,call,
      [{'janitor_agent-RevAB', 'ns_1@172.23.105.39'},
      {if_rebalance,<0.21498.62>,
      {update_vbucket_state,517,replica,
      undefined,undefined}},
      infinity]}}}}]}}}
      ns_orchestrator002 ns_1@172.23.105.192 14:49:15 - Wed Aug 19, 2015
      <0.15455.64> exited with {unexpected_exit,
      {'EXIT',<0.16496.64>,
      {bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.39',
      {'EXIT',
      {{timeout,
      {gen_server,call,
      [<19384.29431.25>,{register,#Ref<19384.0.30.197786>, <19384.2738.27>}

      ]}},
      {gen_server,call,
      [

      {'janitor_agent-RevAB','ns_1@172.23.105.39'}

      ,
      {if_rebalance,<0.21498.62>,
      {update_vbucket_state,517,replica,
      undefined,undefined}},
      infinity]}}}}]}}} ns_vbucket_mover000 ns_1@172.23.105.192 14:49:15 - Wed Aug 19, 2015
      Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@172.23.105.39 14:49:04 - Wed Aug 19, 2015
      IP address seems to have changed. Unable to listen on 'ns_1@172.23.106.57'. menelaus_web_alerts_srv000 ns_1@172.23.106.57 14:48:53 - Wed Aug 19, 2015
      IP address seems to have changed. Unable to listen on 'ns_1@172.23.106.142'. menelaus_web_alerts_srv000 ns_1@172.23.106.142 14:40:01 - Wed Aug 19, 2015
      Bucket "RevAB" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.192 14:17:26 - Wed Aug 19, 2015
      Bucket "RevAB" loaded on node 'ns_1@172.23.105.39' in 0 seconds. ns_memcached000 ns_1@172.23.105.39 14:17:25 - Wed Aug 19, 2015
      Bucket "RevAB" loaded on node 'ns_1@172.23.106.142' in 0 seconds. ns_memcached000 ns_1@172.23.106.142 14:17:25 - Wed Aug 19, 2015
      Started rebalancing bucket RevAB ns_rebalancer000 ns_1@172.23.105.192 14:17:24 - Wed Aug 19, 2015
      Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.192 14:15:35 - Wed Aug 19, 2015
      Bucket "UserInfo" loaded on node 'ns_1@172.23.105.39' in 0 seconds. ns_memcached000 ns_1@172.23.105.39 14:15:34 - Wed Aug 19, 2015
      Bucket "UserInfo" loaded on node 'ns_1@172.23.106.142' in 0 seconds. ns_memcached000 ns_1@172.23.106.142 14:15:34 - Wed Aug 19, 2015
      Started rebalancing bucket UserInfo ns_rebalancer000 ns_1@172.23.105.192 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "AbRegNums" ns_storage_conf000 ns_1@172.23.106.142 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "AbRegNums" ns_storage_conf000 ns_1@172.23.105.39 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "MsgsCalls" ns_storage_conf000 ns_1@172.23.105.39 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "RevAB" ns_storage_conf000 ns_1@172.23.105.39 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "UserInfo" ns_storage_conf000 ns_1@172.23.105.39 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "MsgsCalls" ns_storage_conf000 ns_1@172.23.106.142 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "RevAB" ns_storage_conf000 ns_1@172.23.106.142 14:15:34 - Wed Aug 19, 2015
      Deleting old data files of bucket "UserInfo" ns_storage_conf000 ns_1@172.23.106.142 14:15:34 - Wed Aug 19, 2015
      Starting rebalance, KeepNodes = ['ns_1@172.23.105.192','ns_1@172.23.105.204',
      'ns_1@172.23.106.57','ns_1@172.23.105.39',
      'ns_1@172.23.106.142'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator004 ns_1@172.23.105.192 14:15:33 - Wed Aug 19, 2015

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty