Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6992

rebalance hangs after failing over disconnected node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0
    • 2.0
    • couchbase-bucket, ns_server
    • Security Level: Public
    • build 10.3.3.59

    Description

      1 node went down while loading data on 22 node cluster. (possibly related to xen-hypervisor as it could not ping gateway and network interface needed to be restarted)
      While node was down I tried to fail it over and rebalance.
      However, rebalance never completes and looks like there is no rebalance activity occuring on tap.

      Some activity seen in logs at time of node down:

      10.3.3.59 sees .60 nodedown :

      [user:warn,2012-10-22T11:06:38.896,ns_1@10.3.3.59:ns_node_disco:ns_node_disco:handle_info:168]Node 'ns_1@10.3.3.59' saw that node 'ns_1@10.3.3.60' went down.

      at the same time stamp node .60 shows:

      [ns_server:error,2012-10-22T11:06:00.350,ns_1@10.3.3.60:<0.12281.36>:ns_janitor:cleanup_with_states:84]Bucket "default" not yet ready on ['ns_1@10.3.2.84','ns_1@10.3.2.
      85',
      'ns_1@10.3.2.110','ns_1@10.3.2.111',
      'ns_1@10.3.2.112','ns_1@10.3.2.113',
      'ns_1@10.3.2.114','ns_1@10.3.2.115',
      'ns_1@10.3.3.59','ns_1@10.3.3.62',
      'ns_1@10.3.3.65','ns_1@10.3.3.66',
      'ns_1@10.3.3.69','ns_1@10.3.3.70',
      'ns_1@10.3.121.90','ns_1@10.3.121.91',
      'ns_1@10.3.2.107','ns_1@10.3.2.108',
      'ns_1@10.3.2.109']
      [ns_server:debug,2012-10-22T11:06:07.388,ns_1@10.3.3.60:<0.12508.36>:janitor_agent:new_style_query_vbucket_states_loop:116]Exception from query_vbucket_states of "defau
      lt":'ns_1@10.3.2.85'
      {'EXIT',{{nodedown,'ns_1@10.3.2.85'},
      {gen_server,call,
      [

      {'janitor_agent-default','ns_1@10.3.2.85'}

      ,
      query_vbucket_states,infinity]}}}

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            tommie Tommie McAfee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty