Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8127

cluster is broken after Rebalance exited with reason {important_nodes_went_down,

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      http://qa.hq.northscale.net/job/ubuntu-64-2.0-upgrade/108/consoleFull

      Node 'ns_1@10.3.3.24' saw that node 'ns_1@10.3.3.26' came up. Tags: [] (repeated 1 times) ns_node_disco004 ns_1@10.3.3.24 00:15:11 - Thu Apr 18, 2013
      Server error during processing: ["web request failed",

      {path,"/pools/default/saslBucketsStreaming"}

      ,

      {type,error}

      ,

      {what,badarg}

      ,
      {trace,
      [

      {erlang,integer_to_list,[undefined]}

      ,

      {ns_bucket, '-json_map_from_config/2-fun-0-',3}

      ,

      {lists,map,2},{lists,map,2}

      ,

      {ns_bucket,json_map_from_config,2}

      ,

      {menelaus_web_buckets, '-handle_sasl_buckets_streaming/2-fun-1-', 3}

      ,

      {lists,map,2}

      ,

      {menelaus_web_buckets, '-handle_sasl_buckets_streaming/2-fun-2-', 2}

      ]}] (repeated 12 times) menelaus_web019 ns_1@10.3.3.19 00:15:09 - Thu Apr 18, 2013
      Rebalance exited with reason {important_nodes_went_down,
      {ns_node_disco_events,
      ['ns_1@10.3.3.19','ns_1@10.3.3.24',
      'ns_1@10.3.3.26','ns_1@10.3.3.27'],
      ['ns_1@10.3.3.19','ns_1@10.3.3.24',
      'ns_1@10.3.3.27']}}
      ns_orchestrator002 ns_1@10.3.3.24 00:14:51 - Thu Apr 18, 2013

      for this server is impossible to get rebalance progress
      see related issue MB-8126

      [2013-04-18 19:14:30,550] - [rest_client:608] ERROR - socket error while connecting to http://10.3.3.24:8091/pools/default/rebalanceProgress error timed out

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        andreibaranouski Andrei Baranouski added a comment - https://s3.amazonaws.com/bugdb/jira/MB-8127/50c666b3/10.3.3.19-4192013-439-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-8127/50c666b3/10.3.3.24-4192013-440-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-8127/50c666b3/10.3.3.26-4192013-441-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-8127/50c666b3/10.3.3.27-4192013-443-diag.zip
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        test online_consequentially_upgrade 2.0.0-1976-rel->2.0.2-766-rel

        Show
        andreibaranouski Andrei Baranouski added a comment - test online_consequentially_upgrade 2.0.0-1976-rel->2.0.2-766-rel
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Very possibly something serious. But I'd like to try to complete rebalance progress work today.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Very possibly something serious. But I'd like to try to complete rebalance progress work today.
        Hide
        Aliaksey Artamonau Aliaksey Artamonau added a comment -

        I could not figure out anything from the logs. Is it possible to leave the cluster in this state for online debugging?

        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - I could not figure out anything from the logs. Is it possible to leave the cluster in this state for online debugging?
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        blocker is MB-8163 to provide info/reproduce

        Show
        andreibaranouski Andrei Baranouski added a comment - blocker is MB-8163 to provide info/reproduce
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Again I don't get from your comment how mixed cluster rebalance bug may affect this. Please, elaborate.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Again I don't get from your comment how mixed cluster rebalance bug may affect this. Please, elaborate.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Let's consider duplicate of MB-8126 that's too similar to this one

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Let's consider duplicate of MB-8126 that's too similar to this one
        Hide
        maria Maria McDuff (Inactive) added a comment -
        Show
        maria Maria McDuff (Inactive) added a comment - MB-8126 .

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes