Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6273

Can't rebalance any server, nor add, remove, failover.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Incomplete
    • Affects Version/s: 1.8.1
    • Fix Version/s: None
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Windows Server 2008 R2

      Description

      Hello,

      We can't add or remove any server. Also we can't failover them.

      All the time we got this error:

      Server error during processing: ["web request failed",

      {path,"/controller/rebalance"}

      ,

      {type,exit}

      ,
      {what,
      {noproc,
      {gen_fsm,sync_send_event,
      [

      {global,ns_orchestrator}

      ,

      {start_rebalance, ['ns_1@10.176.40.107', 'ns_1@10.176.42.190','ns_1@10.179.5.6', 'ns_1@10.179.91.14', 'ns_1@10.180.235.71', 'ns_1@10.182.96.255', 'ns_1@10.182.97.253', 'ns_1@10.182.99.53', 'ns_1@10.182.99.220'], [], ['ns_1@10.182.96.246']}

      ]}}},
      {trace,
      [

      {gen_fsm,sync_send_event,2}

      ,

      {menelaus_web,do_handle_rebalance,3}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}]

      It is very (or the same) that this ticket: MB-4214, but we are on 1.8.1.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        Sergius14 Sergius14 added a comment -

        We are running a 10 servers cluster. All of them are Windows Server 2008 R2 (x64).

        Show
        Sergius14 Sergius14 added a comment - We are running a 10 servers cluster. All of them are Windows Server 2008 R2 (x64).
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        This happens every time you click rebalance?

        If some I'll need some more logs. May I ask you to upload diags from ns_1@10.176.40.107 ? Go to Log and click Generate diagnostics link. Compress and upload here please. (NOTE: it may contain some of your passwords)

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - This happens every time you click rebalance? If some I'll need some more logs. May I ask you to upload diags from ns_1@10.176.40.107 ? Go to Log and click Generate diagnostics link. Compress and upload here please. (NOTE: it may contain some of your passwords)
        Hide
        Sergius14 Sergius14 added a comment -

        I've attached the asked log.

        And yes, every single time.

        From CLI or the WEB, same error.

        At this point we are thinking that we are going to do a restart of all Couchbase services.

        Show
        Sergius14 Sergius14 added a comment - I've attached the asked log. And yes, every single time. From CLI or the WEB, same error. At this point we are thinking that we are going to do a restart of all Couchbase services.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Sorry for losing this ticket.

        I'd need logs from other nodes. Particularly it's now clear that previous master node was ns_1@10.182.96.246.

        Given you're running 10 nodes of enterprise edition I assume you have paid support option. I recommend you to escalate via that path.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Sorry for losing this ticket. I'd need logs from other nodes. Particularly it's now clear that previous master node was ns_1@10.182.96.246. Given you're running 10 nodes of enterprise edition I assume you have paid support option. I recommend you to escalate via that path.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        See above

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - See above

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            Sergius14 Sergius14
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes