Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7365

Resetting rebalance status since it's not really running when add XDCR replication

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.1
    • Component/s: None
    • Security Level: Public
    • Labels:
      None

      Description

      Build 1974:
      verified bug MB-7263 Service memcached constantly exited on dest master node after certain steps in XDCR + rebalance scenarious: Port server memcached on node 'ns_1@10.3.121.63' exited with status 71. failed to listen on TCP port 11210: Address already in use - was not reproduced
      but rebalance failed on both cluster when I add replication

      steps: 2 clusters with 5 nodes in each, 10 buckets* 100K
      10.3.121.63* and 10.3.121.11* on
      1. swap rebalance 1 node on each cluster
      2. during rebalance on both clusters create replications

      Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.3.121.64 17:02:45 - Wed Dec 5, 2012
      Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.121.64 17:02:44 - Wed Dec 5, 2012
      Created remote cluster reference "http://10.3.121.112:8091/" via 10.3.121.112. menelaus_web_remote_clusters000 ns_1@10.3.121.63 17:02:42 - Wed Dec 5, 2012
      Server error during processing: ["web request failed",

      {path,"/pools/default/tasks"}, {type,exit},
      {what,
      {timeout, {gen_server,call, [ns_node_disco,nodes_wanted]}}},
      {trace,
      [{gen_server,call,2}, {menelaus_web,handle_tasks,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 17:02:42 - Wed Dec 5, 2012
      Got exit from node disco events subscription ns_vbucket_mover000 ns_1@10.3.121.63 17:02:35 - Wed Dec 5, 2012
      Server error during processing: ["web request failed",{path,"/pools/default/tasks"}

      ,

      {type,exit},
      {what,
      {timeout, {gen_server,call,[ns_doctor,get_nodes]}}},
      {trace,
      [{gen_server,call,2}, {ns_doctor,build_tasks_list,2}, {menelaus_web,handle_tasks,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 17:02:29 - Wed Dec 5, 2012

      Server error during processing: ["web request failed", {path,"/pools/default"},{type,exit}

      ,
      {what,
      {timeout,

      {gen_server,call, [ns_cookie_manager,cookie_get]}

      }},
      {trace,
      [

      {gen_server,call,2}

      ,

      {menelaus_web,build_nodes_info_fun,3}

      ,

      {menelaus_web,build_pool_info,4}

      ,

      {menelaus_web,handle_pool_info,2}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}] menelaus_web019 ns_1@10.3.121.63 17:02:01 - Wed Dec 5, 2012
      Bucket "standard_bucket3" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.3.121.63 16:49:59 - Wed Dec 5, 2012
      Bucket "standard_bucket3" loaded on node 'ns_1@10.3.121.69' in 0 seconds. ns_memcached001 ns_1@10.3.121.69 16:49:58 - Wed Dec 5, 2012
      Shutting down bucket "standard_bucket4" on 'ns_1@10.3.121.68' for deletion ns_memcached002 ns_1@10.3.121.68 16:49:56 - Wed Dec 5, 2012
      Started rebalancing bucket standard_bucket3 ns_rebalancer000 ns_1@10.3.121.63 16:49:56 - Wed Dec 5, 2012
      Bucket "standard_bucket4" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.3.121.63 16:37:59 - Wed Dec 5, 2012
      Bucket "standard_bucket4" loaded on node 'ns_1@10.3.121.69' in 0 seconds. ns_memcached001 ns_1@10.3.121.69 16:37:58 - Wed Dec 5, 2012
      Started rebalancing bucket standard_bucket4 ns_rebalancer000 ns_1@10.3.121.63 16:37:57 - Wed Dec 5, 2012
      Starting rebalance, KeepNodes = ['ns_1@10.3.121.63','ns_1@10.3.121.69',
      'ns_1@10.3.121.67','ns_1@10.3.121.66',
      'ns_1@10.3.121.64'], EjectNodes = ['ns_1@10.3.121.68']
      ns_orchestrator004 ns_1@10.3.121.63 16:37:56 - Wed Dec 5, 2012

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        andreibaranouski Andrei Baranouski added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.112-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.113-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.114-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.115-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.116-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.117-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.63-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.64-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.66-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.67-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.68-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7365/10.3.121.69-8091-diag.txt.gz
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Thanks for putting all logs here, but timeouts here are clear

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Thanks for putting all logs here, but timeouts here are clear

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes