Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-9323

[windows} node down caused by excessive memory used in XDCR

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 3.0
    • 2.2.0
    • XDCR
    • Security Level: Public
    • windows physical server 2008 R2 64-bit

    Description

      Setup xdcr testing

      Environment:
      Source cluster:
      4 node windows physical servers. Each server is 4 core cpu, 32 GB RAM, 2 disks
      10.2.1.64
      10.2.1.65
      10.2.1.66
      10.2.1.67

      Target cluster:
      2 node windows physical servers. Each server is 4 core cpu, 32 GB RAM, 2 disks
      10.2.1.62
      10.2.1.63

      Both source and target clusters are installed couchbase server 2.2.0-821 (rc3)
      Manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.2.0-821-rel.setup.exe.manifest.xml

      At source cluster, create 2 buckets, default and sasl with one replica
      Load 50 million items with key size from 128 bytes to 512 bytes to each bucket
      Each bucket has one doc with one view

      At source cluster, failover 2 nodes (66 and 67), add node 66 back
      Rebalance ==> failed due to node down

      Rebalance exited with reason {important_nodes_went_down,
      {ns_node_disco_events,
      ['ns_1@10.2.1.64','ns_1@10.2.1.65',
      'ns_1@10.2.1.66'],
      ['ns_1@10.2.1.65','ns_1@10.2.1.66']}}
      ns_orchestrator002 ns_1@10.2.1.65 17:38:30 - Thu Oct 10, 2013
      Server error during processing: ["web request failed",

      {path, "/pools/default/buckets/default/nodes/10.2.1.64:8091/stats"}

      ,

      {type,exit}

      ,
      {what,
      {{nodedown,'ns_1@10.2.1.64'},
      {gen_server,call,
      [

      {hot_keys_keeper,'ns_1@10.2.1.64'}

      ,

      {get_local_keys,"default"}

      ]}}},
      {trace,
      [

      {gen_server,call,2}

      ,

      {menelaus_stats,handle_bucket_node_stats,4}

      ,

      {request_throttler,do_request,3}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}] menelaus_web019 ns_1@10.2.1.65 17:38:25 - Thu Oct 10, 2013
      Node 'ns_1@10.2.1.66' saw that node 'ns_1@10.2.1.64' went down. Details: [

      {nodedown_reason, net_tick_timeout}] ns_node_disco005 ns_1@10.2.1.66 17:38:18 - Thu Oct 10, 2013
      Node 'ns_1@10.2.1.65' saw that node 'ns_1@10.2.1.64' went down. Details: [{nodedown_reason, net_tick_timeout}

      ] ns_node_disco005 ns_1@10.2.1.65 17:38:11 - Thu Oct 10, 2013

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            junyi Junyi Xie (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty