Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3998

false alert "IP address seems to have changed" in 3-4 nodes cluster under 10k ops/sec set load

    XMLWordPrintable

Details

    Description

      from logs :

      Starting rebalance, KeepNodes = ['ns_1@10.1.6.149','ns_1@10.1.6.150',
      'ns_1@10.1.6.151','ns_1@10.1.6.152',
      'ns_1@10.1.6.154'], EjectNodes = ['ns_1@10.1.6.153']
      ns_orchestrator004 ns_1@10.1.6.150 09:10:57 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 09:09:53 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.152'. menelaus_web_alerts_srv001 ns_1@10.1.6.152 09:09:32 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.152'. (repeated 1 times) menelaus_web_alerts_srv001 ns_1@10.1.6.152 09:09:25 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.152'. menelaus_web_alerts_srv001 ns_1@10.1.6.152 09:03:29 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.149'. menelaus_web_alerts_srv001 ns_1@10.1.6.149 08:56:36 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:50:10 - Tue Jun 21, 2011
      Rebalance completed successfully.
      ns_orchestrator001 ns_1@10.1.6.149 08:42:46 - Tue Jun 21, 2011
      Starting rebalance, KeepNodes = ['ns_1@10.1.6.149','ns_1@10.1.6.154',
      'ns_1@10.1.6.150','ns_1@10.1.6.151',
      'ns_1@10.1.6.153','ns_1@10.1.6.152'], EjectNodes = []
      ns_orchestrator004 ns_1@10.1.6.149 08:42:44 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.153'. (repeated 1 times) menelaus_web_alerts_srv001 ns_1@10.1.6.153 08:41:42 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.153'. menelaus_web_alerts_srv001 ns_1@10.1.6.153 08:36:27 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:33:10 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. (repeated 4 times) menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:28:41 - Tue Jun 21, 2011
      Server error during processing: ["web request failed",

      {path,"/pools/default/saslBucketsStreaming"}

      ,

      {type,exit}

      ,
      {what,
      {timeout,

      {gen_server,call,[ns_config,get]}

      }},
      {trace,
      [

      {diag_handler,diagnosing_timeouts,1}

      ,

      {ns_bucket,get_buckets,0}

      ,

      {menelaus_web_buckets, '-handle_sasl_buckets_streaming/2-fun-2-', 2}

      ,

      {menelaus_web,streaming_inner,3}

      ,

      {menelaus_web,handle_streaming,4}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}] menelaus_web019 ns_1@10.1.6.151 08:25:35 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:23:22 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.154'. menelaus_web_alerts_srv001 ns_1@10.1.6.154 08:23:20 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. (repeated 1 times) menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:14:41 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:09:13 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. (repeated 2 times) menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:05:40 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 08:00:27 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 07:51:47 - Tue Jun 21, 2011
      Rebalance completed successfully.
      ns_orchestrator001 ns_1@10.1.6.149 07:41:31 - Tue Jun 21, 2011
      Starting rebalance, KeepNodes = ['ns_1@10.1.6.149','ns_1@10.1.6.154',
      'ns_1@10.1.6.150','ns_1@10.1.6.151',
      'ns_1@10.1.6.153','ns_1@10.1.6.152'], EjectNodes = []
      ns_orchestrator004 ns_1@10.1.6.149 07:41:28 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.153'. menelaus_web_alerts_srv001 ns_1@10.1.6.153 07:32:09 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.152'. menelaus_web_alerts_srv001 ns_1@10.1.6.152 07:13:56 - Tue Jun 21, 2011
      Rebalance completed successfully.
      ns_orchestrator001 ns_1@10.1.6.149 06:40:09 - Tue Jun 21, 2011
      Starting rebalance, KeepNodes = ['ns_1@10.1.6.149','ns_1@10.1.6.154',
      'ns_1@10.1.6.150','ns_1@10.1.6.151',
      'ns_1@10.1.6.153','ns_1@10.1.6.152'], EjectNodes = []
      ns_orchestrator004 ns_1@10.1.6.149 06:40:05 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 05:39:10 - Tue Jun 21, 2011
      Rebalance completed successfully.
      ns_orchestrator001 ns_1@10.1.6.149 05:38:41 - Tue Jun 21, 2011
      Starting rebalance, KeepNodes = ['ns_1@10.1.6.149','ns_1@10.1.6.154',
      'ns_1@10.1.6.150','ns_1@10.1.6.151',
      'ns_1@10.1.6.153','ns_1@10.1.6.152'], EjectNodes = []
      ns_orchestrator004 ns_1@10.1.6.149 05:38:38 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.152'. menelaus_web_alerts_srv001 ns_1@10.1.6.152 05:30:30 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.153'. (repeated 1 times) menelaus_web_alerts_srv001 ns_1@10.1.6.153 05:27:41 - Tue Jun 21, 2011
      IP address seems to have changed. Unable to listen on 'ns_1@10.1.6.151'. menelaus_web_alerts_srv001 ns_1@10.1.6.151 05:27:34 - Tue Jun 21, 2011

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            karan Karan Kumar (Inactive)
            farshid Farshid Ghods (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty