Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10226

Nodes go down all the time

    XMLWordPrintable

Details

    • Triaged
    • Windows 64-bit

    Description

      Hi,
      we have small 2 nodes cluster for development, and nodes in this cluster go down all the time.
      They don't do it together, usually it's one of them.
      It happens during re-balance and during regular operation.

      Here is what we see in the log:
      -----------------------------------------------------------------
      Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2014-02-17 08:31:00: (cproxy_config.c.315) env: MOXI_SASL_PLAIN_USR (13)
      2014-02-17 08:31:00: (cproxy_config.c.324) env: MOXI_SASL_PLAIN_PWD (9)
      WARNING: curl error: Recv failure: Connection was reset from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
      EOL on stdin. Exiting

      -------------------------------------------------

      Port server ns_server on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: kernel-poll not supported; "K" parameter ignored
      Apache CouchDB 1.2.0a-a425d97-git (LogLevel=info) is starting.
      Apache CouchDB has started. Time to relax.
      working as port

      Crash dump was written to: erl_crash.dump
      eheap_alloc: Cannot allocate 153052320 bytes of memory (of type "heap").

      This application has requested the Runtime to terminate it in an unusual way.
      Please contact the application's support team for more information.
      in message_loop

      ------------------------------------------------------------------

      Node 'ns_1@172.19.20.8' saw that node 'ns_1@172.19.20.3' went down. Details: [

      {nodedown_reason, net_tick_timeout}

      ]

      --------------------------------------------------------------------------

      Server error during processing: ["web request failed",

      {path,"/pools/default/overviewStats"}

      ,

      {type,exit}

      ,
      {what,
      {{nodedown,'ns_1@172.19.20.3'},
      {gen_server,call,
      [

      {menelaus_stats_gatherer, 'ns_1@172.19.20.3'}

      ,
      {gather_stats,
      "classicguest-englishinternational",
      ['ns_1@172.19.20.3'],
      undefined,
      {1,hour,900}},
      infinity]}}},
      {trace,
      [

      {gen_server,call,3}

      ,

      {menelaus_stats,grab_aggregate_op_stats,4}

      ,

      {lists,map,2},{lists,map,2}

      ,

      {menelaus_stats,handle_overview_stats,2}

      ,

      {request_throttler,do_request,3}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ]}]

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            michaelb michaelb
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty