Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8189

node crashed, impossible to stop/restart server ( alarm_handler: {clear,system_memory_high_watermark})

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.1.0
    • 2.1.0
    • ns_server
    • Security Level: Public
    • None
    • Centos 64-bit

    Description

      it happened on build 2.0.2-782-rel http://qa.hq.northscale.net/job/centos-64-2.0-basic-rebalance-tests-P0

      don't know exactly what happened but node is not available

      I tried to check it status through service, but 10.3.121.69 hangs in console

      [root@cen-2710 couchbase]# ps -ef| grep couchbase
      101 6126 1 0 02:06 ? 00:00:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
      101 6144 1 0 02:06 ? 00:01:34 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -name babysitter_of_ns_1@127.0.0.1 -setcookie nocookie -noshell -noinput -noshell -noinput -eval os:putenv("ERL_MAX_PORTS",""). -run ns_babysitter_bootstrap – -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie-ns-server" -ns_server enable_mlockall true
      101 6171 6144 8 02:06 ? 00:35:44 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -nouser -run child_erlang child_start ns_bootstrap – -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie-ns-server" -ns_server enable_mlockall true
      101 6202 6171 0 02:06 ? 00:00:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup
      101 6204 6171 0 02:06 ? 00:00:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
      101 6205 6171 0 02:06 ? 00:00:00 /opt/couchbase/lib/erlang/lib/ssl-4.1.6/priv/bin/ssl_esock
      101 6212 6144 2 02:06 ? 00:08:44 /opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler.so -X /opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/opt/couchbase/var/lib/couchbase/logs/memcached.log -l 0.0.0.0:11210,0.0.0.0:11209:1000 -p 11210 -E /opt/couchbase/lib/memcached/bucket_engine.so -B binary -r -c 10000 -e admin=_admin;default_bucket_name=default;auto_create=false
      root 19223 19141 0 08:57 pts/0 00:00:00 /bin/sh /etc/init.d/couchbase-server stop
      root 19227 19223 0 08:57 pts/0 00:00:00 /bin/sh /opt/couchbase/bin/couchbase-server -k
      root 19231 19227 0 08:57 pts/0 00:00:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp – -root /opt/couchbase/lib/erlang -progname erl – -home /root – -name executioner@executioner -noshell -hidden -setcookie zuuvfkdwokyawadh -eval ns_babysitter_bootstrap:remote_stop('babysitter_of_ns_1@127.0.0.1')
      root 19557 19305 0 09:01 pts/2 00:00:00 /usr/bin/python /opt/couchbase/bin/cbcollect_info 10.3.121.69-522013-91-diag.zip
      root 19899 19557 0 09:02 pts/2 00:00:00 python /opt/couchbase/lib/python/cbstats -a 127.0.0.1:11210 all -b _admin -p _admin
      root 19928 19276 0 09:09 pts/1 00:00:00 grep couchbase

      [root@cen-2710 couchbase]# /etc/init.d/couchbase-server status
      couchbase-server is running

      /etc/init.d/couchbase-server stop
      Stopping couchbase-server
      HANGS...

      don't see memcached or erlang crashes

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty