Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7247

rebalance-in failed with beam.smp resident set size ~20GB

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
    • Environment:
      Physical machines, CentOS 5.8, 32GB RAM, 4 cores, 2 x SSD
      Build 1956
      10M x 2KB items, 1 ddoc, 8 views, 4 -> 3

      Description

      I've never seen such a mess in logs, but apart from all issues memory consumption looks really weird:

      1. ps -eo rss,vsize,args | grep beam
        19867592 27554248 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -S 16:16 -sbt u -P 327680 -K true – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -noshell -noinput -noshell -noinput -run ns_bootstrap – -name ns_1@10.2.1.66 -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.node" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        pavelpaulau Pavel Paulau added a comment -

        Too late.

        Show
        pavelpaulau Pavel Paulau added a comment - Too late.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Logs are useless here. Logs are rotated past interesting point when problem started about 15:30. From atop I can only see that initial bump in process size appears to coincide with massive increase in dirty pages count.

        Erlang's crash dump would help a lot here.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Logs are useless here. Logs are rotated past interesting point when problem started about 15:30. From atop I can only see that initial bump in process size appears to coincide with massive increase in dirty pages count. Erlang's crash dump would help a lot here.
        Hide
        steve Steve Yen added a comment -

        from bug-scrub mtg – assigning back to @pavel - please try to capture more info the next time you see this.

        Show
        steve Steve Yen added a comment - from bug-scrub mtg – assigning back to @pavel - please try to capture more info the next time you see this.
        Hide
        steve Steve Yen added a comment -

        from perf-sync mtg, moving back to 2.0 to help track this better.

        build 2.0.0-1954 had the 16:16 scheduler threads change, so this is the first time Pavel's tested 16:16 configuration.

        Pavel, can you re-run this experiment and see how reproducible this is?
        Thanks,
        Steve

        Show
        steve Steve Yen added a comment - from perf-sync mtg, moving back to 2.0 to help track this better. build 2.0.0-1954 had the 16:16 scheduler threads change, so this is the first time Pavel's tested 16:16 configuration. Pavel, can you re-run this experiment and see how reproducible this is? Thanks, Steve
        Hide
        steve Steve Yen added a comment -

        resolving this for now as next build 1966/1967 didn't have this issue; let's re-open if it occurs again and add the usual collect-info.

        Show
        steve Steve Yen added a comment - resolving this for now as next build 1966/1967 didn't have this issue; let's re-open if it occurs again and add the usual collect-info.

          People

          • Assignee:
            pavelpaulau Pavel Paulau
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes