Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6582

erl_crash occurred on node after removing max number of buckets(31) in the cluster and rebalance out the cluster/ eheap_alloc: Cannot allocate 5568010120 bytes of memory (of type "heap").- incremental_rebalance_in_out_with_max_buckets_number

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 2.0
    • 2.0-beta
    • ns_server
    • Security Level: Public
    • None

    Description

      build 1700

      /testrunner -i /tmp/incremental_rebalance_in_out_with_max_buckets_number (rebalance.rebalanceinout.RebalanceInOutTests) ... rebalance_in.ini get-logs=True -t rebalance.rebalanceinout.RebalanceInOutTests.incremental_rebalance_in_out_with_max_buckets_number,items=10000,default_bucket=False,GROUP=IN_OUT_LONG
      http://qa.hq.northscale.net/job/centos-64-2.0-new-rebalance/59/consoleFull

      test description:
      This test begins by creating max number of buckets with bucket_size=100( by default):
      one default bucket, all other are sasl and standart buckets. Then we load
      a given number of items(10000) into the cluster. It then removes two nodes,
      rebalances that nodes out the cluster, and then rebalances them back
      in. During the rebalancing we update all of the items in the cluster. Once the
      node has been removed and added back we wait for the disk queues to drain, and
      then verify that there has been no data loss. We then remove and add back two
      nodes at a time and so on until we have reached the point where we are adding
      back and removing at least half of the nodes.

      as a result:
      INFO - total 32 buckets will be created with size 100 MB

      eheap_alloc: Cannot allocate 5568010120 bytes of memory (of type "heap").

      [error_logger:error,2012-09-09T12:18:06.418,ns_1@10.3.3.94:error_logger:ale_error_logger_handler:log_msg:76]Mnesia('ns_1@10.3.3.94'): ** WARNING ** Mnesia is overloaded:

      {dump_log, write_threshold}

      [error_logger:error,2012-09-09T12:18:06.418,ns_1@10.3.3.94:error_logger:ale_error_logger_handler:log_msg:76]** Generic server 'stats_archiver-bucket0' terminating

        • Last message in was init
        • When Server state == {state,"bucket0"}
        • Reason for termination ==
        • {timeout,
          {gen_server,call,
          [mb_mnesia,
          Unknown macro: {ensure_table,'stats_archiver-bucket0-day', [{record_name,stat_entry}, {type,ordered_set}, {local_content,true}, {attributes,[timestamp,values]}]}

          ,
          30000]}}

      Attachments

        1. 10.3.3.92-8091-diag.txt.gz
          12.05 MB
        2. 10.3.3.93-8091-diag.txt.gz
          11.43 MB
        3. 10.3.3.99-8091-diag.txt.gz
          11.65 MB
        4. erl_crash.dump
          13.19 MB
        5. log_94.tar.gz
          14.46 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty