Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5849

Hard Out Of Memory Error from some of nodes in 1.8.1 longevity large cluster test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0
    • 1.8.1
    • couchbase-bucket
    • Security Level: Public
    • None
    • windows 2008 R2 64 bit

    Description

      Install couchbase server 1.8.1-916 on 10 nodes cluster in longevity test.
      Load 40 million items size from 70 to 512 bytes to cluster with around 10K ops
      Keep load running around 10K (update and delete items) during rebalance run below.
      Do rebalance in and out one by one node.
      Load 30 million items more to cluster.
      Delete 20 million items from cluster.
      Do swap rebalance one by one with couchbase server version 1.8.1-937 (4 nodes swapped to build 937)
      During swap node (add 79 and remove 78), restart physical server a node. When this node was done warmup, replica items from other nodes dropdown to almost zero. Then gradually increase then item number.
      After few days, the replica items from node 10 , 11, 12, 182 and 183 did not go back to the same of active items.
      Cluster also gets a lot of Hard Out Of Memory Error from nodes 10 , 11, 12, 182 and 183

      Here are their IPs:
      1:10.3.2.11
      2:10.3.2.12
      3:10.3.2.16
      4:10.3.2.10
      5:10.3.2.75
      6:10.3.2.76
      7:10.3.2.77
      8:10.3.2.78
      9:10.3.2.79
      10:10.3.121.185
      11:10.3.121.182
      12:10.3.121.183

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            farshid Farshid Ghods (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty