Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6592

[longevity] memcached hangs when aborting during swap rebalance operation and fails to restart ( exit 71 )

    Details

      Description

      Cluster information:

      • 11 centos 6.2 64bit server with 4 cores CPU
      • Each server has 10 GB RAM and 150 GB disk.
      • 8 GB RAM for couchbase server at each node (80% total system memmories)
      • Disk format ext3 on both data and root
      • Each server has its own drive, no disk sharing with other server.
      • Load 9 million items to both buckets
      • Initial indexing, so cpu a little heavy load
      • Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
      • Create cluster with 10 nodes installed couchbase server 2.0.0-1697

      10.3.121.13
      10.3.121.14
      10.3.121.15
      10.3.121.16
      10.3.121.17
      10.3.121.20
      10.3.121.22
      10.3.121.24
      10.3.121.25
      10.3.121.23

      • Data path /data
      • View path /data
      • Do swap rebalance. Add node 26 and remove node 25
      • Rebalance failed and saw a lot of error message memcached exited with status 71 in log page.

      Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/11nodes-1697-memcached-exit-71-20120910.tgz

      Link to atop node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13
      Due to large size of atop file, all other atop files are in /tmp directory of each node

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        thuan Thuan Nguyen created issue -
        karan Karan Kumar (Inactive) made changes -
        Field Original Value New Value
        Attachment memcached_logfile [ 14919 ]
        karan Karan Kumar (Inactive) made changes -
        Summary [longevity] memcached constantly exited with status 71 on node 13 [longevity] memcached constantly exited with status 71 and unable to restart
        karan Karan Kumar (Inactive) made changes -
        Assignee Karan Kumar [ karan ] Trond Norbye [ trond ]
        thuan Thuan Nguyen made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        karan Karan Kumar (Inactive) made changes -
        Assignee Trond Norbye [ trond ] Chiyoung Seo [ chiyoung ]
        chiyoung Chiyoung Seo made changes -
        Assignee Chiyoung Seo [ chiyoung ] Mike Wiederhold [ mikew ]
        chiyoung Chiyoung Seo made changes -
        Sprint Status Current Sprint
        mikew Mike Wiederhold made changes -
        Component/s couchbase-bucket [ 10173 ]
        Component/s bucket-engine [ 10010 ]
        mikew Mike Wiederhold made changes -
        Assignee Mike Wiederhold [ mikew ] Trond Norbye [ trond ]
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 2.0-beta-refresh [ 10385 ]
        Fix Version/s 2.0-beta [ 10113 ]
        chiyoung Chiyoung Seo made changes -
        Assignee Trond Norbye [ trond ] Chiyoung Seo [ chiyoung ]
        karan Karan Kumar (Inactive) made changes -
        Labels system-test
        farshid Farshid Ghods (Inactive) made changes -
        Summary [longevity] memcached constantly exited with status 71 and unable to restart [longevity] memcached hangs when aborting during swap rebalance operation and fails to restart
        farshid Farshid Ghods (Inactive) made changes -
        Summary [longevity] memcached hangs when aborting during swap rebalance operation and fails to restart [longevity] memcached hangs when aborting during swap rebalance operation and fails to restart ( exit 71 )
        farshid Farshid Ghods (Inactive) made changes -
        Labels system-test 2.0-beta-release-notes system-test
        chiyoung Chiyoung Seo made changes -
        Assignee Chiyoung Seo [ chiyoung ] Thuan Nguyen [ thuan ]
        chiyoung Chiyoung Seo made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Cannot Reproduce [ 5 ]
        chiyoung Chiyoung Seo made changes -
        Sprint Status Current Sprint
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes