[longevity] memcached hangs when aborting during swap rebalance operation and fails to restart ( exit 71 )

Cluster information:

• 11 centos 6.2 64bit server with 4 cores CPU
• Each server has 10 GB RAM and 150 GB disk.
• 8 GB RAM for couchbase server at each node (80% total system memmories)
• Disk format ext3 on both data and root
• Each server has its own drive, no disk sharing with other server.
• Load 9 million items to both buckets
• Initial indexing, so cpu a little heavy load
• Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

• Create cluster with 10 nodes installed couchbase server 2.0.0-1697

10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23

• Data path /data
• View path /data

• Do swap rebalance. Add node 26 and remove node 25
• Rebalance failed and saw a lot of error message memcached exited with status 71 in log page.

Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/11nodes-1697-memcached-exit-71-20120910.tgz

Link to atop node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13
Due to large size of atop file, all other atop files are in /tmp directory of each node