[system test] view compaction hang at 98 percents

Cluster information:

• 8 centos 6.2 64bit server with 4 cores CPU
• Each server has 32 GB RAM and 400 GB SSD disk.
• 24.8 GB RAM for couchbase server at each node
• SSD disk format ext4 on /data
• Each server has its own SSD drive, no disk sharing with other server.
• Create cluster with 6 nodes installed couchbase server 2.0.0-1793
• Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
• Enable consistent view on cluster (by default)

10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43

• Load 14 million items to each bucket. Each key has size from 512 bytes to 1024 bytes
• Queries all 4 views from 2 docs

• Data path /data
• View path /data

• Manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.0-1793-rel.rpm.manifest.xml
• I just load items to this cluster up to 14 million to each bucket. Then mutate these items. No rebalance

• I monitor view compaction process and disk size. I see disk size increase in node 43.

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../ini/10-c-long.ini "df -kh | grep data"
10.6.2.42
394G 58G 337G 15% /data
10.6.2.39
394G 54G 321G 15% /data
10.6.2.43
394G 68G 327G 18% /data
10.6.2.40
394G 55G 319G 15% /data
10.6.2.38
394G 38G 337G 11% /data
10.6.2.37
394G 38G 336G 11% /data

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../ini/10-c-long.ini "df -kh | grep data"
10.6.2.42
394G 65G 330G 17% /data
10.6.2.39
394G 60G 315G 16% /data
10.6.2.43
394G 77G 318G 20% /data
10.6.2.38
394G 42G 333G 12% /data
10.6.2.40
394G 62G 313G 17% /data
10.6.2.37
394G 42G 333G 12% /data

Going to node 43 and check couchdb.x log, I see view compaction started at Wed Oct 03 2012 15:09:22 GMT-7 and hang at 98 percent and retry to run more than 30 times.

[root@localhost logs]# grep started_on,1349302162 couchdb.*
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},
couchdb.1: {started_on,1349302162}

,
couchdb.1:

{started_on,1349302162},

couchdb.2: {started_on,1349302162}

,
couchdb.2:

{started_on,1349302162},
couchdb.2: {started_on,1349302162}

,

couchdb.3:

{started_on,1349302162},
couchdb.3: {started_on,1349302162}

,
couchdb.3:

{started_on,1349302162}

,

Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-info-1793-view-compaction-hang-not-reb-20121003-164151.tgz

Let me know if you need more information.