[longevity] view compaction crashed on default bucket

Cluster information:

• 11 centos 6.2 64bit server with 4 cores CPU
• Each server has 10 GB RAM and 150 GB disk.
• 8 GB RAM for couchbase server at each node (80% total system memmories)
• Disk format ext3 on both data and root
• Each server has its own drive, no disk sharing with other server.
• Load 9 million items to both buckets and querying them continuously
• Initial indexing, so cpu a little heavy load
• Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
• Add one more doc d2 with 2 views to default bucket

• Create cluster with 10 nodes installed couchbase server 2.0.0-1663
  10.3.121.13
  10.3.121.14
  10.3.121.15
  10.3.121.16
  10.3.121.17
  10.3.121.20
  10.3.121.22
  10.3.121.24
  10.3.121.25
  10.3.121.23
• Data path /data
• View path /data

When analyzing log of node 22 from bug MB-6528, I also see compaction for view of default bucket crashed. We can see this look at file size of 2 docs on this default bucket below.

Link to collect info of node 22
https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_0/201209/info_node22-20120905.zip

ns_server:warn,2012-09-05T0:03:07.145,ns_1@10.3.121.22:<0.5136.402>:compaction_daemon:do_chain_compactors:524]Compactor for view `default/_design/d2` (pid [

{type,view}

,

{name,<<"default/_design/d2">>}

,

{important,false}

,
{fa,
{#Fun<compaction_daemon.21.129945092>,
[<<"default">>,
<<"_design/d2">>,
{config,

{30,18446744073709551616},
{30,18446744073709551616}

,
undefined,false,
{daemon_config,30,131072}},
false,
{[

{type,bucket}

]}]}}]) terminated unexpectedly (ignoring this):

{updater_died, noproc}

$ python scripts/ssh.py -i ../ini/10-c-long.ini "ls -lh /data/@indexes/default "
10.3.121.13
total 174M
rw-rr- 1 couchbase couchbase 51M Sep 5 01:15 main_4eff0a2f4c94df925df1d18aed740056.view.43
rw-rr- 1 couchbase couchbase 48M Sep 5 01:15 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.43
rw-rr- 1 couchbase couchbase 39M Sep 5 01:15 replica_4eff0a2f4c94df925df1d18aed740056.view.48
rw-rr- 1 couchbase couchbase 37M Sep 5 01:15 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.47

10.3.121.22
total 11G
rw-rr-. 1 couchbase couchbase 88M Sep 5 01:21 main_4eff0a2f4c94df925df1d18aed740056.view.16
rw-rr-. 1 couchbase couchbase 83M Sep 5 01:21 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.25
rw-rr-. 1 couchbase couchbase 8.4G Sep 5 01:56 replica_4eff0a2f4c94df925df1d18aed740056.view.20
rw-rr-. 1 couchbase couchbase 0 Sep 5 01:21 replica_4eff0a2f4c94df925df1d18aed740056.view.log
rw-rr-. 1 couchbase couchbase 2.4G Sep 5 01:37 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.33

10.3.121.15
total 536M
rw-rr- 1 couchbase couchbase 126M Sep 5 03:04 main_4eff0a2f4c94df925df1d18aed740056.view.21
rw-rr- 1 couchbase couchbase 119M Sep 5 03:08 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.23
rw-rr- 1 couchbase couchbase 150M Sep 5 03:06 replica_4eff0a2f4c94df925df1d18aed740056.view.25
rw-rr- 1 couchbase couchbase 142M Sep 5 03:21 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.31

10.3.121.23
total 547M
rw-rr-. 1 couchbase couchbase 84M Sep 5 01:47 main_4eff0a2f4c94df925df1d18aed740056.view.14
rw-rr-. 1 couchbase couchbase 80M Sep 5 01:51 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.18
rw-rr-. 1 couchbase couchbase 197M Sep 5 02:11 replica_4eff0a2f4c94df925df1d18aed740056.view.18
rw-rr-. 1 couchbase couchbase 187M Sep 5 02:14 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.23

10.3.121.24
total 477M
rw-rr-. 1 couchbase couchbase 71M Sep 5 01:21 main_4eff0a2f4c94df925df1d18aed740056.view.16
rw-rr-. 1 couchbase couchbase 67M Sep 5 01:32 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.24
rw-rr-. 1 couchbase couchbase 175M Sep 5 01:34 replica_4eff0a2f4c94df925df1d18aed740056.view.16
rw-rr-. 1 couchbase couchbase 165M Sep 5 01:33 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.22

10.3.121.14
total 447M
rw-rr-. 1 couchbase couchbase 94M Sep 5 02:17 main_4eff0a2f4c94df925df1d18aed740056.view.19
rw-rr-. 1 couchbase couchbase 89M Sep 5 02:45 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.22
rw-rr-. 1 couchbase couchbase 136M Sep 5 03:02 replica_4eff0a2f4c94df925df1d18aed740056.view.22
rw-rr-. 1 couchbase couchbase 129M Sep 5 03:03 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.27
drwxr-xr-x. 2 couchbase couchbase 4.0K Sep 3 02:33 tmp_4eff0a2f4c94df925df1d18aed740056_replica
drwxr-xr-x. 2 couchbase couchbase 4.0K Sep 3 02:36 tmp_87d0cc9a8fffc2e1e434f6ddbb0c168d_replica

10.3.121.20
total 515M
rw-rr-. 1 couchbase couchbase 74M Sep 5 01:22 main_4eff0a2f4c94df925df1d18aed740056.view.17
rw-rr-. 1 couchbase couchbase 70M Sep 5 01:22 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.24
rw-rr-. 1 couchbase couchbase 191M Sep 5 02:30 replica_4eff0a2f4c94df925df1d18aed740056.view.25
rw-rr-. 1 couchbase couchbase 181M Sep 5 02:29 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.32

10.3.121.16
total 529M
rw-rr-. 1 couchbase couchbase 114M Sep 5 02:27 main_4eff0a2f4c94df925df1d18aed740056.view.19
rw-rr-. 1 couchbase couchbase 108M Sep 5 02:10 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.20
rw-rr-. 1 couchbase couchbase 158M Sep 5 09:27 replica_4eff0a2f4c94df925df1d18aed740056.view.18
rw-rr-. 1 couchbase couchbase 150M Sep 5 09:27 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.23

10.3.121.17
total 597M
rw-rr-. 1 couchbase couchbase 107M Sep 5 02:09 main_4eff0a2f4c94df925df1d18aed740056.view.18
rw-rr-. 1 couchbase couchbase 101M Sep 5 01:59 main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.23
rw-rr-. 1 couchbase couchbase 199M Sep 5 09:04 replica_4eff0a2f4c94df925df1d18aed740056.view.23
rw-rr-. 1 couchbase couchbase 192M Sep 5 09:04 replica_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.29