[system test] view compaction crashes often

Environment:

• 9 windows 2008 R2 64bit.
• Each server has 4 CPU, 8GB RAM and SSD disk
• Cluster has 2 buckets, default and sasl bucket with consistent view enable.
• Load 26 million items to default bucket and 16 million items to sasl bucket. Each key has size from 128 to 512 bytes
• Each bucket has one doc and 2 views for each doc.

• Rebalance out 2 nodes 10.3.121.173 and 10.3.121.243

Starting rebalance, KeepNodes = ['ns_1@10.3.3.181','ns_1@10.3.121.47',
'ns_1@10.3.3.214','ns_1@10.3.3.182',
'ns_1@10.3.3.180','ns_1@10.3.121.171',
'ns_1@10.3.121.169'], EjectNodes = ['ns_1@10.3.121.173',
'ns_1@10.3.121.243'] ns_orchestrator004 ns_1@10.3.121.169 23:26:03 - Tue Jan 22, 2013

[couchdb:error,2013-01-23T5:02:44.204,ns_1@10.3.121.169:<0.10514.1>:couch_log:error:42]Set view `default`, replica group `_design/d1`, compactor process <0.24289.57> died with unexpected reason: {{badmatch,
{error,

{file_error, "c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact", enoent}

}},
[

{couch_set_view_compactor, merge_files, 3}

,

{couch_set_view_compactor, apply_log, 4}

,

{couch_set_view_compactor, maybe_retry_compact, 5}

]}
[ns_server:warn,2013-01-23T5:02:44.204,ns_1@10.3.121.169:<0.12760.55>:compaction_daemon:do_chain_compactors:660]Compactor for view `default/_design/d1/replica` (pid [

{type,view},
{important,true},
{name, <<"default/_design/d1/replica">>},
{fa,
{#Fun<compaction_daemon.16.119703424>,
[<<"default">>,
<<"_design/d1">>,
replica,
{config,
{30, 18446744073709551616},
{30, 18446744073709551616},
undefined,false,
false,
{daemon_config,30,
131072}},
false,
{[{type,bucket}]}]}}]) terminated unexpectedly: {{badmatch,
{error,
{file_error, "c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact", enoent}}},
[{couch_set_view_compactor, merge_files, 3},
{couch_set_view_compactor, apply_log, 4},
{couch_set_view_compactor, maybe_retry_compact, 5}]}
[error_logger:error,2013-01-23T5:02:44.204,ns_1@10.3.121.169:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: compaction_daemon:spawn_view_index_compactor/6-fun-0/0
pid: <0.29697.56>
registered_name: []
exception exit: {{badmatch,
{error,
{file_error, "c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact", enoent}}},
[{couch_set_view_compactor,merge_files,3},
{couch_set_view_compactor,apply_log,4},
{couch_set_view_compactor,maybe_retry_compact,5}]}
in function compaction_daemon:do_spawn_view_index_compactor/5
in call from compaction_daemon:'spawn_view_index_compactor/6-fun-0'/7
ancestors: [<0.12760.55>,<0.16968.54>,<0.16931.54>,compaction_daemon,
<0.152.1>,ns_server_sup,ns_server_cluster_sup,<0.67.0>]
messages: []
links: [<0.12760.55>]
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 2873
neighbours:

[error_logger:error,2013-01-23T5:02:44.219,ns_1@10.3.121.169:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]Error in process <0.24289.57> on node 'ns_1@10.3.121.169' with exit value: badmatch,{error,{file_error,"c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact",enoent},[{couch_set_view_compactor,merge_files,3},{couch_set_view_compactor...


[ns_server:warn,2013-01-23T5:02:44.219,ns_1@10.3.121.169:<0.16968.54>:compaction_daemon:do_chain_compactors:665]Compactor for view `default/_design/d1` (pid [{type,view}

,

{name,<<"default/_design/d1">>}

,

{important,false}

,
{fa,
{#Fun<compaction_daemon.20.123074146>,
[<<"default">>,
<<"_design/d1">>,
{config,

{30,18446744073709551616},
{30,18446744073709551616}

,
undefined,false,false,
{daemon_config,30,131072}},
false,
{[

{type,bucket}

]}]}}]) terminated unexpectedly (ignoring this): {{badmatch,
{error,

{file_error, "c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact", enoent}

}},
[

{couch_set_view_compactor, merge_files, 3}

,

{couch_set_view_compactor, apply_log, 4}

,

{couch_set_view_compactor, maybe_retry_compact, 5}

]}
[error_logger:error,2013-01-23T5:02:44.219,ns_1@10.3.121.169:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: compaction_daemon:spawn_view_compactor/5-fun-1/0
pid: <0.12760.55>
registered_name: []
exception exit: {{badmatch,
{error,

{file_error, "c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact", enoent}

}},
[

{couch_set_view_compactor,merge_files,3}

,

{couch_set_view_compactor,apply_log,4}

,

{couch_set_view_compactor,maybe_retry_compact,5}

]}
in function compaction_daemon:do_chain_compactors/2
ancestors: [<0.16968.54>,<0.16931.54>,compaction_daemon,<0.152.1>,
ns_server_sup,ns_server_cluster_sup,<0.67.0>]
messages: []
links: [<0.16968.54>]
dictionary: []
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 4965
neighbours:

Crash happens during rebalance. Rebalance failed around 8:30 AM Jan 23 2013

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-140-rel.setup.exe.manifest.xml

Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201301/9nodes-col-201-140-rebalance-failed-buckets-shutdown-20130123-14-11-14.tgz