Compaction daemon keeps crashing on one node with error "view_undefined" resulting in large index files

Compaction daemon keeps crashing on one node with error "view_undefined" resulting in large index files and causing disk space to fill up.

Steps to reproduce:

1. Create a sasl bucket and load 1M items.
2. Create 11 design docs (2 views each, different map functions) and allow initial indexing to finish.
3. Add 3 nodes and rebalance.
4. During rebalance do data mutations and view queries.
5. Disk space usage on one of the nodes goes very high and index files created are very large. Compaction daemon crashes can be seen in the logs.

The disk space usage of @indexes folder is as below:

bash> ./scripts/ssh.py -i ~/workspace/stash/ini/vm-4nodes.ini du -hs /opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket
10.1.3.70
26G /opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket

10.1.3.67
306M /opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket

10.1.3.73
1.6G /opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket

10.1.3.227
1.6G /opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket

10.1.3.70 node is taking 26GB while others are less than < 2GB.

The individual index files are very large on one node:

jenkins@ubu-0106:/opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket$ du -hs
26G .
jenkins@ubu-0106:/opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket$ du -hs *
1.3G main_1cc12c06716f41f8bc75b215f8dbc086.view.1
4.9G main_2382a9a5d7d4d00861777a96541428d9.view.1
1.5G main_251924c21ed8df73e9df82407069c3c1.view.1
1.4G main_4c93c8106130969fb0c31dbb1bc07677.view.1
4.5G main_4eff0a2f4c94df925df1d18aed740056.view.1
1.4G main_6aa8d5b7a578d96117c2a79ddf1be281.view.1
3.8G main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.1
1.3G main_9776f37c13209e06e5ae0c747a7e0dec.view.1
1.3G main_a9af5d78e75c7a9d16844669c0ddb475.view.1
4.9G main_ae72f9d24da5d9368eed3fb3519c1687.view.1
4.0K main_c6f70b45767324955fea92324798eed7.view.1
4.0K tmp_1cc12c06716f41f8bc75b215f8dbc086_main
4.0K tmp_2382a9a5d7d4d00861777a96541428d9_main
4.0K tmp_251924c21ed8df73e9df82407069c3c1_main
4.0K tmp_4c93c8106130969fb0c31dbb1bc07677_main
4.0K tmp_4eff0a2f4c94df925df1d18aed740056_main
4.0K tmp_6aa8d5b7a578d96117c2a79ddf1be281_main
4.0K tmp_87d0cc9a8fffc2e1e434f6ddbb0c168d_main
4.0K tmp_9776f37c13209e06e5ae0c747a7e0dec_main
4.0K tmp_a9af5d78e75c7a9d16844669c0ddb475_main
4.0K tmp_ae72f9d24da5d9368eed3fb3519c1687_main

as compared to other nodes:

root@ubu-0102:/opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket# du -hs
306M .
root@ubu-0102:/opt/couchbase/var/lib/couchbase/data/@indexes/saslbucket# du -hs *
23M main_1cc12c06716f41f8bc75b215f8dbc086.view.3
44M main_2382a9a5d7d4d00861777a96541428d9.view.2
28M main_251924c21ed8df73e9df82407069c3c1.view.9
25M main_4c93c8106130969fb0c31dbb1bc07677.view.5
25M main_4eff0a2f4c94df925df1d18aed740056.view.5
25M main_6aa8d5b7a578d96117c2a79ddf1be281.view.4
24M main_87d0cc9a8fffc2e1e434f6ddbb0c168d.view.7
24M main_9776f37c13209e06e5ae0c747a7e0dec.view.6
25M main_a9af5d78e75c7a9d16844669c0ddb475.view.3
41M main_ae72f9d24da5d9368eed3fb3519c1687.view.2
26M main_c6f70b45767324955fea92324798eed7.view.4
4.0K tmp_1cc12c06716f41f8bc75b215f8dbc086_main
4.0K tmp_2382a9a5d7d4d00861777a96541428d9_main
4.0K tmp_251924c21ed8df73e9df82407069c3c1_main
4.0K tmp_4c93c8106130969fb0c31dbb1bc07677_main
4.0K tmp_4eff0a2f4c94df925df1d18aed740056_main
4.0K tmp_6aa8d5b7a578d96117c2a79ddf1be281_main
4.0K tmp_87d0cc9a8fffc2e1e434f6ddbb0c168d_main
4.0K tmp_9776f37c13209e06e5ae0c747a7e0dec_main
4.0K tmp_a9af5d78e75c7a9d16844669c0ddb475_main
4.0K tmp_ae72f9d24da5d9368eed3fb3519c1687_main
4.0K tmp_c6f70b45767324955fea92324798eed7_main

In the logs, following Crash reports can be seen for the compaction daemon:

=========================CRASH REPORT=========================
crasher:
initial call: compaction_daemon:spawn_view_index_compactor/6-fun-0/0
pid: <0.1673.8>
registered_name: []
exception throw:

{error,view_undefined}
in function couch_set_view_group:get_data_size/1
in call from couch_set_view:get_group_data_size/2
in call from compaction_daemon:get_group_data_info/3
in call from compaction_daemon:view_needs_compaction/4
in call from compaction_daemon:'spawn_view_index_compactor/6-fun-0'/7
ancestors: [<0.1672.8>,<0.1669.8>,<0.1667.8>,compaction_daemon,
<0.3449.0>,ns_server_sup,ns_server_cluster_sup,<0.60.0>]
messages: []
links: [<0.1672.8>]
dictionary: []
trap_exit: false
status: running
heap_size: 610
stack_size: 24
reductions: 241
neighbours:

[ns_server:warn,2012-08-17T23:47:00.004,ns_1@10.1.3.70:<0.1669.8>:compaction_daemon:do_chain_compactors:516]Compactor for view `saslbucket/_design/d3` (pid [{type,view},
{name, <<"saslbucket/_design/d3">>},
{important,false},
{fa,
{#Fun<compaction_daemon.21.91655267>,
[<<"saslbucket">>,
<<"_design/d3">>,
{config,
{30,18446744073709551616},
{30,18446744073709551616},
undefined,false,
{daemon_config,30,
131072}},
false,bucket]}}]) terminated unexpectedly (ignoring this): {error, view_undefined}

[error_logger:error,2012-08-17T23:47:00.004,ns_1@10.1.3.70:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: compaction_daemon:spawn_view_compactor/5-fun-1/0
pid: <0.1672.8>
registered_name: []
exception exit: {error,view_undefined}

in function compaction_daemon:do_chain_compactors/2
ancestors: [<0.1669.8>,<0.1667.8>,compaction_daemon,<0.3449.0>,
ns_server_sup,ns_server_cluster_sup,<0.60.0>]
messages: []
links: [<0.1669.8>]
dictionary: []
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 3766
neighbours:

[ns_server:debug,2012-08-17T23:47:00.005,ns_1@10.1.3.70:compaction_daemon:compaction_daemon:handle_info:306]Finished compaction iteration.