Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
2.0
-
Security Level: Public
-
None
-
build #1598 on Ubuntu 64bit
Description
Index compaction is stuck after rebalacing in nodes. I did the following steps:
1. Create sasl bucket with password(build#1598).
2. Load 1M items
3. Create 11 production ddocs(2 views per ddoc)
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d1' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,doc.key);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d2' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.key_num]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d3' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.name]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d4' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.email]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d5' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.city]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d6' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.country]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d7' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.realm]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d8' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.coins]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d9' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.category]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d10' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.achievements]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@10.1.3.67:8092/saslbucket/_design/d11' \
-d '{"views": { "v1":{"map":"function(doc)
"}, "v2":{"map":"function(doc,meta)
{\nemit(meta.id,[doc.key_num,doc.body]);\n}"} }}'
4. While the initial indexing is in progress, add 3 nodes and rebalance.
5. The index compaction gets stuck for one of the design docs. Following error can be seen in the diag repeatedly. After a while it filled up the disk space on the VM.
[ns_server:info,2012-08-17T4:19:44.465,ns_1@10.1.3.67:<0.29908.4>:compaction_daemon:spawn_view_index_compactor:664]Compacting indexes for saslbucket/_design/d9/main
[couchdb:info,2012-08-17T4:19:44.608,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Starting updater for set view `saslbucket`, main group `_design/d9`
[couchdb:info,2012-08-17T4:19:44.609,ns_1@10.1.3.67:<0.29910.4>:couch_log:info:39]Updater for set view `saslbucket`, main group `_design/d9` started
Active partitions: [32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63]
Passive partitions: []
Cleanup partitions: [55,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,106,107,108,109,110,111,112,113,114,115,116,127]
Replicas to transfer: []
Pending transition:
active: [55]
passive: []
Initial build: false
[couchdb:info,2012-08-17T4:19:44.610,ns_1@10.1.3.67:<0.29916.4>:couch_log:info:39]Updater reading changes from active partitions to update main set view group `_design/d9` from set `saslbucket`
[couchdb:info,2012-08-17T4:19:48.180,ns_1@10.1.3.67:<0.29915.4>:couch_log:info:39]Updater for set view `saslbucket`, main group `_design/d9`, performed cleanup of 118326 key/value pairs in 3.438 seconds
[couchdb:info,2012-08-17T4:19:48.184,ns_1@10.1.3.67:<0.29915.4>:couch_log:info:39]Updater checkpointing set view `saslbucket` update for main group `_design/d9`
[couchdb:info,2012-08-17T4:19:48.188,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Stopping updater for set view `saslbucket`, main group `_design/d9`
[couchdb:info,2012-08-17T4:19:48.192,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Updater, set view `saslbucket`, main group `_design/d9`, stopped with reason:
[couchdb:info,2012-08-17T4:19:48.195,ns_1@10.1.3.67:<0.20718.0>:couch_log:info:39]Restarting compaction for main group `_design/d9`, set view `saslbucket`. Reason: partition states were updated
[ns_server:warn,2012-08-17T4:19:48.195,ns_1@10.1.3.67:<0.29907.4>:compaction_daemon:do_chain_compactors:511]Compactor for view `saslbucket/_design/d9/main` (pid [
{important,true},
{name, <<"saslbucket/_design/d9/main">>},
{fa,
{#Fun<compaction_daemon.16.13428321>,
[<<"saslbucket">>,
<<"_design/d9">>,
main,
{config,
{10, 18446744073709551616},
{10, 18446744073709551616},
undefined,false,
{daemon_config,30,
131072}},
false,bucket]}}]) terminated unexpectedly: shutdown
[ns_server:warn,2012-08-17T4:19:48.200,ns_1@10.1.3.67:<0.29880.4>:compaction_daemon:do_chain_compactors:516]Compactor for view `saslbucket/_design/d9` (pid [{type,view}
,
,
,
{fa,
{#Fun<compaction_daemon.21.91655267>,
[<<"saslbucket">>,
<<"_design/d9">>,
{config,
{10,18446744073709551616}
,
undefined,false,
{daemon_config,30,
131072}},
false,bucket]}}]) terminated unexpectedly (ignoring this): shutdown
[ns_server:debug,2012-08-17T4:19:48.201,ns_1@10.1.3.67:compaction_daemon:compaction_daemon:handle_info:306]Finished compaction iteration.
[ns_server:debug,2012-08-17T4:19:48.203,ns_1@10.1.3.67:compaction_daemon:compaction_daemon:schedule_next_compaction:1129]Finished compaction too soon. Next run will be in 26s
Diagnostics are attached.