Compaction stops on the node with zero draining rate (DWQ > 1M) after data loading

environment: 9 nodes
1:10.3.3.183
2:10.3.3.184
3:10.3.3.185
4:10.3.3.187
5:10.3.121.74
6:10.3.121.77
7:10.3.121.80
8:10.3.121.83
9:10.3.121.185
Total 25. 6GB RAM and 333 GB Disk

load 7~8 Million items to default bucket, another 7~8 Million items to sasl bucket, avg_size 512 bytes

• • Non-View Realted Stats:
    • ops/sec = 2k/8k
    • Disk Write Queue = 3.5 M
    • data compaction = running
    • Tap Replication = 2k/6k
    • Resident = 75%

Due to MB-7046, for bucket default, dwq is 1.23M on node 10.3.3.185 with no draining over a long time. I also notice compaction stuck on this node, from the diag:

[couchdb:info,2012-10-31T15:26:27.652,ns_1@10.3.3.185:<0.20954.0>:couch_log:info:39]Compaction for db "default/205" completed.
[couchdb:info,2012-10-31T15:26:27.667,ns_1@10.3.3.185:<0.20948.0>:couch_log:info:39]Starting compaction for db "default/206"
[couchdb:info,2012-10-31T15:26:27.839,ns_1@10.3.3.185:<0.16589.1>:couch_log:info:39]Native compactor output: Compacted c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/206.c
[couchdb:info,2012-10-31T15:26:27.839,ns_1@10.3.3.185:<0.16589.1>:couch_log:info:39]Native compactor output: ouch.2 -> c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/206.c
[couchdb:info,2012-10-31T15:26:27.839,ns_1@10.3.3.185:<0.16589.1>:couch_log:info:39]Native compactor output: ouch.2.compact
[couchdb:info,2012-10-31T15:26:27.886,ns_1@10.3.3.185:<0.16589.1>:couch_log:info:39]Native initial compact succeeded for "default/206"
[couchdb:info,2012-10-31T15:26:27.886,ns_1@10.3.3.185:<0.20948.0>:couch_log:info:39]CouchDB swapping files c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/206.couch.3 and c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/206.couch.2.compact.
[couchdb:info,2012-10-31T15:26:28.011,ns_1@10.3.3.185:<0.20948.0>:couch_log:info:39]Compaction for db "default/206" completed.
[couchdb:info,2012-10-31T15:26:28.011,ns_1@10.3.3.185:<0.20942.0>:couch_log:info:39]Starting compaction for db "default/207"
[couchdb:info,2012-10-31T15:26:28.323,ns_1@10.3.3.185:<0.16600.1>:couch_log:info:39]Native compactor output: Compacted c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/207.c
[couchdb:info,2012-10-31T15:26:28.323,ns_1@10.3.3.185:<0.16600.1>:couch_log:info:39]Native compactor output: ouch.2 -> c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/207.c
[couchdb:info,2012-10-31T15:26:28.323,ns_1@10.3.3.185:<0.16600.1>:couch_log:info:39]Native compactor output: ouch.2.compact
[couchdb:info,2012-10-31T15:26:28.386,ns_1@10.3.3.185:<0.16600.1>:couch_log:info:39]Native initial compact succeeded for "default/207"
[couchdb:info,2012-10-31T15:26:28.386,ns_1@10.3.3.185:<0.16607.1>:couch_log:info:39]eacces error opening file "c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/207.couch.2.compact" waiting infinityms to retry

Due to MB-7046, for sasl bucket, dwq is 119K on node 10.3.121.80 with no draining over a long time. I also notice compaction stuck on this node, from the diag:

[couchdb:info,2012-10-31T16:06:14.718,ns_1@10.3.121.80:<0.21739.0>:couch_log:info:39]Starting compaction for db "saslbucket/446"
[couchdb:info,2012-10-31T16:06:18.968,ns_1@10.3.121.80:<0.10183.2>:couch_log:info:39]Native compactor output: Compacted c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket/44
[couchdb:info,2012-10-31T16:06:18.968,ns_1@10.3.121.80:<0.10183.2>:couch_log:info:39]Native compactor output: 6.couch.2 -> c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket
[couchdb:info,2012-10-31T16:06:18.968,ns_1@10.3.121.80:<0.10183.2>:couch_log:info:39]Native compactor output: /446.couch.2.compact
[couchdb:info,2012-10-31T16:06:18.999,ns_1@10.3.121.80:<0.10183.2>:couch_log:info:39]Native initial compact succeeded for "saslbucket/446"
[couchdb:info,2012-10-31T16:06:18.999,ns_1@10.3.121.80:<0.21739.0>:couch_log:info:39]CouchDB swapping files c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket/446.couch.3 and c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket/446.couch.2.compact.
[couchdb:info,2012-10-31T16:06:18.999,ns_1@10.3.121.80:<0.21739.0>:couch_log:info:39]Compaction for db "saslbucket/446" completed.
[couchdb:info,2012-10-31T16:06:18.999,ns_1@10.3.121.80:<0.21733.0>:couch_log:info:39]Starting compaction for db "saslbucket/447"
[couchdb:info,2012-10-31T16:06:23.609,ns_1@10.3.121.80:<0.10242.2>:couch_log:info:39]Native compactor output: Compacted c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket/44
[couchdb:info,2012-10-31T16:06:23.609,ns_1@10.3.121.80:<0.10242.2>:couch_log:info:39]Native compactor output: 7.couch.2 -> c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket
[couchdb:info,2012-10-31T16:06:23.609,ns_1@10.3.121.80:<0.10242.2>:couch_log:info:39]Native compactor output: /447.couch.2.compact
[couchdb:info,2012-10-31T16:06:23.609,ns_1@10.3.121.80:<0.10242.2>:couch_log:info:39]Native initial compact succeeded for "saslbucket/447"
[couchdb:info,2012-10-31T16:06:23.609,ns_1@10.3.121.80:<0.10285.2>:couch_log:info:39]eacces error opening file "c:/Program Files/Couchbase/Server/var/lib/couchbase/data/saslbucket/447.couch.2.compact" waiting infinityms to retry