[system test] [windows] replica resident ratio is zero after restart couchbase server in all nodes in cluster

Environment:
9 nodes windows server 2008 R2 64 bit, 8GB RAM with SSD

Create a cluster with 7 nodes

1:10.3.121.173
2:10.3.121.169
3:10.3.121.171
4:10.3.3.214
5:10.3.121.47
6:10.3.3.180
7:10.3.3.181

Create 2 buckets, default (3GB) and sasl (3GB)
Load 20+ million items to both bucket and continue to load until active resident ratio reaches 70% as in spec in the link without view and xdcr http://hub.internal.couchbase.com/confluence/display/QA/views+%28and+now+with+XDCR%29+tests

After rebalance in (10.3.3.182), swap rebalance ("add": "10.3.121.243", "rm": "10.3.121.173") and rebalance out (10.3.121.169), I do warm up test in all nodes of cluster.

After warmup, check raw warmup in all nodes of both buckets, I see access log corrupted in all nodes

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../../ini/sys9winssd.ini "/cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup"
10.3.121.243
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 33521
ep_warmup_estimated_key_count: 6916242
ep_warmup_estimated_value_count: 6916242
ep_warmup_item_expired: 0
ep_warmup_key_count: 6916242
ep_warmup_keys_time: 208425039
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 362988878
ep_warmup_value_count: 3163928

10.3.3.214
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 27137
ep_warmup_estimated_key_count: 6893096
ep_warmup_estimated_value_count: 6893096
ep_warmup_item_expired: 0
ep_warmup_key_count: 6893096
ep_warmup_keys_time: 262102714
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 434113385
ep_warmup_value_count: 3167852

10.3.121.171
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 32976
ep_warmup_estimated_key_count: 6917495
ep_warmup_estimated_value_count: 6917495
ep_warmup_item_expired: 0
ep_warmup_key_count: 6917495
ep_warmup_keys_time: 216321915
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 393767087
ep_warmup_value_count: 3164034

10.3.3.180
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 24976
ep_warmup_estimated_key_count: 6922336
ep_warmup_estimated_value_count: 6922336
ep_warmup_item_expired: 0
ep_warmup_key_count: 6922336
ep_warmup_keys_time: 274695214
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 448358825
ep_warmup_value_count: 3162234

10.3.3.181
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 32313
ep_warmup_estimated_key_count: 6894084
ep_warmup_estimated_value_count: 6894084
ep_warmup_item_expired: 0
ep_warmup_key_count: 6894084
ep_warmup_keys_time: 323674278
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 495902872
ep_warmup_value_count: 3168008

10.3.3.182
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 28491
ep_warmup_estimated_key_count: 6897519
ep_warmup_estimated_value_count: 6897519
ep_warmup_item_expired: 0
ep_warmup_key_count: 6897519
ep_warmup_keys_time: 303874509
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 466692959
ep_warmup_value_count: 3166739

10.3.121.169
Stats 'warmup' are not available from the requested engine.

10.3.121.173
Stats 'warmup' are not available from the requested engine.

10.3.121.47
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 27602
ep_warmup_estimated_key_count: 6921167
ep_warmup_estimated_value_count: 6921167
ep_warmup_item_expired: 0
ep_warmup_key_count: 6921167
ep_warmup_keys_time: 179543513
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 523044113
ep_warmup_value_count: 3162767

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../../ini/sys9winssd.ini "/cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup -b sasl -p password"
10.3.121.171
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 64604
ep_warmup_estimated_key_count: 12051078
ep_warmup_estimated_value_count: 12051078
ep_warmup_item_expired: 0
ep_warmup_key_count: 12051078
ep_warmup_keys_time: 204135789
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 305395753
ep_warmup_value_count: 2112722

10.3.3.214
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 33118
ep_warmup_estimated_key_count: 11265949
ep_warmup_estimated_value_count: 11265949
ep_warmup_item_expired: 0
ep_warmup_key_count: 11265949
ep_warmup_keys_time: 239255385
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 341690182
ep_warmup_value_count: 2297843

10.3.3.180
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 28273
ep_warmup_estimated_key_count: 11345757
ep_warmup_estimated_value_count: 11345757
ep_warmup_item_expired: 0
ep_warmup_key_count: 11345757
ep_warmup_keys_time: 249496951
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 349804246
ep_warmup_value_count: 2282299

10.3.3.181
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 29972
ep_warmup_estimated_key_count: 11255661
ep_warmup_estimated_value_count: 11255661
ep_warmup_item_expired: 0
ep_warmup_key_count: 11255661
ep_warmup_keys_time: 316480602
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 432175751
ep_warmup_value_count: 2272669

10.3.121.243
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 33005
ep_warmup_estimated_key_count: 11641542
ep_warmup_estimated_value_count: 11641542
ep_warmup_item_expired: 0
ep_warmup_key_count: 11641542
ep_warmup_keys_time: 221148861
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 290030482
ep_warmup_value_count: 2193036

10.3.3.182
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 31754
ep_warmup_estimated_key_count: 11648028
ep_warmup_estimated_value_count: 11648028
ep_warmup_item_expired: 0
ep_warmup_key_count: 11648028
ep_warmup_keys_time: 336295362
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 427304010
ep_warmup_value_count: 2206082

10.3.121.169
Authentication error for sasl

10.3.121.47
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 27054
ep_warmup_estimated_key_count: 11524700
ep_warmup_estimated_value_count: 11524700
ep_warmup_item_expired: 0
ep_warmup_key_count: 11524700
ep_warmup_keys_time: 165297428
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 243145155
ep_warmup_value_count: 2213186

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-823-rel.setup.exe.manifest.xml

Link to collect info file of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_06/9nodes_202-823_zero_rep_ratio_after_warmup_20130603-172501.tgz