[system test] [windows] data lost in warmup cluster test

Environment:

• 9 nodes windows server 2008 R2 64-bit with 8GB RAM and SSD storage

Cluster:

• create 7 nodes cluster with build 2.1.0-718
• create 2 buckets: default (3GB with replica index enable) and sasl (3GB with replica index disable)
• Load 20+ items to make active resident ratio down to 70%

Do system test without xdcr as in http://hub.internal.couchbase.com/confluence/display/QA/views+%28and+now+with+XDCR%29+tests

At the last test, restart couchbase server in all nodes in cluster
after finish the warmup, the number items in default bucket is short about 4500 over 22+ million items
access log in both buckets, default and sasl are corrupted (as in closed bug MB-8291)

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../../ini/sys9winssd.ini "/cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup "

10.3.3.214
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 107051
ep_warmup_estimated_key_count: 7656958
ep_warmup_estimated_value_count: 7656958
ep_warmup_item_expired: 0
ep_warmup_key_count: 7656958
ep_warmup_keys_time: 195675294
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 328602589
ep_warmup_value_count: 3198717

10.3.3.180
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 68588
ep_warmup_estimated_key_count: 7648495
ep_warmup_estimated_value_count: 7648495
ep_warmup_item_expired: 0
ep_warmup_key_count: 7648495
ep_warmup_keys_time: 252251931
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 428768618
ep_warmup_value_count: 3200717

10.3.121.243
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 45064
ep_warmup_estimated_key_count: 7525858
ep_warmup_estimated_value_count: 7525858
ep_warmup_item_expired: 0
ep_warmup_key_count: 7525858
ep_warmup_keys_time: 227503571
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 373085092
ep_warmup_value_count: 3232507

10.3.3.182
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 35631
ep_warmup_estimated_key_count: 6106847
ep_warmup_estimated_value_count: 6106847
ep_warmup_item_expired: 0
ep_warmup_key_count: 6106847
ep_warmup_keys_time: 138994714
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 266444557
ep_warmup_value_count: 3525618

10.3.3.181
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 41812
ep_warmup_estimated_key_count: 7643543
ep_warmup_estimated_value_count: 7643543
ep_warmup_item_expired: 0
ep_warmup_key_count: 7643543
ep_warmup_keys_time: 256951178
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 446990644
ep_warmup_value_count: 3201695

10.3.121.47
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 59645
ep_warmup_estimated_key_count: 7662467
ep_warmup_estimated_value_count: 7662467
ep_warmup_item_expired: 0
ep_warmup_key_count: 7662467
ep_warmup_keys_time: 211950348
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 390867428
ep_warmup_value_count: 3198065

Thuans-MacBook-Pro:testrunner thuan$ python scripts/ssh.py -i ../../ini/sys9winssd.ini "/cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup -b sasl -p password"

10.3.3.214
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 50987
ep_warmup_estimated_key_count: 8670319
ep_warmup_estimated_value_count: 8670319
ep_warmup_item_expired: 0
ep_warmup_key_count: 8670319
ep_warmup_keys_time: 244286847
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 404109716
ep_warmup_value_count: 3139440

10.3.121.243
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 62536
ep_warmup_estimated_key_count: 8612707
ep_warmup_estimated_value_count: 8612707
ep_warmup_item_expired: 0
ep_warmup_key_count: 8612707
ep_warmup_keys_time: 247823597
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 349146015
ep_warmup_value_count: 3153808

10.3.3.180
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 120548
ep_warmup_estimated_key_count: 8671127
ep_warmup_estimated_value_count: 8671127
ep_warmup_item_expired: 0
ep_warmup_key_count: 8671127
ep_warmup_keys_time: 267401160
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 430655111
ep_warmup_value_count: 3139018

10.3.3.182
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 41160
ep_warmup_estimated_key_count: 8621363
ep_warmup_estimated_value_count: 8621363
ep_warmup_item_expired: 0
ep_warmup_key_count: 8621363
ep_warmup_keys_time: 202099375
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 328563859
ep_warmup_value_count: 3153141

10.3.121.47
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 65453
ep_warmup_estimated_key_count: 8674478
ep_warmup_estimated_value_count: 8674478
ep_warmup_item_expired: 0
ep_warmup_key_count: 8674478
ep_warmup_keys_time: 214569068
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 352733562
ep_warmup_value_count: 3138872

10.3.3.181
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 27833
ep_warmup_estimated_key_count: 8670917
ep_warmup_estimated_value_count: 8670917
ep_warmup_item_expired: 3
ep_warmup_key_count: 8670917
ep_warmup_keys_time: 256269088
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 431522032
ep_warmup_value_count: 3139881

In log page, I also see memcached exited with status 38

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 38. Restarting. Messages: Mon Jun 17 14:05:39.267281 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_788 - Reset vbucket 24 was completed succecssfully.
Mon Jun 17 14:05:39.267281 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_788 - Reset vbucket 508 was completed succecssfully.
Mon Jun 17 14:05:39.279000 Pacific Daylight Time 3: (default) TAP (Consu

Link to manifest file of the build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.1.0-718-rel.setup.exe.manifest.xml

Link to collect info file of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_1_0/2013-06/6nodes_210-718_data_lost_cluster_warmup_20130617-152529.tgz

Link to all access logs of all nodes of default bucket https://s3.amazonaws.com/packages.couchbase/access_logs/2013_06/6nodes_210-718_access-log-corrupted_20130617-155352.tgz