http://qa.hq.northscale.net/job/windows_rebalance_regression_P1/42/consoleFull
./testrunner -i /tmp/windows_rebalance_regression_P1.ini get-cbcollect-info=True,EXCLUDE_GROUP=NOT_WIND -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=2,keys-count=1000000
[2013-06-07 15:20:12,829] - [rest_client:1031] INFO - rebalance percentage : 81.5092339512 %
[2013-06-07 15:20:14,995] - [rest_client:1031] INFO - rebalance percentage : 81.5463715807 %
[2013-06-07 15:20:18,311] - [rest_client:1014] ERROR -
{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}
- rebalance failed
[2013-06-07 15:20:18,311] - [rest_client:1015] INFO - Latest logs from UI:
[2013-06-07 15:20:22,593] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 2, u'text': u"Rebalance exited with reason {{{badmatch,\n [
{<17958.19413.3>,noproc},\n {<17853.3469.5>,normal},\n {<17852.632.5>,normal}]},\n
[{misc,sync_shutdown_many_i_am_trapping_exits,\n 1},\n {misc,try_with_maybe_ignorant_after,2},\n {gen_server,terminate,6},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n
[<0.25218.18>,\n {shutdown_replicator,'ns_1@10.3.2.136'},\n infinity]}}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1370644282812, u'type': u'info'}
[2013-06-07 15:20:22,593] -
[rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u"<0.25209.18> exited with {{{badmatch,[{<17958.19413.3>,noproc}
,\n
{<17853.3469.5>,normal}
,\n
{<17852.632.5>,normal}
]},\n [
{misc,sync_shutdown_many_i_am_trapping_exits,1}
,\n
{misc,try_with_maybe_ignorant_after,2}
,\n
{gen_server,terminate,6}
,\n
{proc_lib,init_p_do_apply,3}
]},\n {gen_server,call,\n [<0.25218.18>,\n
{shutdown_replicator,'ns_1@10.3.2.136'}
,\n infinity]}}", u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370644282562, u'type': u'critical'}
[2013-06-07 15:20:22,594] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.131\' in 2 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370644281702, u'type': u'info'}
[2013-06-07 15:20:22,594] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.2.131\' in 1 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370644281234, u'type': u'info'}
[2013-06-07 15:20:22,594] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@10.3.2.131\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370644280140, u'type': u'info'}
[2013-06-07 15:20:22,595] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.131' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370644277265, u'type': u'info'}
[2013-06-07 15:20:22,596] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255. Restarting. Messages: Fri Jun 07 15:31:01.961750 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 428\nFri Jun 07 15:31:01.980304 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Schedule the backfill for vbucket 428\nFri Jun 07 15:31:01.980304 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:01.981281 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:01.982257 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 428\nFri Jun 07 15:31:02.014484 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 428, \nFri Jun 07 15:31:02.014484 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 428\nFri Jun 07 15:31:02.234210 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:02.234210 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:02.410968 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.133\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:02.419757 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - Sending TAP_VBUCKET_SET with vbucket 568 and state "pending"\nFri Jun 07 15:31:02.483234 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Schedule the backfill for vbucket 272\nFri Jun 07 15:31:02.491046 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:02.491046 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:02.498859 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 272\nFri Jun 07 15:31:02.493976 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.133\' - Connection is closed by force\nFri Jun 07 15:31:02.502765 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - VBucket <568> is going dead to complete vbucket takeover\nFri Jun 07 15:31:02.504718 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9658"\nFri Jun 07 15:31:02.505695 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.133\' - Clear the tap queues by force\nFri Jun 07 15:31:02.530109 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - Sending TAP_VBUCKET_SET with vbucket 568 and state "active"\nFri Jun 07 15:31:02.534015 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Backfill is completed with VBuckets 272, \nFri Jun 07 15:31:02.587726 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 272\nFri Jun 07 15:31:02.535968 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Schedule the backfill for vbucket 272\nFri Jun 07 15:31:02.588703 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:02.590656 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:02.596515 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 272\nFri Jun 07 15:31:02.589679 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Schedule the backfill for vbucket 272\nFri Jun 07 15:31:02.603351 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:02.613117 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:02.613117 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 272\nFri Jun 07 15:31:02.606281 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_568>\nFri Jun 07 15:31:02.615070 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - disconnected\nFri Jun 07 15:31:02.679523 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 272, \nFri Jun 07 15:31:02.679523 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 272\nFri Jun 07 15:31:02.681476 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Backfill is completed with VBuckets 272, \nFri Jun 07 15:31:02.684406 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 272\nFri Jun 07 15:31:02.688312 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_568"\nFri Jun 07 15:31:02.692218 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.132\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:02.694171 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_568 - Clear the tap queues by force\nFri Jun 07 15:31:02.713703 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.136\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:02.846515 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.132\' - Connection is closed by force\nFri Jun 07 15:31:02.847492 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.136\' - Connection is closed by force\nFri Jun 07 15:31:03.018390 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Schedule the backfill for vbucket 958\nFri Jun 07 15:31:03.018390 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:03.058429 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:03.060382 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 958\nFri Jun 07 15:31:03.022296 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:03.062335 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:03.064289 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - Sending TAP_VBUCKET_SET with vbucket 428 and state "pending"\nFri Jun 07 15:31:03.065265 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9660"\nFri Jun 07 15:31:03.065265 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.136\' - Clear the tap queues by force\nFri Jun 07 15:31:03.066242 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9659"\nFri Jun 07 15:31:03.066242 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_568_\'ns_1@10.3.2.132\' - Clear the tap queues by force\nFri Jun 07 15:31:03.076007 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - VBucket <428> is going dead to complete vbucket takeover\nFri Jun 07 15:31:03.085773 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Backfill is completed with VBuckets 958, \nFri Jun 07 15:31:03.085773 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.135\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 958\nFri Jun 07 15:31:03.106281 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - Sending TAP_VBUCKET_SET with vbucket 428 and state "active"\nFri Jun 07 15:31:03.142414 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Schedule the backfill for vbucket 958\nFri Jun 07 15:31:03.163898 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:03.170734 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_428>\nFri Jun 07 15:31:03.173664 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:03.173664 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 958\nFri Jun 07 15:31:03.173664 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - disconnected\nFri Jun 07 15:31:03.192218 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:03.206867 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Connection is closed by force\nFri Jun 07 15:31:03.308429 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 958, \nFri Jun 07 15:31:03.308429 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 958\nFri Jun 07 15:31:03.311359 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9661"\nFri Jun 07 15:31:03.311359 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_428_\'ns_1@10.3.2.134\' - Clear the tap queues by force\nFri Jun 07 15:31:03.330890 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_428"\nFri Jun 07 15:31:03.331867 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_428 - Clear the tap queues by force\nFri Jun 07 15:31:03.525226 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Schedule the backfill for vbucket 958\nFri Jun 07 15:31:03.525226 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:03.532062 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:03.533039 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 958\nFri Jun 07 15:31:03.619953 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Backfill is completed with VBuckets 958, \nFri Jun 07 15:31:03.619953 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_958_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 958\nFri Jun 07 15:31:05.713703 Pacific Daylight Time 3: (bucket-0) Deletion of vbucket 568 was completed.\nFri Jun 07 15:31:05.759601 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:05.874835 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Connection is closed by force\nFri Jun 07 15:31:06.182453 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 15:31:06.182453 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 15:31:06.184406 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - Sending TAP_VBUCKET_SET with vbucket 272 and state "pending"\nFri Jun 07 15:31:06.187335 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9662"\nFri Jun 07 15:31:06.187335 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.132\' - Clear the tap queues by force\nFri Jun 07 15:31:06.192218 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - VBucket <272> is going dead to complete vbucket takeover\nFri Jun 07 15:31:06.196125 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - Sending TAP_VBUCKET_SET with vbucket 272 and state "active"\nFri Jun 07 15:31:06.201984 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_272>\nFri Jun 07 15:31:06.201984 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - disconnected\nFri Jun 07 15:31:06.220539 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:06.223468 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - disconnected, keep alive for 300 seconds\nFri Jun 07 15:31:06.240070 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Connection is closed by force\nFri Jun 07 15:31:06.241046 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Connection is closed by force\nFri Jun 07 15:31:07.408039 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9663"\nFri Jun 07 15:31:07.408039 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.134\' - Clear the tap queues by force\nFri Jun 07 15:31:07.408039 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_9664"\nFri Jun 07 15:31:07.411945 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_272_\'ns_1@10.3.2.136\' - Clear the tap queues by force\nFri Jun 07 15:31:07.411945 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_272"\nFri Jun 07 15:31:07.412921 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_272 - Clear the tap queues by force\nFri Jun 07 15:31:08.084796 Pacific Daylight Time 3: (bucket-2) Total memory in memoryDeallocated() >= GIGANTOR !!! Disable the memory tracker...', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1370644276827, u'type': u'info'}
[2013-06-07 15:20:22,596] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Bucket "bucket-0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370642631609, u'type': u'info'}
[2013-06-07 15:20:22,597] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.132', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.132\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370642630249, u'type': u'info'}
[2013-06-07 15:20:22,597] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.133', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.133\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370642629883, u'type': u'info'}
don't have crash on windows
Jin, could you take a look at the issue
Bug
Blocker
