http://qa.hq.northscale.net/job/windows_rebalance_regression_P1/42/consoleFull
./testrunner -i /tmp/windows_rebalance_regression_P1.ini get-cbcollect-info=True,EXCLUDE_GROUP=NOT_WIND -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=1,keys-count=1000000,swap-orchestrator=False
[2013-06-07 18:15:53,122] - [rest_client:1031] INFO - rebalance percentage : 82.7724866811 %
[2013-06-07 18:15:55,612] - [rest_client:1031] INFO - rebalance percentage : 82.8096243105 %
[2013-06-07 18:15:57,714] - [rest_client:1014] ERROR -
{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}
- rebalance failed
[2013-06-07 18:15:57,715] - [rest_client:1015] INFO - Latest logs from UI:
[2013-06-07 18:15:58,365] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 2, u'text': u'Rebalance exited with reason {badmatch,\n {memcached_error,not_my_vbucket,\n <<"VBucket 955 not in active state!!!">>,\n [
{janitor_agent,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n
[{\'janitor_agent-bucket-0\',\'ns_1@10.3.2.131\'},\n {if_rebalance,<0.18603.24>,\n {get_replication_persistence_checkpoint_id,\n 955}},\n infinity]}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1370654824812, u'type': u'info'}
[2013-06-07 18:15:58,365] -
[rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'<0.21946.25> exited with {
badmatch,\n {memcached_error,not_my_vbucket,\n <<"VBucket 955 not in active state!!!">>,\n [{janitor_agent,handle_call,3}
,\n
{gen_server,handle_msg,5}
,\n
{proc_lib,init_p_do_apply,3}
]},\n {gen_server,call,\n [
{\'janitor_agent-bucket-0\',\'ns_1@10.3.2.131\'}
,\n {if_rebalance,<0.18603.24>,\n {get_replication_persistence_checkpoint_id,\n 955}},\n infinity]}}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370654824671, u'type': u'critical'}
[2013-06-07 18:15:58,366] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@10.3.2.131\' in 1 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370654822218, u'type': u'info'}
[2013-06-07 18:15:58,366] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.2.131\' in 1 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370654821718, u'type': u'info'}
[2013-06-07 18:15:58,366] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.131\' in 1 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370654821718, u'type': u'info'}
[2013-06-07 18:15:58,367] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.131' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370654819249, u'type': u'info'}
[2013-06-07 18:15:58,367] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.131' disconnected: badmatch,\n {error,\n closed,\n [
{mc_client_binary,\n cmd_binary_vocal,\n 6}
,\n
{mc_client_binary,\n select_bucket,\n 2}
,\n
{ns_memcached,\n ensure_bucket,\n 2}
,\n
{ns_memcached,\n handle_info,\n 2}
,\n
{gen_server,\n handle_msg,\n 5}
,\n
{ns_memcached,\n init,1}
,\n
{gen_server,\n init_it,\n 6}
,\n
{proc_lib,\n init_p_do_apply,\n 3}
]}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370654818890, u'type': u'info'}
[2013-06-07 18:15:58,367] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 253. Restarting. Messages: Fri Jun 07 18:26:47.203937 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:47.201984 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:47.204914 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6350"\nFri Jun 07 18:26:47.205890 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 247\nFri Jun 07 18:26:47.205890 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - Sending TAP_VBUCKET_SET with vbucket 556 and state "pending"\nFri Jun 07 18:26:47.222492 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Schedule the backfill for vbucket 247\nFri Jun 07 18:26:47.243976 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.134\' - Clear the tap queues by force\nFri Jun 07 18:26:47.244953 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:47.244953 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:47.244953 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 247, \nFri Jun 07 18:26:47.245929 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 247\nFri Jun 07 18:26:47.245929 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 247\nFri Jun 07 18:26:47.246906 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - VBucket <556> is going dead to complete vbucket takeover\nFri Jun 07 18:26:47.250812 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Backfill is completed with VBuckets 247, \nFri Jun 07 18:26:47.250812 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Backfill is completed with VBuckets 247, \nFri Jun 07 18:26:47.250812 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 247\nFri Jun 07 18:26:47.251789 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - Sending TAP_VBUCKET_SET with vbucket 556 and state "active"\nFri Jun 07 18:26:47.251789 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 247\nFri Jun 07 18:26:47.255695 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_556>\nFri Jun 07 18:26:47.255695 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - disconnected\nFri Jun 07 18:26:47.288898 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_556"\nFri Jun 07 18:26:47.288898 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_556 - Clear the tap queues by force\nFri Jun 07 18:26:47.297687 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.132\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:47.301593 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.136\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:47.325031 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.132\' - Connection is closed by force\nFri Jun 07 18:26:47.326007 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.136\' - Connection is closed by force\nFri Jun 07 18:26:47.695148 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Schedule the backfill for vbucket 434\nFri Jun 07 18:26:47.695148 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:47.703937 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:47.707843 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 434\nFri Jun 07 18:26:47.715656 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6351"\nFri Jun 07 18:26:47.715656 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.132\' - Clear the tap queues by force\nFri Jun 07 18:26:47.716632 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6352"\nFri Jun 07 18:26:47.724445 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_556_\'ns_1@10.3.2.136\' - Clear the tap queues by force\nFri Jun 07 18:26:47.730304 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Backfill is completed with VBuckets 434, \nFri Jun 07 18:26:47.731281 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 434\nFri Jun 07 18:26:47.742023 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Schedule the backfill for vbucket 434\nFri Jun 07 18:26:47.742023 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:47.745929 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:47.746906 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 434\nFri Jun 07 18:26:47.750812 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 434, \nFri Jun 07 18:26:47.750812 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 434\nFri Jun 07 18:26:47.831867 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:47.856281 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Connection is closed by force\nFri Jun 07 18:26:48.089679 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:48.089679 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:48.090656 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - Sending TAP_VBUCKET_SET with vbucket 434 and state "pending"\nFri Jun 07 18:26:48.090656 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6353"\nFri Jun 07 18:26:48.097492 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - VBucket <434> is going dead to complete vbucket takeover\nFri Jun 07 18:26:48.099445 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.133\' - Clear the tap queues by force\nFri Jun 07 18:26:48.130695 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - Sending TAP_VBUCKET_SET with vbucket 434 and state "active"\nFri Jun 07 18:26:48.174640 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_434>\nFri Jun 07 18:26:48.174640 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - disconnected\nFri Jun 07 18:26:48.184406 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:48.193195 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Connection is closed by force\nFri Jun 07 18:26:49.361164 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6354"\nFri Jun 07 18:26:49.361164 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_434_\'ns_1@10.3.2.134\' - Clear the tap queues by force\nFri Jun 07 18:26:49.362140 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_434"\nFri Jun 07 18:26:49.363117 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_434 - Clear the tap queues by force\nFri Jun 07 18:26:50.293781 Pacific Daylight Time 3: (bucket-2) Total memory in memoryDeallocated() >= GIGANTOR !!! Disable the memory tracker...\nFri Jun 07 18:26:55.213703 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Connection is closed by force\nFri Jun 07 18:26:55.214679 Pacific Daylight Time 3: (bucket-0) Deletion of vbucket 556 was completed.\nFri Jun 07 18:26:55.214679 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - disconnected\nFri Jun 07 18:26:55.230304 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6355"\nFri Jun 07 18:26:55.233234 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.132\' - Clear the tap queues by force\nFri Jun 07 18:26:55.894367 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:55.894367 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:55.901203 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - Sending TAP_VBUCKET_SET with vbucket 247 and state "pending"\nFri Jun 07 18:26:55.920734 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - VBucket <247> is going dead to complete vbucket takeover\nFri Jun 07 18:26:55.935382 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - Sending TAP_VBUCKET_SET with vbucket 247 and state "active"\nFri Jun 07 18:26:55.947101 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_247>\nFri Jun 07 18:26:55.947101 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - disconnected\nFri Jun 07 18:26:55.975421 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.134\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:55.975421 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - disconnected, keep alive for 300 seconds\nFri Jun 07 18:26:55.985187 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.134\' - Connection is closed by force\nFri Jun 07 18:26:55.986164 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Connection is closed by force\nFri Jun 07 18:26:57.184406 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Schedule the backfill for vbucket 955\nFri Jun 07 18:26:57.184406 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:57.228351 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:57.243976 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 955\nFri Jun 07 18:26:57.257648 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Schedule the backfill for vbucket 955\nFri Jun 07 18:26:57.277179 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:57.348468 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:57.401203 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 955\nFri Jun 07 18:26:57.257648 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6356"\nFri Jun 07 18:26:57.490070 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.134\' - Clear the tap queues by force\nFri Jun 07 18:26:57.496906 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:anon_6357"\nFri Jun 07 18:26:57.552570 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Schedule the backfill for vbucket 955\nFri Jun 07 18:26:57.561359 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nFri Jun 07 18:26:57.564289 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nFri Jun 07 18:26:57.564289 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 955\nFri Jun 07 18:26:57.565265 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_247_\'ns_1@10.3.2.136\' - Clear the tap queues by force\nFri Jun 07 18:26:57.571125 Pacific Daylight Time 3: (bucket-0) Clean up "eq_tapq:rebalance_247"\nFri Jun 07 18:26:57.583820 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_247 - Clear the tap queues by force\nFri Jun 07 18:26:57.651203 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Backfill is completed with VBuckets 955, \nFri Jun 07 18:26:57.657062 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.136\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 955\nFri Jun 07 18:26:57.657062 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Backfill is completed with VBuckets 955, \nFri Jun 07 18:26:57.694171 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.133\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 955\nFri Jun 07 18:26:57.657062 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Backfill is completed with VBuckets 955, \nFri Jun 07 18:26:57.799640 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_955_\'ns_1@10.3.2.134\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 955', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1370654818859, u'type': u'info'}
[2013-06-07 18:15:58,370] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Bucket "bucket-0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370653095640, u'type': u'info'}
[2013-06-07 18:15:58,370] - [rest_client:1016] ERROR -
{u'node': u'ns_1@10.3.2.133', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.133\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370653094305, u'type': u'info'}
Jin, as in MB-8450 , I don't have any crash files on windows
Bug
Major
