Rebalance exited with reason {{bulk_set_vbucket_state_failed when create many sasl streaming connections

http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions-P1/219/consoleFull
/testrunner -i /tmp/rebalance_regression.ini wait_timeout=100,get-cbcollect-info=True -t connectionstests.ConnectionTests.create_connections_test,servers_in=1,BUGS=MB-7940_timeout_error-having_about_100_sasl_streaming_connection

we have opened bug MB-7940
but we don't have rebalance failed for the test before

Alk, maybe here is the same reason as MB-7940

[2013-06-08 23:55:13,223] - [rest_client:295] INFO - Opening sasl streaming connection for bucket default
[2013-06-08 23:55:13,380] - [rest_client:295] INFO - Opening sasl streaming connection for bucket default
[2013-06-08 23:55:15,096] - [rest_client:1014] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2013-06-08 23:55:15,096] - [rest_client:1015] INFO - Latest logs from UI:
[2013-06-08 23:55:15,243] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 2, u'text': u"Rebalance exited with reason bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.121.93',\n {'EXIT',\n {{{badmatch,{error,closed,\n {gen_server,call,\n ['ns_memcached-default',\n

{set_vbucket,1011,active}

,\n 180000]}},\n {gen_server,call,\n [

{'janitor_agent-default',\n 'ns_1@10.3.121.93'}

,\n {if_rebalance,<0.9328.197>,\n {update_vbucket_state,1012,replica,\n undefined,'ns_1@10.3.121.98'}},\n infinity]}}}}]},\n [

{janitor_agent,bulk_set_vbucket_state,4}

,\n

{ns_vbucket_mover,\n update_replication_post_move,3}

,\n

{ns_vbucket_mover,on_move_done,2}

,\n

{gen_server,handle_msg,5}

,\n

{proc_lib,init_p_do_apply,3}

]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1370760939413, u'type': u'info'}
[2013-06-08 23:55:15,243] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.121.93\' in 0 seconds. (repeated 1 times)', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370760935505, u'type': u'info'}

[2013-06-08 23:55:15,244] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 137. Restarting. Messages: Sat Jun 8 23:55:11.357901 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1017 - Sending TAP_VBUCKET_SET with vbucket 1017 and state "active"\nSat Jun 8 23:55:11.364017 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1017>\nSat Jun 8 23:55:11.364127 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1017 - disconnected\nSat Jun 8 23:55:11.371822 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1016_\'ns_1@10.3.121.98\' - disconnected, keep alive for 300 seconds\nSat Jun 8 23:55:11.466242 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Schedule the backfill for vbucket 1015\nSat Jun 8 23:55:11.466389 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:11.466464 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:11.466537 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 1015\nSat Jun 8 23:55:11.466708 PDT 3: (default) Clean up "eq_tapq:rebalance_1017"\nSat Jun 8 23:55:11.466737 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1017 - Clear the tap queues by force\nSat Jun 8 23:55:11.468760 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Backfill is completed with VBuckets 1015, \nSat Jun 8 23:55:11.468845 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 1015\nSat Jun 8 23:55:11.475665 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1016_\'ns_1@10.3.121.98\' - Connection is closed by force\nSat Jun 8 23:55:12.453873 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:12.453919 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:12.453930 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - Sending TAP_VBUCKET_SET with vbucket 1016 and state "pending"\nSat Jun 8 23:55:12.454167 PDT 3: (default) Clean up "eq_tapq:anon_21739"\nSat Jun 8 23:55:12.454197 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1016_\'ns_1@10.3.121.98\' - Clear the tap queues by force\nSat Jun 8 23:55:12.513386 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - VBucket <1016> is going dead to complete vbucket takeover\nSat Jun 8 23:55:12.567943 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - Sending TAP_VBUCKET_SET with vbucket 1016 and state "active"\nSat Jun 8 23:55:12.602961 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1016>\nSat Jun 8 23:55:12.603048 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - disconnected\nSat Jun 8 23:55:12.710242 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - disconnected, keep alive for 300 seconds\nSat Jun 8 23:55:12.758070 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Connection is closed by force\nSat Jun 8 23:55:13.102913 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:13.102953 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:13.102965 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - Sending TAP_VBUCKET_SET with vbucket 1015 and state "pending"\nSat Jun 8 23:55:13.103398 PDT 3: (default) Clean up "eq_tapq:anon_21740"\nSat Jun 8 23:55:13.103435 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1015_\'ns_1@10.3.121.98\' - Clear the tap queues by force\nSat Jun 8 23:55:13.103442 PDT 3: (default) Clean up "eq_tapq:rebalance_1016"\nSat Jun 8 23:55:13.103456 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1016 - Clear the tap queues by force\nSat Jun 8 23:55:13.139716 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - VBucket <1015> is going dead to complete vbucket takeover\nSat Jun 8 23:55:13.145783 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - Sending TAP_VBUCKET_SET with vbucket 1015 and state "active"\nSat Jun 8 23:55:13.172026 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1015>\nSat Jun 8 23:55:13.172107 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - disconnected\nSat Jun 8 23:55:14.173102 PDT 3: (default) Clean up "eq_tapq:rebalance_1015"\nSat Jun 8 23:55:14.173156 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1015 - Clear the tap queues by force\nSat Jun 8 23:55:18.056432 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Schedule the backfill for vbucket 1014\nSat Jun 8 23:55:18.056658 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:18.056759 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:18.056898 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 1014\nSat Jun 8 23:55:18.059816 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Backfill is completed with VBuckets 1014, \nSat Jun 8 23:55:18.059915 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 1014\nSat Jun 8 23:55:25.006137 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - disconnected, keep alive for 300 seconds\nSat Jun 8 23:55:25.051986 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Connection is closed by force\nSat Jun 8 23:55:25.064850 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Schedule the backfill for vbucket 1013\nSat Jun 8 23:55:25.065014 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:25.065439 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:25.065523 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 1013\nSat Jun 8 23:55:25.065657 PDT 3: (default) Clean up "eq_tapq:anon_21741"\nSat Jun 8 23:55:25.065756 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1014_\'ns_1@10.3.121.98\' - Clear the tap queues by force\nSat Jun 8 23:55:25.069075 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Backfill is completed with VBuckets 1013, \nSat Jun 8 23:55:25.069214 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 1013\nSat Jun 8 23:55:25.383328 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Schedule the backfill for vbucket 1012\nSat Jun 8 23:55:25.383518 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:25.383627 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:25.383700 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 1012\nSat Jun 8 23:55:25.386544 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Backfill is completed with VBuckets 1012, \nSat Jun 8 23:55:25.386648 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 1012\nSat Jun 8 23:55:25.451250 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - disconnected, keep alive for 300 seconds\nSat Jun 8 23:55:25.562569 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Connection is closed by force\nSat Jun 8 23:55:25.582259 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:25.582388 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:25.582473 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - Sending TAP_VBUCKET_SET with vbucket 1014 and state "pending"\nSat Jun 8 23:55:25.582845 PDT 3: (default) Clean up "eq_tapq:anon_21742"\nSat Jun 8 23:55:25.582876 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1013_\'ns_1@10.3.121.98\' - Clear the tap queues by force\nSat Jun 8 23:55:25.665622 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - VBucket <1014> is going dead to complete vbucket takeover\nSat Jun 8 23:55:25.878632 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - Sending TAP_VBUCKET_SET with vbucket 1014 and state "active"\nSat Jun 8 23:55:25.946376 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1014>\nSat Jun 8 23:55:25.946506 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - disconnected\nSat Jun 8 23:55:25.946870 PDT 3: (default) Clean up "eq_tapq:rebalance_1014"\nSat Jun 8 23:55:25.946945 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1014 - Clear the tap queues by force\nSat Jun 8 23:55:26.112150 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - disconnected, keep alive for 300 seconds\nSat Jun 8 23:55:26.213123 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Connection is closed by force\nSat Jun 8 23:55:26.441857 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Schedule the backfill for vbucket 1011\nSat Jun 8 23:55:26.441937 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:26.441962 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:26.441978 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 1011\nSat Jun 8 23:55:26.442229 PDT 3: (default) Clean up "eq_tapq:anon_21743"\nSat Jun 8 23:55:26.442251 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1012_\'ns_1@10.3.121.98\' - Clear the tap queues by force\nSat Jun 8 23:55:26.445471 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Backfill is completed with VBuckets 1011, \nSat Jun 8 23:55:26.445498 PDT 3: (default) TAP (Producer) eq_tapq:replication_building_1011_\'ns_1@10.3.121.98\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 1011\nSat Jun 8 23:55:27.267740 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:27.267772 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:27.267796 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - Sending TAP_VBUCKET_SET with vbucket 1013 and state "pending"\nSat Jun 8 23:55:27.400970 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 23:55:27.401018 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 23:55:27.401031 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - Sending TAP_VBUCKET_SET with vbucket 1012 and state "pending"\nSat Jun 8 23:55:27.566099 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - VBucket <1013> is going dead to complete vbucket takeover\nSat Jun 8 23:55:27.570369 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - VBucket <1012> is going dead to complete vbucket takeover\nSat Jun 8 23:55:27.680632 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - Sending TAP_VBUCKET_SET with vbucket 1012 and state "active"\nSat Jun 8 23:55:27.686073 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1012>\nSat Jun 8 23:55:27.686215 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - disconnected\nSat Jun 8 23:55:27.686576 PDT 3: (default) Clean up "eq_tapq:rebalance_1012"\nSat Jun 8 23:55:27.686608 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1012 - Clear the tap queues by force\nSat Jun 8 23:55:27.687901 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - Sending TAP_VBUCKET_SET with vbucket 1013 and state "active"\nSat Jun 8 23:55:27.760909 PDT 3: (default) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_1013>\nSat Jun 8 23:55:27.761001 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - disconnected\nSat Jun 8 23:55:28.762043 PDT 3: (default) Clean up "eq_tapq:rebalance_1013"\nSat Jun 8 23:55:28.762097 PDT 3: (default) TAP (Producer) eq_tapq:rebalance_1013 - Clear the tap queues by force', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1370760930318, u'type': u'info'}

[2013-06-08 23:55:15,247] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.121.93' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370760930145, u'type': u'info'}
[2013-06-08 23:55:15,247] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370760902703, u'type': u'info'}

[2013-06-08 23:55:15,248] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.98', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.121.98\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370760901253, u'type': u'info'}

[2013-06-08 23:55:15,249] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1370760901134, u'type': u'info'}

[2013-06-08 23:55:15,250] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.121.93','ns_1@10.3.121.98'], EjectNodes = []\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1370760900879, u'type': u'info'}

[2013-06-08 23:55:15,251] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.98', u'code': 3, u'text': u'Node ns_1@10.3.121.98 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1370760900681, u'type': u'info'}

[2013-06-08 23:55:15,251] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.98', u'code': 1, u'text': u"Couchbase Server has started on web port 8091 on node 'ns_1@10.3.121.98'.", u'shortText': u'web start ok', u'module': u'menelaus_sup', u'tstamp': 1370760900644, u'type': u'info'}

[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 56, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 305, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Sat Jun 8 23:55:15 2013
[2013-06-08 23:55:15,453] - [rest_client:295] INFO - Opening sasl streaming connection for bucket default
[2013-06-08 23:55:17,385] - [rest_client:295] INFO - Opening sasl streaming connection for bucket default
[2013-06-08 23:55:18,957] - [rest_client:295] INFO - Opening sasl streaming connection for bucket default