[2014-06-17 04:18:26,323] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:18:36,341] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:18:46,358] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:18:56,374] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:06,389] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:16,408] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:26,425] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:36,627] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:46,657] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:19:56,674] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:06,691] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:16,708] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:26,724] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:36,740] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:46,757] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:20:56,774] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:21:06,798] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:21:16,818] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:21:26,835] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:21:36,950] - [rest_client:1208] INFO - rebalance percentage : 37.4880300958 % [2014-06-17 04:21:36,981] - [rest_client:1995] INFO - Latest logs from UI on 10.3.3.38: [2014-06-17 04:21:36,982] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-06-17T04:17:28.971Z', u'module': u'ns_vbucket_mover', u'tstamp': 1403003848971, u'type': u'info'} [2014-06-17 04:21:36,982] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 0, u'text': u'Started rebalancing bucket standard_bucket0', u'shortText': u'message', u'serverTime': u'2014-06-17T04:17:28.409Z', u'module': u'ns_rebalancer', u'tstamp': 1403003848409, u'type': u'info'} [2014-06-17 04:21:36,982] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.2.243','ns_1@10.3.3.39',\n 'ns_1@10.3.3.38'], EjectNodes = ['ns_1@10.3.2.239'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-06-17T04:17:28.268Z', u'module': u'ns_orchestrator', u'tstamp': 1403003848268, u'type': u'info'} [2014-06-17 04:21:36,983] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 1, u'text': u'Rebalance completed successfully.\n', u'shortText': u'message', u'serverTime': u'2014-06-17T04:15:08.424Z', u'module': u'ns_orchestrator', u'tstamp': 1403003708424, u'type': u'info'} [2014-06-17 04:21:36,984] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.39', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@10.3.3.39\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-06-17T04:13:39.461Z', u'module': u'ns_memcached', u'tstamp': 1403003619461, u'type': u'info'} [2014-06-17 04:21:36,985] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 0, u'text': u'Bucket "bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-06-17T04:13:37.784Z', u'module': u'ns_vbucket_mover', u'tstamp': 1403003617784, u'type': u'info'} [2014-06-17 04:21:36,985] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.38', u'code': 0, u'text': u'Started rebalancing bucket bucket0', u'shortText': u'message', u'serverTime': u'2014-06-17T04:13:34.705Z', u'module': u'ns_rebalancer', u'tstamp': 1403003614705, u'type': u'info'} [2014-06-17 04:21:36,986] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.2.239', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@10.3.2.239\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-06-17T04:13:28.430Z', u'module': u'ns_memcached', u'tstamp': 1403003608430, u'type': u'info'} [2014-06-17 04:21:36,986] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.2.243', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@10.3.2.243\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-06-17T04:13:27.929Z', u'module': u'ns_memcached', u'tstamp': 1403003607929, u'type': u'info'} [2014-06-17 04:21:36,986] - [rest_client:1996] ERROR - {u'node': u'ns_1@10.3.3.39', u'code': 0, u'text': u'Bucket "standard_bucket0" loaded on node \'ns_1@10.3.3.39\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-06-17T04:12:05.852Z', u'module': u'ns_memcached', u'tstamp': 1403003525852, u'type': u'info'} [('/usr/local/lib/python2.7/threading.py', 783, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/local/lib/python2.7/threading.py', 810, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 57, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 365, 'check', 'self.set_exception(RebalanceFailedException("seems like rebalance hangs. please check logs!"))'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Tue Jun 17 04:21:36 2014 [user:info,2014-06-17T4:26:18.127,ns_1@10.3.3.38:<0.997.0>:ns_orchestrator:handle_info:475]Rebalance stopped by user. [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:<0.1569.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.95698> [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:<0.10770.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.97078> [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:upr_consumer_conn-standard_bucket0-ns_1@10.3.2.243<0.10769.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.97066> [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:upr_consumer_conn-standard_bucket0-ns_1@10.3.3.39<0.1557.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.96518> [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:<0.9016.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.97012> [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:upr_consumer_conn-standard_bucket0-ns_1@10.3.2.239<0.9015.28>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.97008> [error_logger:error,2014-06-17T4:26:18.127,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: erlang:apply/2 pid: <0.13204.30> registered_name: [] exception exit: stopped in function ns_rebalancer:run_mover/7 (src/ns_rebalancer.erl, line 513) in call from ns_rebalancer:rebalance/6 (src/ns_rebalancer.erl, line 477) in call from ns_rebalancer:'-rebalance/5-fun-1-'/6 (src/ns_rebalancer.erl, line 435) in call from lists:foreach/2 (lists.erl, line 1323) in call from ns_rebalancer:rebalance/5 (src/ns_rebalancer.erl, line 387) ancestors: [<0.997.0>,mb_master_sup,mb_master,ns_server_sup, ns_server_cluster_sup,<0.56.0>] messages: [] links: [<0.13261.30>,<0.997.0>] dictionary: [{random_seed,{3688,3451,10969}}] trap_exit: false status: running heap_size: 121536 stack_size: 27 reductions: 3749432 neighbours: neighbour: [{pid,<0.13264.30>}, {registered_name,[]}, {initial_call,{erlang,apply,['Argument__1','Argument__2']}}, {current_function, {ns_rebalance_observer,docs_left_updater_loop,1}}, {ancestors, [<0.13261.30>,<0.13204.30>,<0.997.0>,mb_master_sup, mb_master,ns_server_sup,ns_server_cluster_sup, <0.56.0>]}, {messages,[]}, {links,[<0.13261.30>,<0.272.0>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,75113}, {stack_size,6}, {reductions,499247}] neighbour: [{pid,<0.13261.30>}, {registered_name,[]}, {initial_call,{ns_rebalance_observer,init,['Argument__1']}}, {current_function,{gen_server,loop,6}}, {ancestors,[<0.13204.30>,<0.997.0>,mb_master_sup,mb_master, ns_server_sup,ns_server_cluster_sup,<0.56.0>]}, {messages,[]}, {links,[<0.13262.30>,<0.13264.30>,<0.13204.30>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,46422}, {stack_size,9}, {reductions,1276703}] [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:capi_set_view_manager-standard_bucket0<0.32442.27>:capi_set_view_manager:handle_info:306]doing replicate_newnodes_docs [ns_server:debug,2014-06-17T4:26:18.127,ns_1@10.3.3.38:capi_set_view_manager-bucket0<0.32295.27>:capi_set_view_manager:handle_info:306]doing replicate_newnodes_docs [error_logger:error,2014-06-17T4:26:18.127,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.10770.28> terminating ** Last message in was {'EXIT',<0.10768.28>,nuke} ** When Server state == {state,#Port<0.97078>, {producer, "replication:ns_1@10.3.2.243->ns_1@10.3.3.38:standard_bucket0", 'ns_1@10.3.2.243',"standard_bucket0"}, <<>>,upr_producer_conn,[],#Port<0.97066>, <0.10769.28>} ** Reason for termination == ** nuke [ns_server:debug,2014-06-17T4:26:18.143,ns_1@10.3.3.38:<0.31056.30>:upr_proxy:nuke_connection:177]Nuke UPR connection "replication:ns_1@10.3.3.39->ns_1@10.3.3.38:standard_bucket0" type consumer on node 'ns_1@10.3.3.38' [ns_server:debug,2014-06-17T4:26:18.143,ns_1@10.3.3.38:<0.31057.30>:upr_proxy:nuke_connection:177]Nuke UPR connection "replication:ns_1@10.3.3.39->ns_1@10.3.3.38:standard_bucket0" type producer on node 'ns_1@10.3.3.39' [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: upr_proxy:init/1 pid: <0.10770.28> registered_name: [] exception exit: nuke in function gen_server:terminate/6 (gen_server.erl, line 744) ancestors: ['upr_replicator-standard_bucket0-ns_1@10.3.2.243', 'upr_sup-standard_bucket0', 'single_bucket_sup-standard_bucket0',<0.32440.27>] messages: [] links: [<0.10768.28>] dictionary: [] trap_exit: true status: running heap_size: 987 stack_size: 27 reductions: 1300718 neighbours: [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.1569.28> terminating ** Last message in was {'EXIT',<0.1556.28>,nuke} ** When Server state == {state,#Port<0.95698>, {producer, "replication:ns_1@10.3.3.39->ns_1@10.3.3.38:standard_bucket0", 'ns_1@10.3.3.39',"standard_bucket0"}, <<>>,upr_producer_conn,[],#Port<0.96518>, <0.1557.28>} ** Reason for termination == ** nuke [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: upr_proxy:init/1 pid: <0.1569.28> registered_name: [] exception exit: nuke in function gen_server:terminate/6 (gen_server.erl, line 744) ancestors: ['upr_replicator-standard_bucket0-ns_1@10.3.3.39', 'upr_sup-standard_bucket0', 'single_bucket_sup-standard_bucket0',<0.32440.27>] messages: [] links: [<0.1556.28>] dictionary: [] trap_exit: true status: running heap_size: 987 stack_size: 27 reductions: 2016774 neighbours: [ns_server:debug,2014-06-17T4:26:18.143,ns_1@10.3.3.38:ns_config_log<0.262.0>:ns_config_log:log_common:134]config change: counters -> [{'_vclock',[{<<"ba49a99f6e11c40537de373efb0d94be">>,{53,63570223578}}]}, {rebalance_stop,1}, {rebalance_start,26}, {rebalance_success,24}, {rebalance_fail,1}, {failover_node,1}] [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.9016.28> terminating ** Last message in was {'EXIT',<0.9014.28>,nuke} ** When Server state == {state,#Port<0.97012>, {producer, "replication:ns_1@10.3.2.239->ns_1@10.3.3.38:standard_bucket0", 'ns_1@10.3.2.239',"standard_bucket0"}, <<>>,upr_producer_conn,[],#Port<0.97008>, <0.9015.28>} ** Reason for termination == ** nuke [ns_server:debug,2014-06-17T4:26:18.143,ns_1@10.3.3.38:capi_set_view_manager-bucket0<0.32295.27>:capi_set_view_manager:handle_info:306]doing replicate_newnodes_docs [ns_server:debug,2014-06-17T4:26:18.143,ns_1@10.3.3.38:capi_set_view_manager-standard_bucket0<0.32442.27>:capi_set_view_manager:handle_info:306]doing replicate_newnodes_docs [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: upr_proxy:init/1 pid: <0.9016.28> registered_name: [] exception exit: nuke in function gen_server:terminate/6 (gen_server.erl, line 744) ancestors: ['upr_replicator-standard_bucket0-ns_1@10.3.2.239', 'upr_sup-standard_bucket0', 'single_bucket_sup-standard_bucket0',<0.32440.27>] messages: [] links: [<0.9014.28>] dictionary: [] trap_exit: true status: running heap_size: 987 stack_size: 27 reductions: 2711125 neighbours: [error_logger:error,2014-06-17T4:26:18.143,ns_1@10.3.3.38:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.10769.28> terminating ** Last message in was {'EXIT',<0.10768.28>,nuke} ** When Server state == {state,#Port<0.97066>, {consumer, "replication:ns_1@10.3.2.243->ns_1@10.3.3.38:standard_bucket0", 'ns_1@10.3.3.38',"standard_bucket0"}, <<>>,upr_consumer_conn, {state,idle, [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36, 37,38,39,40,41,42,43,44,45,46,47,48,49,50,51, 52,53,54,55,56,57,58,59,60,61,62,63,64,65,66, 67,68,69,70,71,72,73,74,75,76,77,78,79,80,81, 82,83,84,85,342,343,344,345,346,347,348,349, 350,351,352,353,354,355,356,357,358,359,360, 361,362,363,364,365,366,367,368,369,370,371, 372,373,374,375,376,377,378,379,380,381,382, 383,384,385,386,387,388,389,390,391,392,393, 394,395,396,397,398,399,400,401,402,403,404, 405,406,407,408,409,410,411,412,413,414,415, 416,417,418,419,420,421,422,423,424,425,426]}, #Port<0.97078>,<0.10770.28>} ** Reason for termination == ** nuke