[system test] Rebalance stopped by janitor /Resetting rebalance status since it's not really running (couchdb:error Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}})

build-1956:
orange cluster, 2 buckets * ~40M keys :
1:10.6.2.37
2:10.6.2.38
3:10.6.2.39
4:10.6.2.40
5:10.6.2.42
6:10.6.2.43
7:10.6.2.44
8:10.6.2.45

steps:
1.add a node and rebalance (RB-1) -passed
2. swap the orchestrator with a new node (RB-2) - passed

cluster was about a day without any action( no data loading, rebalance or something else)

3. swap rebalance non orchestrator node (RB-3) ( returned back master node from step 2)

Starting rebalance, KeepNodes = ['ns_1@10.6.2.38','ns_1@10.6.2.39',
'ns_1@10.6.2.40','ns_1@10.6.2.42',
'ns_1@10.6.2.43','ns_1@10.6.2.44',
'ns_1@10.6.2.37'], EjectNodes = ['ns_1@10.6.2.45']

Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.6.2.37 15:34:42 - Mon Nov 26, 2012
killing myself due to exit signal: {'EXIT',<0.29220.5201>,
badmatch,{error,closed,
[

{ebucketmigrator_srv, upstream_sender_loop,1}

]}}

[ns_server:debug,2012-11-26T4:34:41.836,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.27467.5201>
[couchdb:error,2012-11-26T4:34:41.837,ns_1@10.6.2.37:<0.29176.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,

{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:41.837,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
[couchdb:error,2012-11-26T4:34:41.863,ns_1@10.6.2.37:<0.29177.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}

}}

Stacktrace: [

{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:error,2012-11-26T4:34:41.866,ns_1@10.6.2.37:<0.29178.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2}

,

{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:error,2012-11-26T4:34:41.879,ns_1@10.6.2.37:<0.29179.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3}

,

{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:41.905,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
[{93,38},{94,40}]
[couchdb:error,2012-11-26T4:34:41.914,ns_1@10.6.2.37:<0.29180.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2}

,

{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:41.914,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
[couchdb:error,2012-11-26T4:34:41.921,ns_1@10.6.2.37:<0.29182.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6}

,

{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:41.923,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
[ns_server:info,2012-11-26T4:34:41.927,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
[ns_server:debug,2012-11-26T4:34:41.931,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
[ns_server:debug,2012-11-26T4:34:41.934,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
[ns_server:debug,2012-11-26T4:34:41.937,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:41.938,ns_1@10.6.2.37:<0.29266.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:41.941,ns_1@10.6.2.37:<0.29266.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[rebalance:info,2012-11-26T4:34:41.941,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:41.950,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
[ns_server:debug,2012-11-26T4:34:41.954,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
[ns_server:debug,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
[couchdb:error,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.29183.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5}

,

{proc_lib,init_p_do_apply,3}]
[ns_server:debug,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.27467.5201>
[ns_server:debug,2012-11-26T4:34:41.968,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
[ns_server:info,2012-11-26T4:34:41.970,ns_1@10.6.2.37:<0.29269.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
[ns_server:debug,2012-11-26T4:34:41.975,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.27467.5201>:
{state,#Port<0.34659745>,#Port<0.34659734>,#Port<0.34659747>,
#Port<0.34659737>,<0.27520.5201>,<<"cut off">>,<<"cut off">>,[],7,
false,false,0,
{1353,933281,923009},
completed,
{<0.29242.5201>,#Ref<0.0.47906.195375>},
<<"replication_ns_1@10.6.2.37">>,<0.27467.5201>,
{had_backfill,false,undefined,[]},
false}.
[error_logger:info,2012-11-26T4:34:41.976,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,'ns_vbm_new_sup-saslbucket'}
started: [{pid,<0.29269.5201>},
{name,{new_child_id,"]^",'ns_1@10.6.2.40'}},
{mfargs,
{ebucketmigrator_srv,start_link,
[{"10.6.2.40",11209},
{"10.6.2.37",11209},
[{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
{on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
{username,"saslbucket"},
{password,"password"},
{vbuckets,"]^"},
{takeover,false},
{suffix,"ns_1@10.6.2.37"}]]}},
{restart_type,temporary},
{shutdown,60000},
{child_type,worker}]

[ns_server:debug,2012-11-26T4:34:41.977,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29242.5201>,{#Ref<0.0.47906.195218>,<0.29269.5201>}}
[ns_server:info,2012-11-26T4:34:41.979,ns_1@10.6.2.37:<0.25110.5201>:ns_memcached:do_handle_call:521]Changed vbucket 62 state to replica
[views:debug,2012-11-26T4:34:41.985,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:465]
Calling couch_set_view:mark_partitions_indexable([<<"saslbucket">>,
<<"_design/d11">>,
[0,1,2,3,4,5,6,7,8,9,10,11,
12,13,14,15,16,17,18,19,20,
21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,
39,40,41,42,43,44,45,46,47,
48,49,50,51,52,53,54,55,56,
57,58,59,60,61,62,63,64,65,
66,67,68,69,70,71,72,73,74,
75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,
93,94,95,96,97,98,99,100,
101,102,103,104,105,106,
107,108,109,110,111,112,
113,114,115,116,117,118,
119,120,121,122,123,124,
125,126,127,128,129,130,
131,132,133,134,135,136,
137,138,139,140,141,142,
143,144,145,146,147,148,
149,150,151,152,153,154,
155,156,157,158,159,160,
161,162,163,164,165,166,
167,168,169,170,171,172,
173,174,175,176,177,178,
179,180,181,182,183,184,
185,186,187,188,189,190,
191,192,193,194,195,196,
197,198,199,200,201,202,
203,204,205,206,207,208,
209,210,211,212,213,214,
215,216,217,218,219,220,
221,222,223,224,225,226,
227,228,229,230,231,232,
233,234,235,236,237,238,
239,240,241,242,243,244,
245,246,247,248,249,250,
251,252,253,254,255,256,
257,258,259,260,261,262,
263,264,265,266,267,268,
269,270,271,272,273,274,
275,276,277,278,279,280,
281,282,283,284,285,286,
287,288,289,290,291,292,
293,294,295,296,297,298,
299,300,301,302,303,304,
305,306,307,308,309,310,
311,312,313,314,315,316,
317,318,319,320,321,322,
323,324,325,326,327,328,
329,330,331,332,333,334,
335,336,337,338,339,340,
341,342,343,344,345,346,
347,348,349,350,351,352,
353,354,355,356,357,358,
359,360,361,362,363,364,
365,366,367,368,369,370,
371,372,373,374,375,376,
377,378,379,380,381,382,
383,384,385,386,387,388,
389,390,391,392,393,394,
395,396,397,398,399,400,
401,402,403,404,405,406,
407,408,409,410,411,412,
413,414,415,416,417,418,
419,420,421,422,423,424,
425,426,427,428,429,430,
431,432,433,434,435,436,
437,438,439,440,441,442,
443,444,445,446,447,448,
449,450,451,452,453,454,
455,456,457,458,459,460,
461,462,463,464,465,466,
467,468,469,470,471,472,
473,474,475,476,477,478,
479,480,481,482,483,484,
485,486,487,488,489,490,
491,492,493,494,495,496,
497,498,499,500,501,502,
503,504,505,506,507,508,
509,510,511,512,513,514,
515,516,517,518,519,520,
521,522,523,524,525,526,
527,528,529,530,531,532,
533,534,535,536,537,538,
539,540,541,542,543,544,
545,546,547,548,549,550,
551,552,553,554,555,556,
557,558,559,560,561,562,
563,564,565,566,567,568,
569,570,571,572,573,574,
575,576,577,578,579,580,
581,582,583,584,585,586,
587,588,589,590,591,592,
593,594,595,596,597,598,
599,600,601,602,603,604,
605,606,607,608,609,610,
611,612,613,614,615,616,
617,618,619,620,621,622,
623,624,625,626,627,628,
629,630,631,632,633,634,
635,636,637,638,639,640,
641,642,643,644,645,646,
647,648,649,650,651,652,
653,654,655,656,657,658,
659,660,661,662,663,664,
665,666,667,668,669,670,
671,672,673,674,675,676,
677,678,679,680,681,682,
683,684,685,686,687,688,
689,690,691,692,693,694,
695,696,697,698,699,700,
701,702,703,704,705,706,
707,708,709,710,711,712,
713,714,715,716,717,718,
719,720,721,722,723,724,
725,726,727,728,729,730,
731,732,733,734,735,736,
737,738,739,740,741,742,
743,744,745,746,747,748,
749,750,751,752,753,754,
755,756,757,758,759,760,
761,762,763,764,765,766,
767,768,769,770,771,772,
773,774,775,776,777,778,
779,780,781,782,783,784,
785,786,787,788,789,790,
791,792,793,794,795,796,
797,798,799,800,801,802,
803,804,805,806,807,808,
809,810,811,812,813,814,
815,816,817,818,819,820,
821,822,823,824,825,826,
827,828,829,830,831,832,
833,834,835,836,837,838,
839,840,841,842,843,844,
845,846,847,848,849,850,
851,852,853,854,855,856,
857,858,859,860,861,862,
863,864,865,866,867,868,
869,870,871,872,873,874,
875,876,877,878,879,880,
881,882,883,884,885,886,
887,888,889,890,891,892,
893,894,895,896,897,898,
899,900,901,902,903,904,
905,906,907,908,909,910,
911,912,913,914,915,916,
917,918,919,920,921,922,
923,924,925,926,927,928,
929,930,931,932,933,934,
935,936,937,938,939,940,
941,942,943,944,945,946,
947,948,949,950,951,952,
953,954,955,956,957,958,
959,960,961,962,963,964,
965,966,967,968,969,970,
971,972,973,974,975,976,
977,978,979,980,981,982,
983,984,985,986,987,988,
989,990,991,992,993,994,
995,996,997,998,999,1000,
1001,1002,1003,1004,1005,
1006,1007,1008,1009,1010,
1011,1012,1013,1014,1015,
1016,1017,1018,1019,1020,
1021,1022,1023]])
[ns_server:debug,2012-11-26T4:34:42.004,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:569]Reusing old upstream:
[{vbuckets,"]^"},{name,<<"replication_ns_1@10.6.2.37">>},{takeover,false}]
[views:debug,2012-11-26T4:34:41.993,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:466]
couch_set_view:mark_partitions_indexable([<<"saslbucket">>,<<"_design/d11">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,
15,16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,36,
37,38,39,40,41,42,43,44,45,46,47,
48,49,50,51,52,53,54,55,56,57,58,
59,60,61,62,63,64,65,66,67,68,69,
70,71,72,73,74,75,76,77,78,79,80,
81,82,83,84,85,86,87,88,89,90,91,
92,93,94,95,96,97,98,99,100,101,
102,103,104,105,106,107,108,109,
110,111,112,113,114,115,116,117,
118,119,120,121,122,123,124,125,
126,127,128,129,130,131,132,133,
134,135,136,137,138,139,140,141,
142,143,144,145,146,147,148,149,
150,151,152,153,154,155,156,157,
158,159,160,161,162,163,164,165,
166,167,168,169,170,171,172,173,
174,175,176,177,178,179,180,181,
182,183,184,185,186,187,188,189,
190,191,192,193,194,195,196,197,
198,199,200,201,202,203,204,205,
206,207,208,209,210,211,212,213,
214,215,216,217,218,219,220,221,
222,223,224,225,226,227,228,229,
230,231,232,233,234,235,236,237,
238,239,240,241,242,243,244,245,
246,247,248,249,250,251,252,253,
254,255,256,257,258,259,260,261,
262,263,264,265,266,267,268,269,
270,271,272,273,274,275,276,277,
278,279,280,281,282,283,284,285,
286,287,288,289,290,291,292,293,
294,295,296,297,298,299,300,301,
302,303,304,305,306,307,308,309,
310,311,312,313,314,315,316,317,
318,319,320,321,322,323,324,325,
326,327,328,329,330,331,332,333,
334,335,336,337,338,339,340,341,
342,343,344,345,346,347,348,349,
350,351,352,353,354,355,356,357,
358,359,360,361,362,363,364,365,
366,367,368,369,370,371,372,373,
374,375,376,377,378,379,380,381,
382,383,384,385,386,387,388,389,
390,391,392,393,394,395,396,397,
398,399,400,401,402,403,404,405,
406,407,408,409,410,411,412,413,
414,415,416,417,418,419,420,421,
422,423,424,425,426,427,428,429,
430,431,432,433,434,435,436,437,
438,439,440,441,442,443,444,445,
446,447,448,449,450,451,452,453,
454,455,456,457,458,459,460,461,
462,463,464,465,466,467,468,469,
470,471,472,473,474,475,476,477,
478,479,480,481,482,483,484,485,
486,487,488,489,490,491,492,493,
494,495,496,497,498,499,500,501,
502,503,504,505,506,507,508,509,
510,511,512,513,514,515,516,517,
518,519,520,521,522,523,524,525,
526,527,528,529,530,531,532,533,
534,535,536,537,538,539,540,541,
542,543,544,545,546,547,548,549,
550,551,552,553,554,555,556,557,
558,559,560,561,562,563,564,565,
566,567,568,569,570,571,572,573,
574,575,576,577,578,579,580,581,
582,583,584,585,586,587,588,589,
590,591,592,593,594,595,596,597,
598,599,600,601,602,603,604,605,
606,607,608,609,610,611,612,613,
614,615,616,617,618,619,620,621,
622,623,624,625,626,627,628,629,
630,631,632,633,634,635,636,637,
638,639,640,641,642,643,644,645,
646,647,648,649,650,651,652,653,
654,655,656,657,658,659,660,661,
662,663,664,665,666,667,668,669,
670,671,672,673,674,675,676,677,
678,679,680,681,682,683,684,685,
686,687,688,689,690,691,692,693,
694,695,696,697,698,699,700,701,
702,703,704,705,706,707,708,709,
710,711,712,713,714,715,716,717,
718,719,720,721,722,723,724,725,
726,727,728,729,730,731,732,733,
734,735,736,737,738,739,740,741,
742,743,744,745,746,747,748,749,
750,751,752,753,754,755,756,757,
758,759,760,761,762,763,764,765,
766,767,768,769,770,771,772,773,
774,775,776,777,778,779,780,781,
782,783,784,785,786,787,788,789,
790,791,792,793,794,795,796,797,
798,799,800,801,802,803,804,805,
806,807,808,809,810,811,812,813,
814,815,816,817,818,819,820,821,
822,823,824,825,826,827,828,829,
830,831,832,833,834,835,836,837,
838,839,840,841,842,843,844,845,
846,847,848,849,850,851,852,853,
854,855,856,857,858,859,860,861,
862,863,864,865,866,867,868,869,
870,871,872,873,874,875,876,877,
878,879,880,881,882,883,884,885,
886,887,888,889,890,891,892,893,
894,895,896,897,898,899,900,901,
902,903,904,905,906,907,908,909,
910,911,912,913,914,915,916,917,
918,919,920,921,922,923,924,925,
926,927,928,929,930,931,932,933,
934,935,936,937,938,939,940,941,
942,943,944,945,946,947,948,949,
950,951,952,953,954,955,956,957,
958,959,960,961,962,963,964,965,
966,967,968,969,970,971,972,973,
974,975,976,977,978,979,980,981,
982,983,984,985,986,987,988,989,
990,991,992,993,994,995,996,997,
998,999,1000,1001,1002,1003,1004,
1005,1006,1007,1008,1009,1010,1011,
1012,1013,1014,1015,1016,1017,1018,
1019,1020,1021,1022,1023]]) returned ok in 1ms
[rebalance:debug,2012-11-26T4:34:42.005,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:588]upstream_sender pid: <0.29271.5201>
[couchdb:error,2012-11-26T4:34:42.011,ns_1@10.6.2.37:<0.29184.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}

]
[views:debug,2012-11-26T4:34:42.010,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:471]
Calling couch_set_view:set_partition_states([<<"saslbucket">>,
<<"_design/d11">>,
[440],
[441],
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30,31,32,33,
34,35,36,37,38,39,40,41,42,43,
44,45,46,47,48,49,50,51,52,53,
54,55,56,57,58,59,60,61,62,63,
64,65,66,67,68,69,70,71,72,73,
74,75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,93,
94,95,96,97,98,99,100,101,102,
103,104,105,106,107,108,109,110,
111,112,113,114,115,116,117,118,
119,120,121,122,123,124,125,126,
127,128,129,130,131,132,133,134,
135,136,137,138,139,140,141,142,
143,144,145,146,147,148,149,150,
151,152,153,154,155,156,157,158,
159,160,161,162,163,164,165,166,
167,168,169,170,171,172,173,174,
175,176,177,178,179,180,181,182,
183,184,185,186,187,188,189,190,
191,192,193,194,195,196,197,198,
199,200,201,202,203,204,205,206,
207,208,209,210,211,212,213,214,
215,216,217,218,219,220,221,222,
223,224,225,226,227,228,229,230,
231,232,233,234,235,236,237,238,
239,240,241,242,243,244,245,246,
247,248,249,250,251,252,253,254,
255,256,257,258,259,260,261,262,
263,264,265,266,267,268,269,270,
271,272,273,274,275,276,277,278,
279,280,281,282,283,284,285,286,
287,288,289,290,291,292,293,294,
295,296,297,298,299,300,301,302,
303,304,305,306,307,308,309,310,
311,312,313,314,315,316,317,318,
319,320,321,322,323,324,325,326,
327,328,329,330,331,332,333,334,
335,336,337,338,339,340,341,342,
343,344,345,346,347,348,349,350,
351,352,353,354,355,356,357,358,
359,360,361,362,363,364,365,366,
367,368,369,370,371,372,373,374,
375,376,377,378,379,380,381,382,
383,384,385,386,387,388,389,390,
391,392,393,394,395,396,397,398,
399,400,401,402,403,404,405,406,
407,408,409,410,411,412,413,414,
415,416,417,418,419,420,421,422,
423,424,425,426,427,428,429,430,
431,432,433,434,435,436,437,438,
439,442,443,444,445,446,447,448,
449,450,451,452,453,454,455,456,
457,458,459,460,461,462,463,464,
465,466,467,468,469,470,471,472,
473,474,475,476,477,478,479,480,
481,482,483,484,485,486,487,488,
489,490,491,492,493,494,495,496,
497,498,499,500,501,502,503,504,
505,506,507,508,509,510,511,512,
513,514,515,516,517,518,519,520,
521,522,523,524,525,526,527,528,
529,530,531,532,533,534,535,536,
537,538,539,540,541,542,543,544,
545,546,547,548,549,550,551,552,
553,554,555,556,557,558,559,560,
561,562,563,564,565,566,567,568,
569,570,571,572,573,574,575,576,
577,578,579,580,581,582,583,584,
585,586,587,588,589,590,591,592,
593,594,595,596,597,598,599,600,
601,602,603,604,605,606,607,608,
609,610,611,612,613,614,615,616,
617,618,619,620,621,622,623,624,
625,626,627,628,629,630,631,632,
633,634,635,636,637,638,639,640,
641,642,643,644,645,646,647,648,
649,650,651,652,653,654,655,656,
657,658,659,660,661,662,663,664,
665,666,667,668,669,670,671,672,
673,674,675,676,677,678,679,680,
681,682,683,684,685,686,687,688,
689,690,691,692,693,694,695,696,
697,698,699,700,701,702,703,704,
705,706,707,708,709,710,711,712,
713,714,715,716,717,718,719,720,
721,722,723,724,725,726,727,728,
729,730,731,732,733,734,735,736,
737,738,739,740,741,742,743,744,
745,746,747,748,749,750,751,752,
753,754,755,756,757,758,759,760,
761,762,763,764,765,766,767,768,
769,770,771,772,773,774,775,776,
777,778,779,780,781,782,783,784,
785,786,787,788,789,790,791,792,
793,794,795,796,797,798,799,800,
801,802,803,804,805,806,807,808,
809,810,811,812,813,814,815,816,
817,818,819,820,821,822,823,824,
825,826,827,828,829,830,831,832,
833,834,835,836,837,838,839,840,
841,842,843,844,845,846,847,848,
849,850,851,852,853,854,855,856,
857,858,859,860,861,862,863,864,
865,866,867,868,869,870,871,872,
873,874,875,876,877,878,879,880,
881,882,883,884,885,886,887,888,
889,890,891,892,893,894,895,896,
897,898,899,900,901,902,903,904,
905,906,907,908,909,910,911,912,
913,914,915,916,917,918,919,920,
921,922,923,924,925,926,927,928,
929,930,931,932,933,934,935,936,
937,938,939,940,941,942,943,944,
945,946,947,948,949,950,951,952,
953,954,955,956,957,958,959,960,
961,962,963,964,965,966,967,968,
969,970,971,972,973,974,975,976,
977,978,979,980,981,982,983,984,
985,986,987,988,989,990,991,992,
993,994,995,996,997,998,999,
1000,1001,1002,1003,1004,1005,
1006,1007,1008,1009,1010,1011,
1012,1013,1014,1015,1016,1017,
1018,1019,1020,1021,1022,1023]])
[views:debug,2012-11-26T4:34:42.016,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:472]
couch_set_view:set_partition_states([<<"saslbucket">>,<<"_design/d11">>,
[440],
[441],
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63,64,65,66,67,
68,69,70,71,72,73,74,75,76,77,78,79,80,
81,82,83,84,85,86,87,88,89,90,91,92,93,
94,95,96,97,98,99,100,101,102,103,104,
105,106,107,108,109,110,111,112,113,114,
115,116,117,118,119,120,121,122,123,124,
125,126,127,128,129,130,131,132,133,134,
135,136,137,138,139,140,141,142,143,144,
145,146,147,148,149,150,151,152,153,154,
155,156,157,158,159,160,161,162,163,164,
165,166,167,168,169,170,171,172,173,174,
175,176,177,178,179,180,181,182,183,184,
185,186,187,188,189,190,191,192,193,194,
195,196,197,198,199,200,201,202,203,204,
205,206,207,208,209,210,211,212,213,214,
215,216,217,218,219,220,221,222,223,224,
225,226,227,228,229,230,231,232,233,234,
235,236,237,238,239,240,241,242,243,244,
245,246,247,248,249,250,251,252,253,254,
255,256,257,258,259,260,261,262,263,264,
265,266,267,268,269,270,271,272,273,274,
275,276,277,278,279,280,281,282,283,284,
285,286,287,288,289,290,291,292,293,294,
295,296,297,298,299,300,301,302,303,304,
305,306,307,308,309,310,311,312,313,314,
315,316,317,318,319,320,321,322,323,324,
325,326,327,328,329,330,331,332,333,334,
335,336,337,338,339,340,341,342,343,344,
345,346,347,348,349,350,351,352,353,354,
355,356,357,358,359,360,361,362,363,364,
365,366,367,368,369,370,371,372,373,374,
375,376,377,378,379,380,381,382,383,384,
385,386,387,388,389,390,391,392,393,394,
395,396,397,398,399,400,401,402,403,404,
405,406,407,408,409,410,411,412,413,414,
415,416,417,418,419,420,421,422,423,424,
425,426,427,428,429,430,431,432,433,434,
435,436,437,438,439,442,443,444,445,446,
447,448,449,450,451,452,453,454,455,456,
457,458,459,460,461,462,463,464,465,466,
467,468,469,470,471,472,473,474,475,476,
477,478,479,480,481,482,483,484,485,486,
487,488,489,490,491,492,493,494,495,496,
497,498,499,500,501,502,503,504,505,506,
507,508,509,510,511,512,513,514,515,516,
517,518,519,520,521,522,523,524,525,526,
527,528,529,530,531,532,533,534,535,536,
537,538,539,540,541,542,543,544,545,546,
547,548,549,550,551,552,553,554,555,556,
557,558,559,560,561,562,563,564,565,566,
567,568,569,570,571,572,573,574,575,576,
577,578,579,580,581,582,583,584,585,586,
587,588,589,590,591,592,593,594,595,596,
597,598,599,600,601,602,603,604,605,606,
607,608,609,610,611,612,613,614,615,616,
617,618,619,620,621,622,623,624,625,626,
627,628,629,630,631,632,633,634,635,636,
637,638,639,640,641,642,643,644,645,646,
647,648,649,650,651,652,653,654,655,656,
657,658,659,660,661,662,663,664,665,666,
667,668,669,670,671,672,673,674,675,676,
677,678,679,680,681,682,683,684,685,686,
687,688,689,690,691,692,693,694,695,696,
697,698,699,700,701,702,703,704,705,706,
707,708,709,710,711,712,713,714,715,716,
717,718,719,720,721,722,723,724,725,726,
727,728,729,730,731,732,733,734,735,736,
737,738,739,740,741,742,743,744,745,746,
747,748,749,750,751,752,753,754,755,756,
757,758,759,760,761,762,763,764,765,766,
767,768,769,770,771,772,773,774,775,776,
777,778,779,780,781,782,783,784,785,786,
787,788,789,790,791,792,793,794,795,796,
797,798,799,800,801,802,803,804,805,806,
807,808,809,810,811,812,813,814,815,816,
817,818,819,820,821,822,823,824,825,826,
827,828,829,830,831,832,833,834,835,836,
837,838,839,840,841,842,843,844,845,846,
847,848,849,850,851,852,853,854,855,856,
857,858,859,860,861,862,863,864,865,866,
867,868,869,870,871,872,873,874,875,876,
877,878,879,880,881,882,883,884,885,886,
887,888,889,890,891,892,893,894,895,896,
897,898,899,900,901,902,903,904,905,906,
907,908,909,910,911,912,913,914,915,916,
917,918,919,920,921,922,923,924,925,926,
927,928,929,930,931,932,933,934,935,936,
937,938,939,940,941,942,943,944,945,946,
947,948,949,950,951,952,953,954,955,956,
957,958,959,960,961,962,963,964,965,966,
967,968,969,970,971,972,973,974,975,976,
977,978,979,980,981,982,983,984,985,986,
987,988,989,990,991,992,993,994,995,996,
997,998,999,1000,1001,1002,1003,1004,
1005,1006,1007,1008,1009,1010,1011,1012,
1013,1014,1015,1016,1017,1018,1019,1020,
1021,1022,1023]]) returned ok in 1ms
[views:debug,2012-11-26T4:34:42.022,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:488]
Calling couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
<<"_design/d11">>,[]])
[ns_server:info,2012-11-26T4:34:42.023,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:41.821519 PST 3: TAP (Consumer) eq_tapq:anon_22 - Reset vbucket 163 was completed succecssfully.

[views:debug,2012-11-26T4:34:42.024,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:489]
couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
<<"_design/d11">>,[]]) returned ok in 0ms
[rebalance:debug,2012-11-26T4:34:42.025,ns_1@10.6.2.37:<0.29272.5201>:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 50 in vbucket 163
[ns_server:debug,2012-11-26T4:34:42.076,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.29272.5201>
[ns_server:debug,2012-11-26T4:34:42.076,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28872.5201>
[ns_server:debug,2012-11-26T4:34:42.077,ns_1@10.6.2.37:<0.28887.5201>:capi_set_view_manager:do_wait_index_updated:624]Caller died:

{'DOWN',#Ref<0.0.47906.158245>,process,<0.28872.5201>,shutdown}

[ns_server:debug,2012-11-26T4:34:42.077,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28397.5201>
[ns_server:debug,2012-11-26T4:34:42.078,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28008.5201>
[ns_server:debug,2012-11-26T4:34:42.084,ns_1@10.6.2.37:<0.28321.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.084,ns_1@10.6.2.37:<0.29297.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[views:debug,2012-11-26T4:34:42.083,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:465]
Calling couch_set_view:mark_partitions_indexable([<<"saslbucket">>,
<<"_design/d11">>,
[0,1,2,3,4,5,6,7,8,9,10,11,
12,13,14,15,16,17,18,19,20,
21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,
39,40,41,42,43,44,45,46,47,
48,49,50,51,52,53,54,55,56,
57,58,59,60,61,62,63,64,65,
66,67,68,69,70,71,72,73,74,
75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,
93,94,95,96,97,98,99,100,
101,102,103,104,105,106,
107,108,109,110,111,112,
113,114,115,116,117,118,
119,120,121,122,123,124,
125,126,127,128,129,130,
131,132,133,134,135,136,
137,138,139,140,141,142,
143,144,145,146,147,148,
149,150,151,152,153,154,
155,156,157,158,159,160,
161,162,163,164,165,166,
167,168,169,170,171,172,
173,174,175,176,177,178,
179,180,181,182,183,184,
185,186,187,188,189,190,
191,192,193,194,195,196,
197,198,199,200,201,202,
203,204,205,206,207,208,
209,210,211,212,213,214,
215,216,217,218,219,220,
221,222,223,224,225,226,
227,228,229,230,231,232,
233,234,235,236,237,238,
239,240,241,242,243,244,
245,246,247,248,249,250,
251,252,253,254,255,256,
257,258,259,260,261,262,
263,264,265,266,267,268,
269,270,271,272,273,274,
275,276,277,278,279,280,
281,282,283,284,285,286,
287,288,289,290,291,292,
293,294,295,296,297,298,
299,300,301,302,303,304,
305,306,307,308,309,310,
311,312,313,314,315,316,
317,318,319,320,321,322,
323,324,325,326,327,328,
329,330,331,332,333,334,
335,336,337,338,339,340,
341,342,343,344,345,346,
347,348,349,350,351,352,
353,354,355,356,357,358,
359,360,361,362,363,364,
365,366,367,368,369,370,
371,372,373,374,375,376,
377,378,379,380,381,382,
383,384,385,386,387,388,
389,390,391,392,393,394,
395,396,397,398,399,400,
401,402,403,404,405,406,
407,408,409,410,411,412,
413,414,415,416,417,418,
419,420,421,422,423,424,
425,426,427,428,429,430,
431,432,433,434,435,436,
437,438,439,440,441,442,
443,444,445,446,447,448,
449,450,451,452,453,454,
455,456,457,458,459,460,
461,462,463,464,465,466,
467,468,469,470,471,472,
473,474,475,476,477,478,
479,480,481,482,483,484,
485,486,487,488,489,490,
491,492,493,494,495,496,
497,498,499,500,501,502,
503,504,505,506,507,508,
509,510,511,512,513,514,
515,516,517,518,519,520,
521,522,523,524,525,526,
527,528,529,530,531,532,
533,534,535,536,537,538,
539,540,541,542,543,544,
545,546,547,548,549,550,
551,552,553,554,555,556,
557,558,559,560,561,562,
563,564,565,566,567,568,
569,570,571,572,573,574,
575,576,577,578,579,580,
581,582,583,584,585,586,
587,588,589,590,591,592,
593,594,595,596,597,598,
599,600,601,602,603,604,
605,606,607,608,609,610,
611,612,613,614,615,616,
617,618,619,620,621,622,
623,624,625,626,627,628,
629,630,631,632,633,634,
635,636,637,638,639,640,
641,642,643,644,645,646,
647,648,649,650,651,652,
653,654,655,656,657,658,
659,660,661,662,663,664,
665,666,667,668,669,670,
671,672,673,674,675,676,
677,678,679,680,681,682,
683,684,685,686,687,688,
689,690,691,692,693,694,
695,696,697,698,699,700,
701,702,703,704,705,706,
707,708,709,710,711,712,
713,714,715,716,717,718,
719,720,721,722,723,724,
725,726,727,728,729,730,
731,732,733,734,735,736,
737,738,739,740,741,742,
743,744,745,746,747,748,
749,750,751,752,753,754,
755,756,757,758,759,760,
761,762,763,764,765,766,
767,768,769,770,771,772,
773,774,775,776,777,778,
779,780,781,782,783,784,
785,786,787,788,789,790,
791,792,793,794,795,796,
797,798,799,800,801,802,
803,804,805,806,807,808,
809,810,811,812,813,814,
815,816,817,818,819,820,
821,822,823,824,825,826,
827,828,829,830,831,832,
833,834,835,836,837,838,
839,840,841,842,843,844,
845,846,847,848,849,850,
851,852,853,854,855,856,
857,858,859,860,861,862,
863,864,865,866,867,868,
869,870,871,872,873,874,
875,876,877,878,879,880,
881,882,883,884,885,886,
887,888,889,890,891,892,
893,894,895,896,897,898,
899,900,901,902,903,904,
905,906,907,908,909,910,
911,912,913,914,915,916,
917,918,919,920,921,922,
923,924,925,926,927,928,
929,930,931,932,933,934,
935,936,937,938,939,940,
941,942,943,944,945,946,
947,948,949,950,951,952,
953,954,955,956,957,958,
959,960,961,962,963,964,
965,966,967,968,969,970,
971,972,973,974,975,976,
977,978,979,980,981,982,
983,984,985,986,987,988,
989,990,991,992,993,994,
995,996,997,998,999,1000,
1001,1002,1003,1004,1005,
1006,1007,1008,1009,1010,
1011,1012,1013,1014,1015,
1016,1017,1018,1019,1020,
1021,1022,1023]])
[ns_server:debug,2012-11-26T4:34:42.087,ns_1@10.6.2.37:<0.29297.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[rebalance:info,2012-11-26T4:34:42.088,ns_1@10.6.2.37:<0.28321.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:42.092,ns_1@10.6.2.37:<0.27922.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.092,ns_1@10.6.2.37:<0.29314.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:42.101,ns_1@10.6.2.37:<0.29314.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[views:debug,2012-11-26T4:34:42.091,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:466]
couch_set_view:mark_partitions_indexable([<<"saslbucket">>,<<"_design/d11">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,
15,16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,36,
37,38,39,40,41,42,43,44,45,46,47,
48,49,50,51,52,53,54,55,56,57,58,
59,60,61,62,63,64,65,66,67,68,69,
70,71,72,73,74,75,76,77,78,79,80,
81,82,83,84,85,86,87,88,89,90,91,
92,93,94,95,96,97,98,99,100,101,
102,103,104,105,106,107,108,109,
110,111,112,113,114,115,116,117,
118,119,120,121,122,123,124,125,
126,127,128,129,130,131,132,133,
134,135,136,137,138,139,140,141,
142,143,144,145,146,147,148,149,
150,151,152,153,154,155,156,157,
158,159,160,161,162,163,164,165,
166,167,168,169,170,171,172,173,
174,175,176,177,178,179,180,181,
182,183,184,185,186,187,188,189,
190,191,192,193,194,195,196,197,
198,199,200,201,202,203,204,205,
206,207,208,209,210,211,212,213,
214,215,216,217,218,219,220,221,
222,223,224,225,226,227,228,229,
230,231,232,233,234,235,236,237,
238,239,240,241,242,243,244,245,
246,247,248,249,250,251,252,253,
254,255,256,257,258,259,260,261,
262,263,264,265,266,267,268,269,
270,271,272,273,274,275,276,277,
278,279,280,281,282,283,284,285,
286,287,288,289,290,291,292,293,
294,295,296,297,298,299,300,301,
302,303,304,305,306,307,308,309,
310,311,312,313,314,315,316,317,
318,319,320,321,322,323,324,325,
326,327,328,329,330,331,332,333,
334,335,336,337,338,339,340,341,
342,343,344,345,346,347,348,349,
350,351,352,353,354,355,356,357,
358,359,360,361,362,363,364,365,
366,367,368,369,370,371,372,373,
374,375,376,377,378,379,380,381,
382,383,384,385,386,387,388,389,
390,391,392,393,394,395,396,397,
398,399,400,401,402,403,404,405,
406,407,408,409,410,411,412,413,
414,415,416,417,418,419,420,421,
422,423,424,425,426,427,428,429,
430,431,432,433,434,435,436,437,
438,439,440,441,442,443,444,445,
446,447,448,449,450,451,452,453,
454,455,456,457,458,459,460,461,
462,463,464,465,466,467,468,469,
470,471,472,473,474,475,476,477,
478,479,480,481,482,483,484,485,
486,487,488,489,490,491,492,493,
494,495,496,497,498,499,500,501,
502,503,504,505,506,507,508,509,
510,511,512,513,514,515,516,517,
518,519,520,521,522,523,524,525,
526,527,528,529,530,531,532,533,
534,535,536,537,538,539,540,541,
542,543,544,545,546,547,548,549,
550,551,552,553,554,555,556,557,
558,559,560,561,562,563,564,565,
566,567,568,569,570,571,572,573,
574,575,576,577,578,579,580,581,
582,583,584,585,586,587,588,589,
590,591,592,593,594,595,596,597,
598,599,600,601,602,603,604,605,
606,607,608,609,610,611,612,613,
614,615,616,617,618,619,620,621,
622,623,624,625,626,627,628,629,
630,631,632,633,634,635,636,637,
638,639,640,641,642,643,644,645,
646,647,648,649,650,651,652,653,
654,655,656,657,658,659,660,661,
662,663,664,665,666,667,668,669,
670,671,672,673,674,675,676,677,
678,679,680,681,682,683,684,685,
686,687,688,689,690,691,692,693,
694,695,696,697,698,699,700,701,
702,703,704,705,706,707,708,709,
710,711,712,713,714,715,716,717,
718,719,720,721,722,723,724,725,
726,727,728,729,730,731,732,733,
734,735,736,737,738,739,740,741,
742,743,744,745,746,747,748,749,
750,751,752,753,754,755,756,757,
758,759,760,761,762,763,764,765,
766,767,768,769,770,771,772,773,
774,775,776,777,778,779,780,781,
782,783,784,785,786,787,788,789,
790,791,792,793,794,795,796,797,
798,799,800,801,802,803,804,805,
806,807,808,809,810,811,812,813,
814,815,816,817,818,819,820,821,
822,823,824,825,826,827,828,829,
830,831,832,833,834,835,836,837,
838,839,840,841,842,843,844,845,
846,847,848,849,850,851,852,853,
854,855,856,857,858,859,860,861,
862,863,864,865,866,867,868,869,
870,871,872,873,874,875,876,877,
878,879,880,881,882,883,884,885,
886,887,888,889,890,891,892,893,
894,895,896,897,898,899,900,901,
902,903,904,905,906,907,908,909,
910,911,912,913,914,915,916,917,
918,919,920,921,922,923,924,925,
926,927,928,929,930,931,932,933,
934,935,936,937,938,939,940,941,
942,943,944,945,946,947,948,949,
950,951,952,953,954,955,956,957,
958,959,960,961,962,963,964,965,
966,967,968,969,970,971,972,973,
974,975,976,977,978,979,980,981,
982,983,984,985,986,987,988,989,
990,991,992,993,994,995,996,997,
998,999,1000,1001,1002,1003,1004,
1005,1006,1007,1008,1009,1010,1011,
1012,1013,1014,1015,1016,1017,1018,
1019,1020,1021,1022,1023]]) returned ok in 3ms
[rebalance:info,2012-11-26T4:34:42.105,ns_1@10.6.2.37:<0.27922.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:42.110,ns_1@10.6.2.37:compaction_daemon<0.23916.5201>:compaction_daemon:handle_info:353]Got config_changed in state idle. Nothing to do since compaction is not running
[ns_server:debug,2012-11-26T4:34:42.112,ns_1@10.6.2.37:ns_config_log<0.768.0>:ns_config_log:log_common:111]config change:
buckets ->
[{configs,[{"default",
[

{map,[]}

,

{fastForwardMap,[]},
{uuid,<<"ef2292c303b134a8debe20aa2f604c2f">>},
{sasl_password,[]},
{num_replicas,1},
{replica_index,false},
{ram_quota,10485760000},
{auth_type,sasl},
{autocompaction,false},
{flush_enabled,false},
{type,membase},
{num_vbuckets,1024},
{servers,['ns_1@10.6.2.38','ns_1@10.6.2.39','ns_1@10.6.2.40', 'ns_1@10.6.2.42','ns_1@10.6.2.44','ns_1@10.6.2.45', 'ns_1@10.6.2.43']}]},
{"saslbucket",
[{map,[{94, ['ns_1@10.6.2.40','ns_1@10.6.2.45'], ['ns_1@10.6.2.40','ns_1@10.6.2.37']}]},
{fastForwardMap,[]}

,

{uuid,<<"e344089af6ea1f8153912ad043034083">>}

,

{num_replicas,1}

,

{replica_index,false}

,

{ram_quota,10485760000}

,

{auth_type,sasl}

,

{sasl_password,"password"}

,

{autocompaction,false}

,

{flush_enabled,false}

,

{type,membase}

,

{num_vbuckets,1024}

,

{servers,['ns_1@10.6.2.38','ns_1@10.6.2.39','ns_1@10.6.2.40', 'ns_1@10.6.2.42','ns_1@10.6.2.43','ns_1@10.6.2.44', 'ns_1@10.6.2.37','ns_1@10.6.2.45']}

]}]}]
[views:debug,2012-11-26T4:34:42.120,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:471]
Calling couch_set_view:set_partition_states([<<"saslbucket">>,
<<"_design/d11">>,
[440],
[],
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30,31,32,33,
34,35,36,37,38,39,40,41,42,43,
44,45,46,47,48,49,50,51,52,53,
54,55,56,57,58,59,60,61,62,63,
64,65,66,67,68,69,70,71,72,73,
74,75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,93,
94,95,96,97,98,99,100,101,102,
103,104,105,106,107,108,109,110,
111,112,113,114,115,116,117,118,
119,120,121,122,123,124,125,126,
127,128,129,130,131,132,133,134,
135,136,137,138,139,140,141,142,
143,144,145,146,147,148,149,150,
151,152,153,154,155,156,157,158,
159,160,161,162,163,164,165,166,
167,168,169,170,171,172,173,174,
175,176,177,178,179,180,181,182,
183,184,185,186,187,188,189,190,
191,192,193,194,195,196,197,198,
199,200,201,202,203,204,205,206,
207,208,209,210,211,212,213,214,
215,216,217,218,219,220,221,222,
223,224,225,226,227,228,229,230,
231,232,233,234,235,236,237,238,
239,240,241,242,243,244,245,246,
247,248,249,250,251,252,253,254,
255,256,257,258,259,260,261,262,
263,264,265,266,267,268,269,270,
271,272,273,274,275,276,277,278,
279,280,281,282,283,284,285,286,
287,288,289,290,291,292,293,294,
295,296,297,298,299,300,301,302,
303,304,305,306,307,308,309,310,
311,312,313,314,315,316,317,318,
319,320,321,322,323,324,325,326,
327,328,329,330,331,332,333,334,
335,336,337,338,339,340,341,342,
343,344,345,346,347,348,349,350,
351,352,353,354,355,356,357,358,
359,360,361,362,363,364,365,366,
367,368,369,370,371,372,373,374,
375,376,377,378,379,380,381,382,
383,384,385,386,387,388,389,390,
391,392,393,394,395,396,397,398,
399,400,401,402,403,404,405,406,
407,408,409,410,411,412,413,414,
415,416,417,418,419,420,421,422,
423,424,425,426,427,428,429,430,
431,432,433,434,435,436,437,438,
439,441,442,443,444,445,446,447,
448,449,450,451,452,453,454,455,
456,457,458,459,460,461,462,463,
464,465,466,467,468,469,470,471,
472,473,474,475,476,477,478,479,
480,481,482,483,484,485,486,487,
488,489,490,491,492,493,494,495,
496,497,498,499,500,501,502,503,
504,505,506,507,508,509,510,511,
512,513,514,515,516,517,518,519,
520,521,522,523,524,525,526,527,
528,529,530,531,532,533,534,535,
536,537,538,539,540,541,542,543,
544,545,546,547,548,549,550,551,
552,553,554,555,556,557,558,559,
560,561,562,563,564,565,566,567,
568,569,570,571,572,573,574,575,
576,577,578,579,580,581,582,583,
584,585,586,587,588,589,590,591,
592,593,594,595,596,597,598,599,
600,601,602,603,604,605,606,607,
608,609,610,611,612,613,614,615,
616,617,618,619,620,621,622,623,
624,625,626,627,628,629,630,631,
632,633,634,635,636,637,638,639,
640,641,642,643,644,645,646,647,
648,649,650,651,652,653,654,655,
656,657,658,659,660,661,662,663,
664,665,666,667,668,669,670,671,
672,673,674,675,676,677,678,679,
680,681,682,683,684,685,686,687,
688,689,690,691,692,693,694,695,
696,697,698,699,700,701,702,703,
704,705,706,707,708,709,710,711,
712,713,714,715,716,717,718,719,
720,721,722,723,724,725,726,727,
728,729,730,731,732,733,734,735,
736,737,738,739,740,741,742,743,
744,745,746,747,748,749,750,751,
752,753,754,755,756,757,758,759,
760,761,762,763,764,765,766,767,
768,769,770,771,772,773,774,775,
776,777,778,779,780,781,782,783,
784,785,786,787,788,789,790,791,
792,793,794,795,796,797,798,799,
800,801,802,803,804,805,806,807,
808,809,810,811,812,813,814,815,
816,817,818,819,820,821,822,823,
824,825,826,827,828,829,830,831,
832,833,834,835,836,837,838,839,
840,841,842,843,844,845,846,847,
848,849,850,851,852,853,854,855,
856,857,858,859,860,861,862,863,
864,865,866,867,868,869,870,871,
872,873,874,875,876,877,878,879,
880,881,882,883,884,885,886,887,
888,889,890,891,892,893,894,895,
896,897,898,899,900,901,902,903,
904,905,906,907,908,909,910,911,
912,913,914,915,916,917,918,919,
920,921,922,923,924,925,926,927,
928,929,930,931,932,933,934,935,
936,937,938,939,940,941,942,943,
944,945,946,947,948,949,950,951,
952,953,954,955,956,957,958,959,
960,961,962,963,964,965,966,967,
968,969,970,971,972,973,974,975,
976,977,978,979,980,981,982,983,
984,985,986,987,988,989,990,991,
992,993,994,995,996,997,998,999,
1000,1001,1002,1003,1004,1005,
1006,1007,1008,1009,1010,1011,
1012,1013,1014,1015,1016,1017,
1018,1019,1020,1021,1022,1023]])
[rebalance:error,2012-11-26T4:34:42.126,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:handle_info:420]killing myself due to exit signal: {'EXIT',<0.29220.5201>,
badmatch,{error,closed,
[

{ebucketmigrator_srv, upstream_sender_loop,1}

]}}
[ns_server:debug,2012-11-26T4:34:42.129,ns_1@10.6.2.37:<0.26815.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.130,ns_1@10.6.2.37:<0.29318.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:42.138,ns_1@10.6.2.37:<0.29318.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[ns_server:debug,2012-11-26T4:34:42.139,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.139,ns_1@10.6.2.37:<0.29319.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:42.147,ns_1@10.6.2.37:<0.29319.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[rebalance:info,2012-11-26T4:34:42.149,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[rebalance:info,2012-11-26T4:34:42.159,ns_1@10.6.2.37:<0.26815.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:42.183,ns_1@10.6.2.37:ns_config_rep<0.23781.5201>:ns_config_rep:handle_call:113]Got full synchronization request from 'ns_1@10.6.2.38'
[ns_server:debug,2012-11-26T4:34:42.201,ns_1@10.6.2.37:ns_config_rep<0.23781.5201>:ns_config_rep:handle_call:119]Fully synchronized config in 497 us
[ns_server:info,2012-11-26T4:34:42.202,ns_1@10.6.2.37:<0.29316.5201>:ebucketmigrator_srv:init:492]Setting

{"10.6.2.37",11209} vbucket 62 to state replica
[views:debug,2012-11-26T4:34:42.204,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:472]
couch_set_view:set_partition_states([<<"saslbucket">>,<<"_design/d11">>,
[440],
[],
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63,64,65,66,67,
68,69,70,71,72,73,74,75,76,77,78,79,80,
81,82,83,84,85,86,87,88,89,90,91,92,93,
94,95,96,97,98,99,100,101,102,103,104,
105,106,107,108,109,110,111,112,113,114,
115,116,117,118,119,120,121,122,123,124,
125,126,127,128,129,130,131,132,133,134,
135,136,137,138,139,140,141,142,143,144,
145,146,147,148,149,150,151,152,153,154,
155,156,157,158,159,160,161,162,163,164,
165,166,167,168,169,170,171,172,173,174,
175,176,177,178,179,180,181,182,183,184,
185,186,187,188,189,190,191,192,193,194,
195,196,197,198,199,200,201,202,203,204,
205,206,207,208,209,210,211,212,213,214,
215,216,217,218,219,220,221,222,223,224,
225,226,227,228,229,230,231,232,233,234,
235,236,237,238,239,240,241,242,243,244,
245,246,247,248,249,250,251,252,253,254,
255,256,257,258,259,260,261,262,263,264,
265,266,267,268,269,270,271,272,273,274,
275,276,277,278,279,280,281,282,283,284,
285,286,287,288,289,290,291,292,293,294,
295,296,297,298,299,300,301,302,303,304,
305,306,307,308,309,310,311,312,313,314,
315,316,317,318,319,320,321,322,323,324,
325,326,327,328,329,330,331,332,333,334,
335,336,337,338,339,340,341,342,343,344,
345,346,347,348,349,350,351,352,353,354,
355,356,357,358,359,360,361,362,363,364,
365,366,367,368,369,370,371,372,373,374,
375,376,377,378,379,380,381,382,383,384,
385,386,387,388,389,390,391,392,393,394,
395,396,397,398,399,400,401,402,403,404,
405,406,407,408,409,410,411,412,413,414,
415,416,417,418,419,420,421,422,423,424,
425,426,427,428,429,430,431,432,433,434,
435,436,437,438,439,441,442,443,444,445,
446,447,448,449,450,451,452,453,454,455,
456,457,458,459,460,461,462,463,464,465,
466,467,468,469,470,471,472,473,474,475,
476,477,478,479,480,481,482,483,484,485,
486,487,488,489,490,491,492,493,494,495,
496,497,498,499,500,501,502,503,504,505,
506,507,508,509,510,511,512,513,514,515,
516,517,518,519,520,521,522,523,524,525,
526,527,528,529,530,531,532,533,534,535,
536,537,538,539,540,541,542,543,544,545,
546,547,548,549,550,551,552,553,554,555,
556,557,558,559,560,561,562,563,564,565,
566,567,568,569,570,571,572,573,574,575,
576,577,578,579,580,581,582,583,584,585,
586,587,588,589,590,591,592,593,594,595,
596,597,598,599,600,601,602,603,604,605,
606,607,608,609,610,611,612,613,614,615,
616,617,618,619,620,621,622,623,624,625,
626,627,628,629,630,631,632,633,634,635,
636,637,638,639,640,641,642,643,644,645,
646,647,648,649,650,651,652,653,654,655,
656,657,658,659,660,661,662,663,664,665,
666,667,668,669,670,671,672,673,674,675,
676,677,678,679,680,681,682,683,684,685,
686,687,688,689,690,691,692,693,694,695,
696,697,698,699,700,701,702,703,704,705,
706,707,708,709,710,711,712,713,714,715,
716,717,718,719,720,721,722,723,724,725,
726,727,728,729,730,731,732,733,734,735,
736,737,738,739,740,741,742,743,744,745,
746,747,748,749,750,751,752,753,754,755,
756,757,758,759,760,761,762,763,764,765,
766,767,768,769,770,771,772,773,774,775,
776,777,778,779,780,781,782,783,784,785,
786,787,788,789,790,791,792,793,794,795,
796,797,798,799,800,801,802,803,804,805,
806,807,808,809,810,811,812,813,814,815,
816,817,818,819,820,821,822,823,824,825,
826,827,828,829,830,831,832,833,834,835,
836,837,838,839,840,841,842,843,844,845,
846,847,848,849,850,851,852,853,854,855,
856,857,858,859,860,861,862,863,864,865,
866,867,868,869,870,871,872,873,874,875,
876,877,878,879,880,881,882,883,884,885,
886,887,888,889,890,891,892,893,894,895,
896,897,898,899,900,901,902,903,904,905,
906,907,908,909,910,911,912,913,914,915,
916,917,918,919,920,921,922,923,924,925,
926,927,928,929,930,931,932,933,934,935,
936,937,938,939,940,941,942,943,944,945,
946,947,948,949,950,951,952,953,954,955,
956,957,958,959,960,961,962,963,964,965,
966,967,968,969,970,971,972,973,974,975,
976,977,978,979,980,981,982,983,984,985,
986,987,988,989,990,991,992,993,994,995,
996,997,998,999,1000,1001,1002,1003,
1004,1005,1006,1007,1008,1009,1010,1011,
1012,1013,1014,1015,1016,1017,1018,1019,
1020,1021,1022,1023]]) returned ok in 80ms
[views:debug,2012-11-26T4:34:42.226,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:488]
Calling couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
<<"_design/d11">>,[]])
[views:debug,2012-11-26T4:34:42.239,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:489]
couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
<<"_design/d11">>,[]]) returned ok in 0ms
[ns_server:debug,2012-11-26T4:34:42.262,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
[ns_server:info,2012-11-26T4:34:42.266,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:42.040152 PST 3: Deletion of vbucket 441 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.040581 PST 3: Deletion of vbucket 441 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.054922 PST 3: Deletion of vbucket 197 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.056613 PST 3: Deletion of vbucket 197 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.060198 PST 3: Deletion of vbucket 163 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.067666 PST 3: Deletion of vbucket 163 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.069318 PST 3: Deletion of vbucket 128 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.070137 PST 3: Deletion of vbucket 128 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.071390 PST 3: Deletion of vbucket 94 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.071774 PST 3: Deletion of vbucket 94 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.072726 PST 3: Deletion of vbucket 62 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.073136 PST 3: Deletion of vbucket 62 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.074003 PST 3: Deletion of vbucket 61 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.074337 PST 3: Deletion of vbucket 61 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.074697 PST 3: Deletion of vbucket 26 failed because the vbucket is not in a dead state
memcached<0.23890.5201>: Mon Nov 26 04:34:42.075026 PST 3: Deletion of vbucket 26 was completed.
memcached<0.23890.5201>: Mon Nov 26 04:34:42.092594 PST 3: TAP (Consumer) eq_tapq:anon_21 - disconnected
memcached<0.23890.5201>: Mon Nov 26 04:34:42.106474 PST 3: TAP (Consumer) eq_tapq:anon_19 - disconnected
memcached<0.23890.5201>: Mon Nov 26 04:34:42.159445 PST 3: TAP (Consumer) eq_tapq:anon_15 - disconnected
memcached<0.23890.5201>: Mon Nov 26 04:34:42.184971 PST 3: TAP (Consumer) eq_tapq:anon_22 - disconnected

[ns_server:debug,2012-11-26T4:34:42.294,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
[couchdb:error,2012-11-26T4:34:42.309,ns_1@10.6.2.37:<0.29237.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:error,2012-11-26T4:34:42.352,ns_1@10.6.2.37:<0.29238.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:error,2012-11-26T4:34:42.370,ns_1@10.6.2.37:<0.29239.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:42.371,ns_1@10.6.2.37:tap_replication_manager-saslbucket<0.25114.5201>:tap_replication_manager:change_vbucket_filter:185]Going to change replication from 'ns_1@10.6.2.39' to have
"<" ([], "=")
[ns_server:debug,2012-11-26T4:34:42.378,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:117]Registered myself under id:{"saslbucket",
{new_child_id,"<",'ns_1@10.6.2.39'},
#Ref<0.0.47906.208825>}
Args:[{"10.6.2.39",11209},
{"10.6.2.37",11209}

,
[

{old_state_retriever,#Fun<ns_vbm_new_sup.0.113560106>},
{on_not_ready_vbuckets,#Fun<tap_replication_manager.2.88495417>},
{username,"saslbucket"},
{password,"password"},
{vbuckets,"<"},
{takeover,false},
{suffix,"ns_1@10.6.2.37"}]]
[ns_server:debug,2012-11-26T4:34:42.379,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.29120.5201>
[ns_server:info,2012-11-26T4:34:42.379,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
[couchdb:error,2012-11-26T4:34:42.412,ns_1@10.6.2.37:<0.29244.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:42.435,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
[{60,26}]
[ns_server:info,2012-11-26T4:34:42.437,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
[ns_server:info,2012-11-26T4:34:42.438,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
[ns_server:info,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
[ns_server:debug,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
[ns_server:debug,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
[ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29378.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29378.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[rebalance:info,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
[ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
[ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
[ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.29120.5201>
[ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
[ns_server:info,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.29380.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
[ns_server:debug,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.29120.5201>:
{state,#Port<0.34658446>,#Port<0.34658438>,#Port<0.34658449>,
#Port<0.34658440>,<0.29132.5201>,<<"cut off">>,<<"cut off">>,[],10,
false,false,0,
{1353,933282,438008},
completed,
{<0.29342.5201>,#Ref<0.0.47906.208842>},
<<"replication_ns_1@10.6.2.37">>,<0.29120.5201>,
{had_backfill,false,undefined,[]},
false}.
[ns_server:debug,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29342.5201>,{#Ref<0.0.47906.208827>,<0.29380.5201>}}
[ns_server:info,2012-11-26T4:34:42.449,ns_1@10.6.2.37:tap_replication_manager-saslbucket<0.25114.5201>:tap_replication_manager:change_vbucket_filter:185]Going to change replication from 'ns_1@10.6.2.40' to have
"]" ([], "^")
[error_logger:info,2012-11-26T4:34:42.450,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,'ns_vbm_new_sup-saslbucket'}
started: [{pid,<0.29380.5201>},
{name,{new_child_id,"<",'ns_1@10.6.2.39'}},
{mfargs,
{ebucketmigrator_srv,start_link,
[{"10.6.2.39",11209},
{"10.6.2.37",11209},
[{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
{on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
{username,"saslbucket"},
{password,"password"},
{vbuckets,"<"},
{takeover,false},
{suffix,"ns_1@10.6.2.37"}]]}},
{restart_type,temporary},
{shutdown,60000},
{child_type,worker}]

[ns_server:debug,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:117]Registered myself under id:{"saslbucket",
{new_child_id,"]",'ns_1@10.6.2.40'},
#Ref<0.0.47906.210506>}
Args:[{"10.6.2.40",11209},
{"10.6.2.37",11209},
[{old_state_retriever,#Fun<ns_vbm_new_sup.0.113560106>}

,

{on_not_ready_vbuckets,#Fun<tap_replication_manager.2.88495417>}

,

{username,"saslbucket"},
{password,"password"},
{vbuckets,"]"},
{takeover,false},
{suffix,"ns_1@10.6.2.37"}]]
[ns_server:debug,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.29269.5201>
[ns_server:info,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
[couchdb:error,2012-11-26T4:34:42.482,ns_1@10.6.2.37:<0.29245.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:debug,2012-11-26T4:34:42.492,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:569]Reusing old upstream:
[{vbuckets,"<"},{name,<<"replication_ns_1@10.6.2.37">>},{takeover,false}]
[rebalance:debug,2012-11-26T4:34:42.493,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:588]upstream_sender pid: <0.29385.5201>
[couchdb:error,2012-11-26T4:34:42.513,ns_1@10.6.2.37:<0.29247.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:42.516,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:42.308712 PST 3: Notified the completion of checkpoint persistence for vbucket 128, cookie 0x6aaeb00
memcached<0.23890.5201>: Mon Nov 26 04:34:42.444246 PST 3: Schedule cleanup of "eq_tapq:anon_15"
memcached<0.23890.5201>: Mon Nov 26 04:34:42.444311 PST 3: Schedule cleanup of "eq_tapq:anon_19"
memcached<0.23890.5201>: Mon Nov 26 04:34:42.444330 PST 3: Schedule cleanup of "eq_tapq:anon_21"
memcached<0.23890.5201>: Mon Nov 26 04:34:42.444365 PST 3: Schedule cleanup of "eq_tapq:anon_22"

[couchdb:error,2012-11-26T4:34:42.521,ns_1@10.6.2.37:<0.29248.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

Stacktrace: [{capi_utils,must_open_vbucket,2},
{capi_frontend,with_subdb,3},
{couch_httpd_db,handle_design_req,2},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[ns_server:info,2012-11-26T4:34:42.523,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
[{93,38}]
[ns_server:info,2012-11-26T4:34:42.525,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
[ns_server:info,2012-11-26T4:34:42.526,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
[ns_server:info,2012-11-26T4:34:42.526,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
[ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
[ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
[ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
[ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29405.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
[ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29405.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
[rebalance:info,2012-11-26T4:34:42.530,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

[ns_server:debug,2012-11-26T4:34:42.533,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
[ns_server:debug,2012-11-26T4:34:42.534,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
[ns_server:debug,2012-11-26T4:34:42.534,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
[ns_server:debug,2012-11-26T4:34:42.536,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.29269.5201>
[ns_server:debug,2012-11-26T4:34:42.537,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
[ns_server:info,2012-11-26T4:34:42.537,ns_1@10.6.2.37:<0.29423.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
[ns_server:debug,2012-11-26T4:34:42.538,ns_1@10.6.2.37:<0.29423.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.29269.5201>:
{state,#Port<0.34659745>,#Port<0.34659734>,#Port<0.34659747>,
#Port<0.34659737>,<0.29271.5201>,<<"cut off">>,<<"cut off">>,[],10,
false,false,0,
{1353,933282,526685},
completed,
{<0.29381.5201>,#Ref<0.0.47906.210588>},
<<"replication_ns_1@10.6.2.37">>,<0.29269.5201>,
{had_backfill,false,undefined,[]},
false}.
[ns_server:debug,2012-11-26T4:34:42.538,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29381.5201>,{#Ref<0.0.47906.210536>,<0.29423.5201>}}
[error_logger:info,2012-11-26T4:34:42.538,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,'ns_vbm_new_sup-saslbucket'}
started: [{pid,<0.29423.5201>},
{name,{new_child_id,"]",'ns_1@10.6.2.40'}},
{mfargs,
{ebucketmigrator_srv,start_link,
[{"10.6.2.40",11209},
{"10.6.2.37",11209},
[{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
{on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
{username,"saslbucket"}

,

{password,"password"}

,

{vbuckets,"]"}

,

{takeover,false}

,

{suffix,"ns_1@10.6.2.37"}

]]}},

{restart_type,temporary}

,

{shutdown,60000}

,

{child_type,worker}

]

[user:info,2012-11-26T4:34:42.540,ns_1@10.6.2.37:ns_config<0.762.0>:ns_janitor:maybe_stop_rebalance_status:147]Resetting rebalance status since it's not really running
[ns_server:debug,2012-11-26T4:34:42.543,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
[couchdb:error,2012-11-26T4:34:42.556,ns_1@10.6.2.37:<0.29267.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,

{not_found,no_db_file}

}}

Stacktrace: [

{capi_utils,must_open_vbucket,2}

,

{capi_frontend,with_subdb,3}

,

{couch_httpd_db,handle_design_req,2}

,

{couch_httpd,handle_request,6}

,

{mochiweb_http,headers,5}

,

{proc_lib,init_p_do_apply,3}

]