Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6595 [RN 2.0.1]][longevity] something unknown is causing severe timeouts in ns_server. Particularly under views building and/or compaction. Which causes rebalance to fail and other types of badness.
  3. MB-7261

[system test] Rebalance stopped by janitor /Resetting rebalance status since it's not really running (couchdb:error Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}})

    Details

    • Type: Technical task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.0.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      build-1956:
      orange cluster, 2 buckets * ~40M keys :
      1:10.6.2.37
      2:10.6.2.38
      3:10.6.2.39
      4:10.6.2.40
      5:10.6.2.42
      6:10.6.2.43
      7:10.6.2.44
      8:10.6.2.45

      steps:
      1.add a node and rebalance (RB-1) -passed
      2. swap the orchestrator with a new node (RB-2) - passed

      cluster was about a day without any action( no data loading, rebalance or something else)

      3. swap rebalance non orchestrator node (RB-3) ( returned back master node from step 2)

      Starting rebalance, KeepNodes = ['ns_1@10.6.2.38','ns_1@10.6.2.39',
      'ns_1@10.6.2.40','ns_1@10.6.2.42',
      'ns_1@10.6.2.43','ns_1@10.6.2.44',
      'ns_1@10.6.2.37'], EjectNodes = ['ns_1@10.6.2.45']

      Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.6.2.37 15:34:42 - Mon Nov 26, 2012
      killing myself due to exit signal: {'EXIT',<0.29220.5201>,
      badmatch,{error,closed,
      [

      {ebucketmigrator_srv, upstream_sender_loop,1}

      ]}}

      [ns_server:debug,2012-11-26T4:34:41.836,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.27467.5201>
      [couchdb:error,2012-11-26T4:34:41.837,ns_1@10.6.2.37:<0.29176.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,

      {not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:41.837,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
      [couchdb:error,2012-11-26T4:34:41.863,ns_1@10.6.2.37:<0.29177.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}

      }}

      Stacktrace: [

      {capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [couchdb:error,2012-11-26T4:34:41.866,ns_1@10.6.2.37:<0.29178.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2}

      ,

      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [couchdb:error,2012-11-26T4:34:41.879,ns_1@10.6.2.37:<0.29179.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3}

      ,

      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:41.905,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
      [{93,38},{94,40}]
      [couchdb:error,2012-11-26T4:34:41.914,ns_1@10.6.2.37:<0.29180.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2}

      ,

      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:41.914,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
      [couchdb:error,2012-11-26T4:34:41.921,ns_1@10.6.2.37:<0.29182.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6}

      ,

      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:41.923,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
      [ns_server:info,2012-11-26T4:34:41.927,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
      [ns_server:debug,2012-11-26T4:34:41.931,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
      [ns_server:debug,2012-11-26T4:34:41.934,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
      [ns_server:debug,2012-11-26T4:34:41.937,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:41.938,ns_1@10.6.2.37:<0.29266.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:41.941,ns_1@10.6.2.37:<0.29266.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [rebalance:info,2012-11-26T4:34:41.941,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:41.950,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
      [ns_server:debug,2012-11-26T4:34:41.954,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
      [ns_server:debug,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.27467.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
      [couchdb:error,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.29183.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}]
      [ns_server:debug,2012-11-26T4:34:41.962,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.27467.5201>
      [ns_server:debug,2012-11-26T4:34:41.968,ns_1@10.6.2.37:<0.29242.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
      [ns_server:info,2012-11-26T4:34:41.970,ns_1@10.6.2.37:<0.29269.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
      [ns_server:debug,2012-11-26T4:34:41.975,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.27467.5201>:
      {state,#Port<0.34659745>,#Port<0.34659734>,#Port<0.34659747>,
      #Port<0.34659737>,<0.27520.5201>,<<"cut off">>,<<"cut off">>,[],7,
      false,false,0,
      {1353,933281,923009},
      completed,
      {<0.29242.5201>,#Ref<0.0.47906.195375>},
      <<"replication_ns_1@10.6.2.37">>,<0.27467.5201>,
      {had_backfill,false,undefined,[]},
      false}.
      [error_logger:info,2012-11-26T4:34:41.976,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
      =========================PROGRESS REPORT=========================
      supervisor: {local,'ns_vbm_new_sup-saslbucket'}
      started: [{pid,<0.29269.5201>},
      {name,{new_child_id,"]^",'ns_1@10.6.2.40'}},
      {mfargs,
      {ebucketmigrator_srv,start_link,
      [{"10.6.2.40",11209},
      {"10.6.2.37",11209},
      [{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
      {on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
      {username,"saslbucket"},
      {password,"password"},
      {vbuckets,"]^"},
      {takeover,false},
      {suffix,"ns_1@10.6.2.37"}]]}},
      {restart_type,temporary},
      {shutdown,60000},
      {child_type,worker}]

      [ns_server:debug,2012-11-26T4:34:41.977,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29242.5201>,{#Ref<0.0.47906.195218>,<0.29269.5201>}}
      [ns_server:info,2012-11-26T4:34:41.979,ns_1@10.6.2.37:<0.25110.5201>:ns_memcached:do_handle_call:521]Changed vbucket 62 state to replica
      [views:debug,2012-11-26T4:34:41.985,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:465]
      Calling couch_set_view:mark_partitions_indexable([<<"saslbucket">>,
      <<"_design/d11">>,
      [0,1,2,3,4,5,6,7,8,9,10,11,
      12,13,14,15,16,17,18,19,20,
      21,22,23,24,25,26,27,28,29,
      30,31,32,33,34,35,36,37,38,
      39,40,41,42,43,44,45,46,47,
      48,49,50,51,52,53,54,55,56,
      57,58,59,60,61,62,63,64,65,
      66,67,68,69,70,71,72,73,74,
      75,76,77,78,79,80,81,82,83,
      84,85,86,87,88,89,90,91,92,
      93,94,95,96,97,98,99,100,
      101,102,103,104,105,106,
      107,108,109,110,111,112,
      113,114,115,116,117,118,
      119,120,121,122,123,124,
      125,126,127,128,129,130,
      131,132,133,134,135,136,
      137,138,139,140,141,142,
      143,144,145,146,147,148,
      149,150,151,152,153,154,
      155,156,157,158,159,160,
      161,162,163,164,165,166,
      167,168,169,170,171,172,
      173,174,175,176,177,178,
      179,180,181,182,183,184,
      185,186,187,188,189,190,
      191,192,193,194,195,196,
      197,198,199,200,201,202,
      203,204,205,206,207,208,
      209,210,211,212,213,214,
      215,216,217,218,219,220,
      221,222,223,224,225,226,
      227,228,229,230,231,232,
      233,234,235,236,237,238,
      239,240,241,242,243,244,
      245,246,247,248,249,250,
      251,252,253,254,255,256,
      257,258,259,260,261,262,
      263,264,265,266,267,268,
      269,270,271,272,273,274,
      275,276,277,278,279,280,
      281,282,283,284,285,286,
      287,288,289,290,291,292,
      293,294,295,296,297,298,
      299,300,301,302,303,304,
      305,306,307,308,309,310,
      311,312,313,314,315,316,
      317,318,319,320,321,322,
      323,324,325,326,327,328,
      329,330,331,332,333,334,
      335,336,337,338,339,340,
      341,342,343,344,345,346,
      347,348,349,350,351,352,
      353,354,355,356,357,358,
      359,360,361,362,363,364,
      365,366,367,368,369,370,
      371,372,373,374,375,376,
      377,378,379,380,381,382,
      383,384,385,386,387,388,
      389,390,391,392,393,394,
      395,396,397,398,399,400,
      401,402,403,404,405,406,
      407,408,409,410,411,412,
      413,414,415,416,417,418,
      419,420,421,422,423,424,
      425,426,427,428,429,430,
      431,432,433,434,435,436,
      437,438,439,440,441,442,
      443,444,445,446,447,448,
      449,450,451,452,453,454,
      455,456,457,458,459,460,
      461,462,463,464,465,466,
      467,468,469,470,471,472,
      473,474,475,476,477,478,
      479,480,481,482,483,484,
      485,486,487,488,489,490,
      491,492,493,494,495,496,
      497,498,499,500,501,502,
      503,504,505,506,507,508,
      509,510,511,512,513,514,
      515,516,517,518,519,520,
      521,522,523,524,525,526,
      527,528,529,530,531,532,
      533,534,535,536,537,538,
      539,540,541,542,543,544,
      545,546,547,548,549,550,
      551,552,553,554,555,556,
      557,558,559,560,561,562,
      563,564,565,566,567,568,
      569,570,571,572,573,574,
      575,576,577,578,579,580,
      581,582,583,584,585,586,
      587,588,589,590,591,592,
      593,594,595,596,597,598,
      599,600,601,602,603,604,
      605,606,607,608,609,610,
      611,612,613,614,615,616,
      617,618,619,620,621,622,
      623,624,625,626,627,628,
      629,630,631,632,633,634,
      635,636,637,638,639,640,
      641,642,643,644,645,646,
      647,648,649,650,651,652,
      653,654,655,656,657,658,
      659,660,661,662,663,664,
      665,666,667,668,669,670,
      671,672,673,674,675,676,
      677,678,679,680,681,682,
      683,684,685,686,687,688,
      689,690,691,692,693,694,
      695,696,697,698,699,700,
      701,702,703,704,705,706,
      707,708,709,710,711,712,
      713,714,715,716,717,718,
      719,720,721,722,723,724,
      725,726,727,728,729,730,
      731,732,733,734,735,736,
      737,738,739,740,741,742,
      743,744,745,746,747,748,
      749,750,751,752,753,754,
      755,756,757,758,759,760,
      761,762,763,764,765,766,
      767,768,769,770,771,772,
      773,774,775,776,777,778,
      779,780,781,782,783,784,
      785,786,787,788,789,790,
      791,792,793,794,795,796,
      797,798,799,800,801,802,
      803,804,805,806,807,808,
      809,810,811,812,813,814,
      815,816,817,818,819,820,
      821,822,823,824,825,826,
      827,828,829,830,831,832,
      833,834,835,836,837,838,
      839,840,841,842,843,844,
      845,846,847,848,849,850,
      851,852,853,854,855,856,
      857,858,859,860,861,862,
      863,864,865,866,867,868,
      869,870,871,872,873,874,
      875,876,877,878,879,880,
      881,882,883,884,885,886,
      887,888,889,890,891,892,
      893,894,895,896,897,898,
      899,900,901,902,903,904,
      905,906,907,908,909,910,
      911,912,913,914,915,916,
      917,918,919,920,921,922,
      923,924,925,926,927,928,
      929,930,931,932,933,934,
      935,936,937,938,939,940,
      941,942,943,944,945,946,
      947,948,949,950,951,952,
      953,954,955,956,957,958,
      959,960,961,962,963,964,
      965,966,967,968,969,970,
      971,972,973,974,975,976,
      977,978,979,980,981,982,
      983,984,985,986,987,988,
      989,990,991,992,993,994,
      995,996,997,998,999,1000,
      1001,1002,1003,1004,1005,
      1006,1007,1008,1009,1010,
      1011,1012,1013,1014,1015,
      1016,1017,1018,1019,1020,
      1021,1022,1023]])
      [ns_server:debug,2012-11-26T4:34:42.004,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:569]Reusing old upstream:
      [{vbuckets,"]^"},{name,<<"replication_ns_1@10.6.2.37">>},{takeover,false}]
      [views:debug,2012-11-26T4:34:41.993,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:466]
      couch_set_view:mark_partitions_indexable([<<"saslbucket">>,<<"_design/d11">>,
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,
      15,16,17,18,19,20,21,22,23,24,25,
      26,27,28,29,30,31,32,33,34,35,36,
      37,38,39,40,41,42,43,44,45,46,47,
      48,49,50,51,52,53,54,55,56,57,58,
      59,60,61,62,63,64,65,66,67,68,69,
      70,71,72,73,74,75,76,77,78,79,80,
      81,82,83,84,85,86,87,88,89,90,91,
      92,93,94,95,96,97,98,99,100,101,
      102,103,104,105,106,107,108,109,
      110,111,112,113,114,115,116,117,
      118,119,120,121,122,123,124,125,
      126,127,128,129,130,131,132,133,
      134,135,136,137,138,139,140,141,
      142,143,144,145,146,147,148,149,
      150,151,152,153,154,155,156,157,
      158,159,160,161,162,163,164,165,
      166,167,168,169,170,171,172,173,
      174,175,176,177,178,179,180,181,
      182,183,184,185,186,187,188,189,
      190,191,192,193,194,195,196,197,
      198,199,200,201,202,203,204,205,
      206,207,208,209,210,211,212,213,
      214,215,216,217,218,219,220,221,
      222,223,224,225,226,227,228,229,
      230,231,232,233,234,235,236,237,
      238,239,240,241,242,243,244,245,
      246,247,248,249,250,251,252,253,
      254,255,256,257,258,259,260,261,
      262,263,264,265,266,267,268,269,
      270,271,272,273,274,275,276,277,
      278,279,280,281,282,283,284,285,
      286,287,288,289,290,291,292,293,
      294,295,296,297,298,299,300,301,
      302,303,304,305,306,307,308,309,
      310,311,312,313,314,315,316,317,
      318,319,320,321,322,323,324,325,
      326,327,328,329,330,331,332,333,
      334,335,336,337,338,339,340,341,
      342,343,344,345,346,347,348,349,
      350,351,352,353,354,355,356,357,
      358,359,360,361,362,363,364,365,
      366,367,368,369,370,371,372,373,
      374,375,376,377,378,379,380,381,
      382,383,384,385,386,387,388,389,
      390,391,392,393,394,395,396,397,
      398,399,400,401,402,403,404,405,
      406,407,408,409,410,411,412,413,
      414,415,416,417,418,419,420,421,
      422,423,424,425,426,427,428,429,
      430,431,432,433,434,435,436,437,
      438,439,440,441,442,443,444,445,
      446,447,448,449,450,451,452,453,
      454,455,456,457,458,459,460,461,
      462,463,464,465,466,467,468,469,
      470,471,472,473,474,475,476,477,
      478,479,480,481,482,483,484,485,
      486,487,488,489,490,491,492,493,
      494,495,496,497,498,499,500,501,
      502,503,504,505,506,507,508,509,
      510,511,512,513,514,515,516,517,
      518,519,520,521,522,523,524,525,
      526,527,528,529,530,531,532,533,
      534,535,536,537,538,539,540,541,
      542,543,544,545,546,547,548,549,
      550,551,552,553,554,555,556,557,
      558,559,560,561,562,563,564,565,
      566,567,568,569,570,571,572,573,
      574,575,576,577,578,579,580,581,
      582,583,584,585,586,587,588,589,
      590,591,592,593,594,595,596,597,
      598,599,600,601,602,603,604,605,
      606,607,608,609,610,611,612,613,
      614,615,616,617,618,619,620,621,
      622,623,624,625,626,627,628,629,
      630,631,632,633,634,635,636,637,
      638,639,640,641,642,643,644,645,
      646,647,648,649,650,651,652,653,
      654,655,656,657,658,659,660,661,
      662,663,664,665,666,667,668,669,
      670,671,672,673,674,675,676,677,
      678,679,680,681,682,683,684,685,
      686,687,688,689,690,691,692,693,
      694,695,696,697,698,699,700,701,
      702,703,704,705,706,707,708,709,
      710,711,712,713,714,715,716,717,
      718,719,720,721,722,723,724,725,
      726,727,728,729,730,731,732,733,
      734,735,736,737,738,739,740,741,
      742,743,744,745,746,747,748,749,
      750,751,752,753,754,755,756,757,
      758,759,760,761,762,763,764,765,
      766,767,768,769,770,771,772,773,
      774,775,776,777,778,779,780,781,
      782,783,784,785,786,787,788,789,
      790,791,792,793,794,795,796,797,
      798,799,800,801,802,803,804,805,
      806,807,808,809,810,811,812,813,
      814,815,816,817,818,819,820,821,
      822,823,824,825,826,827,828,829,
      830,831,832,833,834,835,836,837,
      838,839,840,841,842,843,844,845,
      846,847,848,849,850,851,852,853,
      854,855,856,857,858,859,860,861,
      862,863,864,865,866,867,868,869,
      870,871,872,873,874,875,876,877,
      878,879,880,881,882,883,884,885,
      886,887,888,889,890,891,892,893,
      894,895,896,897,898,899,900,901,
      902,903,904,905,906,907,908,909,
      910,911,912,913,914,915,916,917,
      918,919,920,921,922,923,924,925,
      926,927,928,929,930,931,932,933,
      934,935,936,937,938,939,940,941,
      942,943,944,945,946,947,948,949,
      950,951,952,953,954,955,956,957,
      958,959,960,961,962,963,964,965,
      966,967,968,969,970,971,972,973,
      974,975,976,977,978,979,980,981,
      982,983,984,985,986,987,988,989,
      990,991,992,993,994,995,996,997,
      998,999,1000,1001,1002,1003,1004,
      1005,1006,1007,1008,1009,1010,1011,
      1012,1013,1014,1015,1016,1017,1018,
      1019,1020,1021,1022,1023]]) returned ok in 1ms
      [rebalance:debug,2012-11-26T4:34:42.005,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:init:588]upstream_sender pid: <0.29271.5201>
      [couchdb:error,2012-11-26T4:34:42.011,ns_1@10.6.2.37:<0.29184.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}

      ]
      [views:debug,2012-11-26T4:34:42.010,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:471]
      Calling couch_set_view:set_partition_states([<<"saslbucket">>,
      <<"_design/d11">>,
      [440],
      [441],
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,
      14,15,16,17,18,19,20,21,22,23,
      24,25,26,27,28,29,30,31,32,33,
      34,35,36,37,38,39,40,41,42,43,
      44,45,46,47,48,49,50,51,52,53,
      54,55,56,57,58,59,60,61,62,63,
      64,65,66,67,68,69,70,71,72,73,
      74,75,76,77,78,79,80,81,82,83,
      84,85,86,87,88,89,90,91,92,93,
      94,95,96,97,98,99,100,101,102,
      103,104,105,106,107,108,109,110,
      111,112,113,114,115,116,117,118,
      119,120,121,122,123,124,125,126,
      127,128,129,130,131,132,133,134,
      135,136,137,138,139,140,141,142,
      143,144,145,146,147,148,149,150,
      151,152,153,154,155,156,157,158,
      159,160,161,162,163,164,165,166,
      167,168,169,170,171,172,173,174,
      175,176,177,178,179,180,181,182,
      183,184,185,186,187,188,189,190,
      191,192,193,194,195,196,197,198,
      199,200,201,202,203,204,205,206,
      207,208,209,210,211,212,213,214,
      215,216,217,218,219,220,221,222,
      223,224,225,226,227,228,229,230,
      231,232,233,234,235,236,237,238,
      239,240,241,242,243,244,245,246,
      247,248,249,250,251,252,253,254,
      255,256,257,258,259,260,261,262,
      263,264,265,266,267,268,269,270,
      271,272,273,274,275,276,277,278,
      279,280,281,282,283,284,285,286,
      287,288,289,290,291,292,293,294,
      295,296,297,298,299,300,301,302,
      303,304,305,306,307,308,309,310,
      311,312,313,314,315,316,317,318,
      319,320,321,322,323,324,325,326,
      327,328,329,330,331,332,333,334,
      335,336,337,338,339,340,341,342,
      343,344,345,346,347,348,349,350,
      351,352,353,354,355,356,357,358,
      359,360,361,362,363,364,365,366,
      367,368,369,370,371,372,373,374,
      375,376,377,378,379,380,381,382,
      383,384,385,386,387,388,389,390,
      391,392,393,394,395,396,397,398,
      399,400,401,402,403,404,405,406,
      407,408,409,410,411,412,413,414,
      415,416,417,418,419,420,421,422,
      423,424,425,426,427,428,429,430,
      431,432,433,434,435,436,437,438,
      439,442,443,444,445,446,447,448,
      449,450,451,452,453,454,455,456,
      457,458,459,460,461,462,463,464,
      465,466,467,468,469,470,471,472,
      473,474,475,476,477,478,479,480,
      481,482,483,484,485,486,487,488,
      489,490,491,492,493,494,495,496,
      497,498,499,500,501,502,503,504,
      505,506,507,508,509,510,511,512,
      513,514,515,516,517,518,519,520,
      521,522,523,524,525,526,527,528,
      529,530,531,532,533,534,535,536,
      537,538,539,540,541,542,543,544,
      545,546,547,548,549,550,551,552,
      553,554,555,556,557,558,559,560,
      561,562,563,564,565,566,567,568,
      569,570,571,572,573,574,575,576,
      577,578,579,580,581,582,583,584,
      585,586,587,588,589,590,591,592,
      593,594,595,596,597,598,599,600,
      601,602,603,604,605,606,607,608,
      609,610,611,612,613,614,615,616,
      617,618,619,620,621,622,623,624,
      625,626,627,628,629,630,631,632,
      633,634,635,636,637,638,639,640,
      641,642,643,644,645,646,647,648,
      649,650,651,652,653,654,655,656,
      657,658,659,660,661,662,663,664,
      665,666,667,668,669,670,671,672,
      673,674,675,676,677,678,679,680,
      681,682,683,684,685,686,687,688,
      689,690,691,692,693,694,695,696,
      697,698,699,700,701,702,703,704,
      705,706,707,708,709,710,711,712,
      713,714,715,716,717,718,719,720,
      721,722,723,724,725,726,727,728,
      729,730,731,732,733,734,735,736,
      737,738,739,740,741,742,743,744,
      745,746,747,748,749,750,751,752,
      753,754,755,756,757,758,759,760,
      761,762,763,764,765,766,767,768,
      769,770,771,772,773,774,775,776,
      777,778,779,780,781,782,783,784,
      785,786,787,788,789,790,791,792,
      793,794,795,796,797,798,799,800,
      801,802,803,804,805,806,807,808,
      809,810,811,812,813,814,815,816,
      817,818,819,820,821,822,823,824,
      825,826,827,828,829,830,831,832,
      833,834,835,836,837,838,839,840,
      841,842,843,844,845,846,847,848,
      849,850,851,852,853,854,855,856,
      857,858,859,860,861,862,863,864,
      865,866,867,868,869,870,871,872,
      873,874,875,876,877,878,879,880,
      881,882,883,884,885,886,887,888,
      889,890,891,892,893,894,895,896,
      897,898,899,900,901,902,903,904,
      905,906,907,908,909,910,911,912,
      913,914,915,916,917,918,919,920,
      921,922,923,924,925,926,927,928,
      929,930,931,932,933,934,935,936,
      937,938,939,940,941,942,943,944,
      945,946,947,948,949,950,951,952,
      953,954,955,956,957,958,959,960,
      961,962,963,964,965,966,967,968,
      969,970,971,972,973,974,975,976,
      977,978,979,980,981,982,983,984,
      985,986,987,988,989,990,991,992,
      993,994,995,996,997,998,999,
      1000,1001,1002,1003,1004,1005,
      1006,1007,1008,1009,1010,1011,
      1012,1013,1014,1015,1016,1017,
      1018,1019,1020,1021,1022,1023]])
      [views:debug,2012-11-26T4:34:42.016,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:472]
      couch_set_view:set_partition_states([<<"saslbucket">>,<<"_design/d11">>,
      [440],
      [441],
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
      16,17,18,19,20,21,22,23,24,25,26,27,28,
      29,30,31,32,33,34,35,36,37,38,39,40,41,
      42,43,44,45,46,47,48,49,50,51,52,53,54,
      55,56,57,58,59,60,61,62,63,64,65,66,67,
      68,69,70,71,72,73,74,75,76,77,78,79,80,
      81,82,83,84,85,86,87,88,89,90,91,92,93,
      94,95,96,97,98,99,100,101,102,103,104,
      105,106,107,108,109,110,111,112,113,114,
      115,116,117,118,119,120,121,122,123,124,
      125,126,127,128,129,130,131,132,133,134,
      135,136,137,138,139,140,141,142,143,144,
      145,146,147,148,149,150,151,152,153,154,
      155,156,157,158,159,160,161,162,163,164,
      165,166,167,168,169,170,171,172,173,174,
      175,176,177,178,179,180,181,182,183,184,
      185,186,187,188,189,190,191,192,193,194,
      195,196,197,198,199,200,201,202,203,204,
      205,206,207,208,209,210,211,212,213,214,
      215,216,217,218,219,220,221,222,223,224,
      225,226,227,228,229,230,231,232,233,234,
      235,236,237,238,239,240,241,242,243,244,
      245,246,247,248,249,250,251,252,253,254,
      255,256,257,258,259,260,261,262,263,264,
      265,266,267,268,269,270,271,272,273,274,
      275,276,277,278,279,280,281,282,283,284,
      285,286,287,288,289,290,291,292,293,294,
      295,296,297,298,299,300,301,302,303,304,
      305,306,307,308,309,310,311,312,313,314,
      315,316,317,318,319,320,321,322,323,324,
      325,326,327,328,329,330,331,332,333,334,
      335,336,337,338,339,340,341,342,343,344,
      345,346,347,348,349,350,351,352,353,354,
      355,356,357,358,359,360,361,362,363,364,
      365,366,367,368,369,370,371,372,373,374,
      375,376,377,378,379,380,381,382,383,384,
      385,386,387,388,389,390,391,392,393,394,
      395,396,397,398,399,400,401,402,403,404,
      405,406,407,408,409,410,411,412,413,414,
      415,416,417,418,419,420,421,422,423,424,
      425,426,427,428,429,430,431,432,433,434,
      435,436,437,438,439,442,443,444,445,446,
      447,448,449,450,451,452,453,454,455,456,
      457,458,459,460,461,462,463,464,465,466,
      467,468,469,470,471,472,473,474,475,476,
      477,478,479,480,481,482,483,484,485,486,
      487,488,489,490,491,492,493,494,495,496,
      497,498,499,500,501,502,503,504,505,506,
      507,508,509,510,511,512,513,514,515,516,
      517,518,519,520,521,522,523,524,525,526,
      527,528,529,530,531,532,533,534,535,536,
      537,538,539,540,541,542,543,544,545,546,
      547,548,549,550,551,552,553,554,555,556,
      557,558,559,560,561,562,563,564,565,566,
      567,568,569,570,571,572,573,574,575,576,
      577,578,579,580,581,582,583,584,585,586,
      587,588,589,590,591,592,593,594,595,596,
      597,598,599,600,601,602,603,604,605,606,
      607,608,609,610,611,612,613,614,615,616,
      617,618,619,620,621,622,623,624,625,626,
      627,628,629,630,631,632,633,634,635,636,
      637,638,639,640,641,642,643,644,645,646,
      647,648,649,650,651,652,653,654,655,656,
      657,658,659,660,661,662,663,664,665,666,
      667,668,669,670,671,672,673,674,675,676,
      677,678,679,680,681,682,683,684,685,686,
      687,688,689,690,691,692,693,694,695,696,
      697,698,699,700,701,702,703,704,705,706,
      707,708,709,710,711,712,713,714,715,716,
      717,718,719,720,721,722,723,724,725,726,
      727,728,729,730,731,732,733,734,735,736,
      737,738,739,740,741,742,743,744,745,746,
      747,748,749,750,751,752,753,754,755,756,
      757,758,759,760,761,762,763,764,765,766,
      767,768,769,770,771,772,773,774,775,776,
      777,778,779,780,781,782,783,784,785,786,
      787,788,789,790,791,792,793,794,795,796,
      797,798,799,800,801,802,803,804,805,806,
      807,808,809,810,811,812,813,814,815,816,
      817,818,819,820,821,822,823,824,825,826,
      827,828,829,830,831,832,833,834,835,836,
      837,838,839,840,841,842,843,844,845,846,
      847,848,849,850,851,852,853,854,855,856,
      857,858,859,860,861,862,863,864,865,866,
      867,868,869,870,871,872,873,874,875,876,
      877,878,879,880,881,882,883,884,885,886,
      887,888,889,890,891,892,893,894,895,896,
      897,898,899,900,901,902,903,904,905,906,
      907,908,909,910,911,912,913,914,915,916,
      917,918,919,920,921,922,923,924,925,926,
      927,928,929,930,931,932,933,934,935,936,
      937,938,939,940,941,942,943,944,945,946,
      947,948,949,950,951,952,953,954,955,956,
      957,958,959,960,961,962,963,964,965,966,
      967,968,969,970,971,972,973,974,975,976,
      977,978,979,980,981,982,983,984,985,986,
      987,988,989,990,991,992,993,994,995,996,
      997,998,999,1000,1001,1002,1003,1004,
      1005,1006,1007,1008,1009,1010,1011,1012,
      1013,1014,1015,1016,1017,1018,1019,1020,
      1021,1022,1023]]) returned ok in 1ms
      [views:debug,2012-11-26T4:34:42.022,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:488]
      Calling couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
      <<"_design/d11">>,[]])
      [ns_server:info,2012-11-26T4:34:42.023,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:41.821519 PST 3: TAP (Consumer) eq_tapq:anon_22 - Reset vbucket 163 was completed succecssfully.

      [views:debug,2012-11-26T4:34:42.024,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:489]
      couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
      <<"_design/d11">>,[]]) returned ok in 0ms
      [rebalance:debug,2012-11-26T4:34:42.025,ns_1@10.6.2.37:<0.29272.5201>:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 50 in vbucket 163
      [ns_server:debug,2012-11-26T4:34:42.076,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.29272.5201>
      [ns_server:debug,2012-11-26T4:34:42.076,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28872.5201>
      [ns_server:debug,2012-11-26T4:34:42.077,ns_1@10.6.2.37:<0.28887.5201>:capi_set_view_manager:do_wait_index_updated:624]Caller died:

      {'DOWN',#Ref<0.0.47906.158245>,process,<0.28872.5201>,shutdown}

      [ns_server:debug,2012-11-26T4:34:42.077,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28397.5201>
      [ns_server:debug,2012-11-26T4:34:42.078,ns_1@10.6.2.37:janitor_agent-saslbucket<0.25117.5201>:janitor_agent:set_rebalance_mref:719]Killing rebalance-related subprocess: <0.28008.5201>
      [ns_server:debug,2012-11-26T4:34:42.084,ns_1@10.6.2.37:<0.28321.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.084,ns_1@10.6.2.37:<0.29297.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [views:debug,2012-11-26T4:34:42.083,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:465]
      Calling couch_set_view:mark_partitions_indexable([<<"saslbucket">>,
      <<"_design/d11">>,
      [0,1,2,3,4,5,6,7,8,9,10,11,
      12,13,14,15,16,17,18,19,20,
      21,22,23,24,25,26,27,28,29,
      30,31,32,33,34,35,36,37,38,
      39,40,41,42,43,44,45,46,47,
      48,49,50,51,52,53,54,55,56,
      57,58,59,60,61,62,63,64,65,
      66,67,68,69,70,71,72,73,74,
      75,76,77,78,79,80,81,82,83,
      84,85,86,87,88,89,90,91,92,
      93,94,95,96,97,98,99,100,
      101,102,103,104,105,106,
      107,108,109,110,111,112,
      113,114,115,116,117,118,
      119,120,121,122,123,124,
      125,126,127,128,129,130,
      131,132,133,134,135,136,
      137,138,139,140,141,142,
      143,144,145,146,147,148,
      149,150,151,152,153,154,
      155,156,157,158,159,160,
      161,162,163,164,165,166,
      167,168,169,170,171,172,
      173,174,175,176,177,178,
      179,180,181,182,183,184,
      185,186,187,188,189,190,
      191,192,193,194,195,196,
      197,198,199,200,201,202,
      203,204,205,206,207,208,
      209,210,211,212,213,214,
      215,216,217,218,219,220,
      221,222,223,224,225,226,
      227,228,229,230,231,232,
      233,234,235,236,237,238,
      239,240,241,242,243,244,
      245,246,247,248,249,250,
      251,252,253,254,255,256,
      257,258,259,260,261,262,
      263,264,265,266,267,268,
      269,270,271,272,273,274,
      275,276,277,278,279,280,
      281,282,283,284,285,286,
      287,288,289,290,291,292,
      293,294,295,296,297,298,
      299,300,301,302,303,304,
      305,306,307,308,309,310,
      311,312,313,314,315,316,
      317,318,319,320,321,322,
      323,324,325,326,327,328,
      329,330,331,332,333,334,
      335,336,337,338,339,340,
      341,342,343,344,345,346,
      347,348,349,350,351,352,
      353,354,355,356,357,358,
      359,360,361,362,363,364,
      365,366,367,368,369,370,
      371,372,373,374,375,376,
      377,378,379,380,381,382,
      383,384,385,386,387,388,
      389,390,391,392,393,394,
      395,396,397,398,399,400,
      401,402,403,404,405,406,
      407,408,409,410,411,412,
      413,414,415,416,417,418,
      419,420,421,422,423,424,
      425,426,427,428,429,430,
      431,432,433,434,435,436,
      437,438,439,440,441,442,
      443,444,445,446,447,448,
      449,450,451,452,453,454,
      455,456,457,458,459,460,
      461,462,463,464,465,466,
      467,468,469,470,471,472,
      473,474,475,476,477,478,
      479,480,481,482,483,484,
      485,486,487,488,489,490,
      491,492,493,494,495,496,
      497,498,499,500,501,502,
      503,504,505,506,507,508,
      509,510,511,512,513,514,
      515,516,517,518,519,520,
      521,522,523,524,525,526,
      527,528,529,530,531,532,
      533,534,535,536,537,538,
      539,540,541,542,543,544,
      545,546,547,548,549,550,
      551,552,553,554,555,556,
      557,558,559,560,561,562,
      563,564,565,566,567,568,
      569,570,571,572,573,574,
      575,576,577,578,579,580,
      581,582,583,584,585,586,
      587,588,589,590,591,592,
      593,594,595,596,597,598,
      599,600,601,602,603,604,
      605,606,607,608,609,610,
      611,612,613,614,615,616,
      617,618,619,620,621,622,
      623,624,625,626,627,628,
      629,630,631,632,633,634,
      635,636,637,638,639,640,
      641,642,643,644,645,646,
      647,648,649,650,651,652,
      653,654,655,656,657,658,
      659,660,661,662,663,664,
      665,666,667,668,669,670,
      671,672,673,674,675,676,
      677,678,679,680,681,682,
      683,684,685,686,687,688,
      689,690,691,692,693,694,
      695,696,697,698,699,700,
      701,702,703,704,705,706,
      707,708,709,710,711,712,
      713,714,715,716,717,718,
      719,720,721,722,723,724,
      725,726,727,728,729,730,
      731,732,733,734,735,736,
      737,738,739,740,741,742,
      743,744,745,746,747,748,
      749,750,751,752,753,754,
      755,756,757,758,759,760,
      761,762,763,764,765,766,
      767,768,769,770,771,772,
      773,774,775,776,777,778,
      779,780,781,782,783,784,
      785,786,787,788,789,790,
      791,792,793,794,795,796,
      797,798,799,800,801,802,
      803,804,805,806,807,808,
      809,810,811,812,813,814,
      815,816,817,818,819,820,
      821,822,823,824,825,826,
      827,828,829,830,831,832,
      833,834,835,836,837,838,
      839,840,841,842,843,844,
      845,846,847,848,849,850,
      851,852,853,854,855,856,
      857,858,859,860,861,862,
      863,864,865,866,867,868,
      869,870,871,872,873,874,
      875,876,877,878,879,880,
      881,882,883,884,885,886,
      887,888,889,890,891,892,
      893,894,895,896,897,898,
      899,900,901,902,903,904,
      905,906,907,908,909,910,
      911,912,913,914,915,916,
      917,918,919,920,921,922,
      923,924,925,926,927,928,
      929,930,931,932,933,934,
      935,936,937,938,939,940,
      941,942,943,944,945,946,
      947,948,949,950,951,952,
      953,954,955,956,957,958,
      959,960,961,962,963,964,
      965,966,967,968,969,970,
      971,972,973,974,975,976,
      977,978,979,980,981,982,
      983,984,985,986,987,988,
      989,990,991,992,993,994,
      995,996,997,998,999,1000,
      1001,1002,1003,1004,1005,
      1006,1007,1008,1009,1010,
      1011,1012,1013,1014,1015,
      1016,1017,1018,1019,1020,
      1021,1022,1023]])
      [ns_server:debug,2012-11-26T4:34:42.087,ns_1@10.6.2.37:<0.29297.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [rebalance:info,2012-11-26T4:34:42.088,ns_1@10.6.2.37:<0.28321.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:42.092,ns_1@10.6.2.37:<0.27922.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.092,ns_1@10.6.2.37:<0.29314.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:42.101,ns_1@10.6.2.37:<0.29314.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [views:debug,2012-11-26T4:34:42.091,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:466]
      couch_set_view:mark_partitions_indexable([<<"saslbucket">>,<<"_design/d11">>,
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,
      15,16,17,18,19,20,21,22,23,24,25,
      26,27,28,29,30,31,32,33,34,35,36,
      37,38,39,40,41,42,43,44,45,46,47,
      48,49,50,51,52,53,54,55,56,57,58,
      59,60,61,62,63,64,65,66,67,68,69,
      70,71,72,73,74,75,76,77,78,79,80,
      81,82,83,84,85,86,87,88,89,90,91,
      92,93,94,95,96,97,98,99,100,101,
      102,103,104,105,106,107,108,109,
      110,111,112,113,114,115,116,117,
      118,119,120,121,122,123,124,125,
      126,127,128,129,130,131,132,133,
      134,135,136,137,138,139,140,141,
      142,143,144,145,146,147,148,149,
      150,151,152,153,154,155,156,157,
      158,159,160,161,162,163,164,165,
      166,167,168,169,170,171,172,173,
      174,175,176,177,178,179,180,181,
      182,183,184,185,186,187,188,189,
      190,191,192,193,194,195,196,197,
      198,199,200,201,202,203,204,205,
      206,207,208,209,210,211,212,213,
      214,215,216,217,218,219,220,221,
      222,223,224,225,226,227,228,229,
      230,231,232,233,234,235,236,237,
      238,239,240,241,242,243,244,245,
      246,247,248,249,250,251,252,253,
      254,255,256,257,258,259,260,261,
      262,263,264,265,266,267,268,269,
      270,271,272,273,274,275,276,277,
      278,279,280,281,282,283,284,285,
      286,287,288,289,290,291,292,293,
      294,295,296,297,298,299,300,301,
      302,303,304,305,306,307,308,309,
      310,311,312,313,314,315,316,317,
      318,319,320,321,322,323,324,325,
      326,327,328,329,330,331,332,333,
      334,335,336,337,338,339,340,341,
      342,343,344,345,346,347,348,349,
      350,351,352,353,354,355,356,357,
      358,359,360,361,362,363,364,365,
      366,367,368,369,370,371,372,373,
      374,375,376,377,378,379,380,381,
      382,383,384,385,386,387,388,389,
      390,391,392,393,394,395,396,397,
      398,399,400,401,402,403,404,405,
      406,407,408,409,410,411,412,413,
      414,415,416,417,418,419,420,421,
      422,423,424,425,426,427,428,429,
      430,431,432,433,434,435,436,437,
      438,439,440,441,442,443,444,445,
      446,447,448,449,450,451,452,453,
      454,455,456,457,458,459,460,461,
      462,463,464,465,466,467,468,469,
      470,471,472,473,474,475,476,477,
      478,479,480,481,482,483,484,485,
      486,487,488,489,490,491,492,493,
      494,495,496,497,498,499,500,501,
      502,503,504,505,506,507,508,509,
      510,511,512,513,514,515,516,517,
      518,519,520,521,522,523,524,525,
      526,527,528,529,530,531,532,533,
      534,535,536,537,538,539,540,541,
      542,543,544,545,546,547,548,549,
      550,551,552,553,554,555,556,557,
      558,559,560,561,562,563,564,565,
      566,567,568,569,570,571,572,573,
      574,575,576,577,578,579,580,581,
      582,583,584,585,586,587,588,589,
      590,591,592,593,594,595,596,597,
      598,599,600,601,602,603,604,605,
      606,607,608,609,610,611,612,613,
      614,615,616,617,618,619,620,621,
      622,623,624,625,626,627,628,629,
      630,631,632,633,634,635,636,637,
      638,639,640,641,642,643,644,645,
      646,647,648,649,650,651,652,653,
      654,655,656,657,658,659,660,661,
      662,663,664,665,666,667,668,669,
      670,671,672,673,674,675,676,677,
      678,679,680,681,682,683,684,685,
      686,687,688,689,690,691,692,693,
      694,695,696,697,698,699,700,701,
      702,703,704,705,706,707,708,709,
      710,711,712,713,714,715,716,717,
      718,719,720,721,722,723,724,725,
      726,727,728,729,730,731,732,733,
      734,735,736,737,738,739,740,741,
      742,743,744,745,746,747,748,749,
      750,751,752,753,754,755,756,757,
      758,759,760,761,762,763,764,765,
      766,767,768,769,770,771,772,773,
      774,775,776,777,778,779,780,781,
      782,783,784,785,786,787,788,789,
      790,791,792,793,794,795,796,797,
      798,799,800,801,802,803,804,805,
      806,807,808,809,810,811,812,813,
      814,815,816,817,818,819,820,821,
      822,823,824,825,826,827,828,829,
      830,831,832,833,834,835,836,837,
      838,839,840,841,842,843,844,845,
      846,847,848,849,850,851,852,853,
      854,855,856,857,858,859,860,861,
      862,863,864,865,866,867,868,869,
      870,871,872,873,874,875,876,877,
      878,879,880,881,882,883,884,885,
      886,887,888,889,890,891,892,893,
      894,895,896,897,898,899,900,901,
      902,903,904,905,906,907,908,909,
      910,911,912,913,914,915,916,917,
      918,919,920,921,922,923,924,925,
      926,927,928,929,930,931,932,933,
      934,935,936,937,938,939,940,941,
      942,943,944,945,946,947,948,949,
      950,951,952,953,954,955,956,957,
      958,959,960,961,962,963,964,965,
      966,967,968,969,970,971,972,973,
      974,975,976,977,978,979,980,981,
      982,983,984,985,986,987,988,989,
      990,991,992,993,994,995,996,997,
      998,999,1000,1001,1002,1003,1004,
      1005,1006,1007,1008,1009,1010,1011,
      1012,1013,1014,1015,1016,1017,1018,
      1019,1020,1021,1022,1023]]) returned ok in 3ms
      [rebalance:info,2012-11-26T4:34:42.105,ns_1@10.6.2.37:<0.27922.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:42.110,ns_1@10.6.2.37:compaction_daemon<0.23916.5201>:compaction_daemon:handle_info:353]Got config_changed in state idle. Nothing to do since compaction is not running
      [ns_server:debug,2012-11-26T4:34:42.112,ns_1@10.6.2.37:ns_config_log<0.768.0>:ns_config_log:log_common:111]config change:
      buckets ->
      [{configs,[{"default",
      [

      {map,[]}

      ,

      {fastForwardMap,[]},
      {uuid,<<"ef2292c303b134a8debe20aa2f604c2f">>},
      {sasl_password,[]},
      {num_replicas,1},
      {replica_index,false},
      {ram_quota,10485760000},
      {auth_type,sasl},
      {autocompaction,false},
      {flush_enabled,false},
      {type,membase},
      {num_vbuckets,1024},
      {servers,['ns_1@10.6.2.38','ns_1@10.6.2.39','ns_1@10.6.2.40', 'ns_1@10.6.2.42','ns_1@10.6.2.44','ns_1@10.6.2.45', 'ns_1@10.6.2.43']}]},
      {"saslbucket",
      [{map,[{94, ['ns_1@10.6.2.40','ns_1@10.6.2.45'], ['ns_1@10.6.2.40','ns_1@10.6.2.37']}]},
      {fastForwardMap,[]}

      ,

      {uuid,<<"e344089af6ea1f8153912ad043034083">>}

      ,

      {num_replicas,1}

      ,

      {replica_index,false}

      ,

      {ram_quota,10485760000}

      ,

      {auth_type,sasl}

      ,

      {sasl_password,"password"}

      ,

      {autocompaction,false}

      ,

      {flush_enabled,false}

      ,

      {type,membase}

      ,

      {num_vbuckets,1024}

      ,

      {servers,['ns_1@10.6.2.38','ns_1@10.6.2.39','ns_1@10.6.2.40', 'ns_1@10.6.2.42','ns_1@10.6.2.43','ns_1@10.6.2.44', 'ns_1@10.6.2.37','ns_1@10.6.2.45']}

      ]}]}]
      [views:debug,2012-11-26T4:34:42.120,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:471]
      Calling couch_set_view:set_partition_states([<<"saslbucket">>,
      <<"_design/d11">>,
      [440],
      [],
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,
      14,15,16,17,18,19,20,21,22,23,
      24,25,26,27,28,29,30,31,32,33,
      34,35,36,37,38,39,40,41,42,43,
      44,45,46,47,48,49,50,51,52,53,
      54,55,56,57,58,59,60,61,62,63,
      64,65,66,67,68,69,70,71,72,73,
      74,75,76,77,78,79,80,81,82,83,
      84,85,86,87,88,89,90,91,92,93,
      94,95,96,97,98,99,100,101,102,
      103,104,105,106,107,108,109,110,
      111,112,113,114,115,116,117,118,
      119,120,121,122,123,124,125,126,
      127,128,129,130,131,132,133,134,
      135,136,137,138,139,140,141,142,
      143,144,145,146,147,148,149,150,
      151,152,153,154,155,156,157,158,
      159,160,161,162,163,164,165,166,
      167,168,169,170,171,172,173,174,
      175,176,177,178,179,180,181,182,
      183,184,185,186,187,188,189,190,
      191,192,193,194,195,196,197,198,
      199,200,201,202,203,204,205,206,
      207,208,209,210,211,212,213,214,
      215,216,217,218,219,220,221,222,
      223,224,225,226,227,228,229,230,
      231,232,233,234,235,236,237,238,
      239,240,241,242,243,244,245,246,
      247,248,249,250,251,252,253,254,
      255,256,257,258,259,260,261,262,
      263,264,265,266,267,268,269,270,
      271,272,273,274,275,276,277,278,
      279,280,281,282,283,284,285,286,
      287,288,289,290,291,292,293,294,
      295,296,297,298,299,300,301,302,
      303,304,305,306,307,308,309,310,
      311,312,313,314,315,316,317,318,
      319,320,321,322,323,324,325,326,
      327,328,329,330,331,332,333,334,
      335,336,337,338,339,340,341,342,
      343,344,345,346,347,348,349,350,
      351,352,353,354,355,356,357,358,
      359,360,361,362,363,364,365,366,
      367,368,369,370,371,372,373,374,
      375,376,377,378,379,380,381,382,
      383,384,385,386,387,388,389,390,
      391,392,393,394,395,396,397,398,
      399,400,401,402,403,404,405,406,
      407,408,409,410,411,412,413,414,
      415,416,417,418,419,420,421,422,
      423,424,425,426,427,428,429,430,
      431,432,433,434,435,436,437,438,
      439,441,442,443,444,445,446,447,
      448,449,450,451,452,453,454,455,
      456,457,458,459,460,461,462,463,
      464,465,466,467,468,469,470,471,
      472,473,474,475,476,477,478,479,
      480,481,482,483,484,485,486,487,
      488,489,490,491,492,493,494,495,
      496,497,498,499,500,501,502,503,
      504,505,506,507,508,509,510,511,
      512,513,514,515,516,517,518,519,
      520,521,522,523,524,525,526,527,
      528,529,530,531,532,533,534,535,
      536,537,538,539,540,541,542,543,
      544,545,546,547,548,549,550,551,
      552,553,554,555,556,557,558,559,
      560,561,562,563,564,565,566,567,
      568,569,570,571,572,573,574,575,
      576,577,578,579,580,581,582,583,
      584,585,586,587,588,589,590,591,
      592,593,594,595,596,597,598,599,
      600,601,602,603,604,605,606,607,
      608,609,610,611,612,613,614,615,
      616,617,618,619,620,621,622,623,
      624,625,626,627,628,629,630,631,
      632,633,634,635,636,637,638,639,
      640,641,642,643,644,645,646,647,
      648,649,650,651,652,653,654,655,
      656,657,658,659,660,661,662,663,
      664,665,666,667,668,669,670,671,
      672,673,674,675,676,677,678,679,
      680,681,682,683,684,685,686,687,
      688,689,690,691,692,693,694,695,
      696,697,698,699,700,701,702,703,
      704,705,706,707,708,709,710,711,
      712,713,714,715,716,717,718,719,
      720,721,722,723,724,725,726,727,
      728,729,730,731,732,733,734,735,
      736,737,738,739,740,741,742,743,
      744,745,746,747,748,749,750,751,
      752,753,754,755,756,757,758,759,
      760,761,762,763,764,765,766,767,
      768,769,770,771,772,773,774,775,
      776,777,778,779,780,781,782,783,
      784,785,786,787,788,789,790,791,
      792,793,794,795,796,797,798,799,
      800,801,802,803,804,805,806,807,
      808,809,810,811,812,813,814,815,
      816,817,818,819,820,821,822,823,
      824,825,826,827,828,829,830,831,
      832,833,834,835,836,837,838,839,
      840,841,842,843,844,845,846,847,
      848,849,850,851,852,853,854,855,
      856,857,858,859,860,861,862,863,
      864,865,866,867,868,869,870,871,
      872,873,874,875,876,877,878,879,
      880,881,882,883,884,885,886,887,
      888,889,890,891,892,893,894,895,
      896,897,898,899,900,901,902,903,
      904,905,906,907,908,909,910,911,
      912,913,914,915,916,917,918,919,
      920,921,922,923,924,925,926,927,
      928,929,930,931,932,933,934,935,
      936,937,938,939,940,941,942,943,
      944,945,946,947,948,949,950,951,
      952,953,954,955,956,957,958,959,
      960,961,962,963,964,965,966,967,
      968,969,970,971,972,973,974,975,
      976,977,978,979,980,981,982,983,
      984,985,986,987,988,989,990,991,
      992,993,994,995,996,997,998,999,
      1000,1001,1002,1003,1004,1005,
      1006,1007,1008,1009,1010,1011,
      1012,1013,1014,1015,1016,1017,
      1018,1019,1020,1021,1022,1023]])
      [rebalance:error,2012-11-26T4:34:42.126,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:handle_info:420]killing myself due to exit signal: {'EXIT',<0.29220.5201>,
      badmatch,{error,closed,
      [

      {ebucketmigrator_srv, upstream_sender_loop,1}

      ]}}
      [ns_server:debug,2012-11-26T4:34:42.129,ns_1@10.6.2.37:<0.26815.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.130,ns_1@10.6.2.37:<0.29318.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:42.138,ns_1@10.6.2.37:<0.29318.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [ns_server:debug,2012-11-26T4:34:42.139,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.139,ns_1@10.6.2.37:<0.29319.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:42.147,ns_1@10.6.2.37:<0.29319.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [rebalance:info,2012-11-26T4:34:42.149,ns_1@10.6.2.37:<0.29175.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [rebalance:info,2012-11-26T4:34:42.159,ns_1@10.6.2.37:<0.26815.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:42.183,ns_1@10.6.2.37:ns_config_rep<0.23781.5201>:ns_config_rep:handle_call:113]Got full synchronization request from 'ns_1@10.6.2.38'
      [ns_server:debug,2012-11-26T4:34:42.201,ns_1@10.6.2.37:ns_config_rep<0.23781.5201>:ns_config_rep:handle_call:119]Fully synchronized config in 497 us
      [ns_server:info,2012-11-26T4:34:42.202,ns_1@10.6.2.37:<0.29316.5201>:ebucketmigrator_srv:init:492]Setting

      {"10.6.2.37",11209} vbucket 62 to state replica
      [views:debug,2012-11-26T4:34:42.204,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:472]
      couch_set_view:set_partition_states([<<"saslbucket">>,<<"_design/d11">>,
      [440],
      [],
      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
      16,17,18,19,20,21,22,23,24,25,26,27,28,
      29,30,31,32,33,34,35,36,37,38,39,40,41,
      42,43,44,45,46,47,48,49,50,51,52,53,54,
      55,56,57,58,59,60,61,62,63,64,65,66,67,
      68,69,70,71,72,73,74,75,76,77,78,79,80,
      81,82,83,84,85,86,87,88,89,90,91,92,93,
      94,95,96,97,98,99,100,101,102,103,104,
      105,106,107,108,109,110,111,112,113,114,
      115,116,117,118,119,120,121,122,123,124,
      125,126,127,128,129,130,131,132,133,134,
      135,136,137,138,139,140,141,142,143,144,
      145,146,147,148,149,150,151,152,153,154,
      155,156,157,158,159,160,161,162,163,164,
      165,166,167,168,169,170,171,172,173,174,
      175,176,177,178,179,180,181,182,183,184,
      185,186,187,188,189,190,191,192,193,194,
      195,196,197,198,199,200,201,202,203,204,
      205,206,207,208,209,210,211,212,213,214,
      215,216,217,218,219,220,221,222,223,224,
      225,226,227,228,229,230,231,232,233,234,
      235,236,237,238,239,240,241,242,243,244,
      245,246,247,248,249,250,251,252,253,254,
      255,256,257,258,259,260,261,262,263,264,
      265,266,267,268,269,270,271,272,273,274,
      275,276,277,278,279,280,281,282,283,284,
      285,286,287,288,289,290,291,292,293,294,
      295,296,297,298,299,300,301,302,303,304,
      305,306,307,308,309,310,311,312,313,314,
      315,316,317,318,319,320,321,322,323,324,
      325,326,327,328,329,330,331,332,333,334,
      335,336,337,338,339,340,341,342,343,344,
      345,346,347,348,349,350,351,352,353,354,
      355,356,357,358,359,360,361,362,363,364,
      365,366,367,368,369,370,371,372,373,374,
      375,376,377,378,379,380,381,382,383,384,
      385,386,387,388,389,390,391,392,393,394,
      395,396,397,398,399,400,401,402,403,404,
      405,406,407,408,409,410,411,412,413,414,
      415,416,417,418,419,420,421,422,423,424,
      425,426,427,428,429,430,431,432,433,434,
      435,436,437,438,439,441,442,443,444,445,
      446,447,448,449,450,451,452,453,454,455,
      456,457,458,459,460,461,462,463,464,465,
      466,467,468,469,470,471,472,473,474,475,
      476,477,478,479,480,481,482,483,484,485,
      486,487,488,489,490,491,492,493,494,495,
      496,497,498,499,500,501,502,503,504,505,
      506,507,508,509,510,511,512,513,514,515,
      516,517,518,519,520,521,522,523,524,525,
      526,527,528,529,530,531,532,533,534,535,
      536,537,538,539,540,541,542,543,544,545,
      546,547,548,549,550,551,552,553,554,555,
      556,557,558,559,560,561,562,563,564,565,
      566,567,568,569,570,571,572,573,574,575,
      576,577,578,579,580,581,582,583,584,585,
      586,587,588,589,590,591,592,593,594,595,
      596,597,598,599,600,601,602,603,604,605,
      606,607,608,609,610,611,612,613,614,615,
      616,617,618,619,620,621,622,623,624,625,
      626,627,628,629,630,631,632,633,634,635,
      636,637,638,639,640,641,642,643,644,645,
      646,647,648,649,650,651,652,653,654,655,
      656,657,658,659,660,661,662,663,664,665,
      666,667,668,669,670,671,672,673,674,675,
      676,677,678,679,680,681,682,683,684,685,
      686,687,688,689,690,691,692,693,694,695,
      696,697,698,699,700,701,702,703,704,705,
      706,707,708,709,710,711,712,713,714,715,
      716,717,718,719,720,721,722,723,724,725,
      726,727,728,729,730,731,732,733,734,735,
      736,737,738,739,740,741,742,743,744,745,
      746,747,748,749,750,751,752,753,754,755,
      756,757,758,759,760,761,762,763,764,765,
      766,767,768,769,770,771,772,773,774,775,
      776,777,778,779,780,781,782,783,784,785,
      786,787,788,789,790,791,792,793,794,795,
      796,797,798,799,800,801,802,803,804,805,
      806,807,808,809,810,811,812,813,814,815,
      816,817,818,819,820,821,822,823,824,825,
      826,827,828,829,830,831,832,833,834,835,
      836,837,838,839,840,841,842,843,844,845,
      846,847,848,849,850,851,852,853,854,855,
      856,857,858,859,860,861,862,863,864,865,
      866,867,868,869,870,871,872,873,874,875,
      876,877,878,879,880,881,882,883,884,885,
      886,887,888,889,890,891,892,893,894,895,
      896,897,898,899,900,901,902,903,904,905,
      906,907,908,909,910,911,912,913,914,915,
      916,917,918,919,920,921,922,923,924,925,
      926,927,928,929,930,931,932,933,934,935,
      936,937,938,939,940,941,942,943,944,945,
      946,947,948,949,950,951,952,953,954,955,
      956,957,958,959,960,961,962,963,964,965,
      966,967,968,969,970,971,972,973,974,975,
      976,977,978,979,980,981,982,983,984,985,
      986,987,988,989,990,991,992,993,994,995,
      996,997,998,999,1000,1001,1002,1003,
      1004,1005,1006,1007,1008,1009,1010,1011,
      1012,1013,1014,1015,1016,1017,1018,1019,
      1020,1021,1022,1023]]) returned ok in 80ms
      [views:debug,2012-11-26T4:34:42.226,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:488]
      Calling couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
      <<"_design/d11">>,[]])
      [views:debug,2012-11-26T4:34:42.239,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:apply_index_states:489]
      couch_set_view:mark_partitions_unindexable([<<"saslbucket">>,
      <<"_design/d11">>,[]]) returned ok in 0ms
      [ns_server:debug,2012-11-26T4:34:42.262,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
      [ns_server:info,2012-11-26T4:34:42.266,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:42.040152 PST 3: Deletion of vbucket 441 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.040581 PST 3: Deletion of vbucket 441 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.054922 PST 3: Deletion of vbucket 197 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.056613 PST 3: Deletion of vbucket 197 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.060198 PST 3: Deletion of vbucket 163 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.067666 PST 3: Deletion of vbucket 163 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.069318 PST 3: Deletion of vbucket 128 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.070137 PST 3: Deletion of vbucket 128 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.071390 PST 3: Deletion of vbucket 94 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.071774 PST 3: Deletion of vbucket 94 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.072726 PST 3: Deletion of vbucket 62 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.073136 PST 3: Deletion of vbucket 62 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.074003 PST 3: Deletion of vbucket 61 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.074337 PST 3: Deletion of vbucket 61 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.074697 PST 3: Deletion of vbucket 26 failed because the vbucket is not in a dead state
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.075026 PST 3: Deletion of vbucket 26 was completed.
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.092594 PST 3: TAP (Consumer) eq_tapq:anon_21 - disconnected
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.106474 PST 3: TAP (Consumer) eq_tapq:anon_19 - disconnected
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.159445 PST 3: TAP (Consumer) eq_tapq:anon_15 - disconnected
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.184971 PST 3: TAP (Consumer) eq_tapq:anon_22 - disconnected

      [ns_server:debug,2012-11-26T4:34:42.294,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
      [couchdb:error,2012-11-26T4:34:42.309,ns_1@10.6.2.37:<0.29237.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [couchdb:error,2012-11-26T4:34:42.352,ns_1@10.6.2.37:<0.29238.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [couchdb:error,2012-11-26T4:34:42.370,ns_1@10.6.2.37:<0.29239.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:42.371,ns_1@10.6.2.37:tap_replication_manager-saslbucket<0.25114.5201>:tap_replication_manager:change_vbucket_filter:185]Going to change replication from 'ns_1@10.6.2.39' to have
      "<" ([], "=")
      [ns_server:debug,2012-11-26T4:34:42.378,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:117]Registered myself under id:{"saslbucket",
      {new_child_id,"<",'ns_1@10.6.2.39'},
      #Ref<0.0.47906.208825>}
      Args:[{"10.6.2.39",11209},
      {"10.6.2.37",11209}

      ,
      [

      {old_state_retriever,#Fun<ns_vbm_new_sup.0.113560106>},
      {on_not_ready_vbuckets,#Fun<tap_replication_manager.2.88495417>},
      {username,"saslbucket"},
      {password,"password"},
      {vbuckets,"<"},
      {takeover,false},
      {suffix,"ns_1@10.6.2.37"}]]
      [ns_server:debug,2012-11-26T4:34:42.379,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.29120.5201>
      [ns_server:info,2012-11-26T4:34:42.379,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
      [couchdb:error,2012-11-26T4:34:42.412,ns_1@10.6.2.37:<0.29244.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:42.435,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
      [{60,26}]
      [ns_server:info,2012-11-26T4:34:42.437,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
      [ns_server:info,2012-11-26T4:34:42.438,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
      [ns_server:info,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
      [ns_server:debug,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
      [ns_server:debug,2012-11-26T4:34:42.445,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
      [ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29378.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29378.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [rebalance:info,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:42.446,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
      [ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
      [ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29120.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
      [ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.29120.5201>
      [ns_server:debug,2012-11-26T4:34:42.447,ns_1@10.6.2.37:<0.29342.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
      [ns_server:info,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.29380.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
      [ns_server:debug,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.29120.5201>:
      {state,#Port<0.34658446>,#Port<0.34658438>,#Port<0.34658449>,
      #Port<0.34658440>,<0.29132.5201>,<<"cut off">>,<<"cut off">>,[],10,
      false,false,0,
      {1353,933282,438008},
      completed,
      {<0.29342.5201>,#Ref<0.0.47906.208842>},
      <<"replication_ns_1@10.6.2.37">>,<0.29120.5201>,
      {had_backfill,false,undefined,[]},
      false}.
      [ns_server:debug,2012-11-26T4:34:42.448,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29342.5201>,{#Ref<0.0.47906.208827>,<0.29380.5201>}}
      [ns_server:info,2012-11-26T4:34:42.449,ns_1@10.6.2.37:tap_replication_manager-saslbucket<0.25114.5201>:tap_replication_manager:change_vbucket_filter:185]Going to change replication from 'ns_1@10.6.2.40' to have
      "]" ([], "^")
      [error_logger:info,2012-11-26T4:34:42.450,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
      =========================PROGRESS REPORT=========================
      supervisor: {local,'ns_vbm_new_sup-saslbucket'}
      started: [{pid,<0.29380.5201>},
      {name,{new_child_id,"<",'ns_1@10.6.2.39'}},
      {mfargs,
      {ebucketmigrator_srv,start_link,
      [{"10.6.2.39",11209},
      {"10.6.2.37",11209},
      [{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
      {on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
      {username,"saslbucket"},
      {password,"password"},
      {vbuckets,"<"},
      {takeover,false},
      {suffix,"ns_1@10.6.2.37"}]]}},
      {restart_type,temporary},
      {shutdown,60000},
      {child_type,worker}]

      [ns_server:debug,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:117]Registered myself under id:{"saslbucket",
      {new_child_id,"]",'ns_1@10.6.2.40'},
      #Ref<0.0.47906.210506>}
      Args:[{"10.6.2.40",11209},
      {"10.6.2.37",11209},
      [{old_state_retriever,#Fun<ns_vbm_new_sup.0.113560106>}

      ,

      {on_not_ready_vbuckets,#Fun<tap_replication_manager.2.88495417>}

      ,

      {username,"saslbucket"},
      {password,"password"},
      {vbuckets,"]"},
      {takeover,false},
      {suffix,"ns_1@10.6.2.37"}]]
      [ns_server:debug,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:121]Linked myself to old ebucketmigrator <0.29269.5201>
      [ns_server:info,2012-11-26T4:34:42.457,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:273]Starting new-style vbucket filter change on stream `replication_ns_1@10.6.2.37`
      [couchdb:error,2012-11-26T4:34:42.482,ns_1@10.6.2.37:<0.29245.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:debug,2012-11-26T4:34:42.492,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:569]Reusing old upstream:
      [{vbuckets,"<"},{name,<<"replication_ns_1@10.6.2.37">>},{takeover,false}]
      [rebalance:debug,2012-11-26T4:34:42.493,ns_1@10.6.2.37:<0.29380.5201>:ebucketmigrator_srv:init:588]upstream_sender pid: <0.29385.5201>
      [couchdb:error,2012-11-26T4:34:42.513,ns_1@10.6.2.37:<0.29247.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:42.516,ns_1@10.6.2.37:ns_port_memcached<0.23890.5201>:ns_port_server:log:171]memcached<0.23890.5201>: Mon Nov 26 04:34:42.308712 PST 3: Notified the completion of checkpoint persistence for vbucket 128, cookie 0x6aaeb00
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.444246 PST 3: Schedule cleanup of "eq_tapq:anon_15"
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.444311 PST 3: Schedule cleanup of "eq_tapq:anon_19"
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.444330 PST 3: Schedule cleanup of "eq_tapq:anon_21"
      memcached<0.23890.5201>: Mon Nov 26 04:34:42.444365 PST 3: Schedule cleanup of "eq_tapq:anon_22"

      [couchdb:error,2012-11-26T4:34:42.521,ns_1@10.6.2.37:<0.29248.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,{not_found,no_db_file}}}

      Stacktrace: [{capi_utils,must_open_vbucket,2},
      {capi_frontend,with_subdb,3},
      {couch_httpd_db,handle_design_req,2},
      {couch_httpd,handle_request,6},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]
      [ns_server:info,2012-11-26T4:34:42.523,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:296]Changing vbucket filter on tap stream `replication_ns_1@10.6.2.37`:
      [{93,38}]
      [ns_server:info,2012-11-26T4:34:42.525,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_call:306]Successfully changed vbucket filter on tap stream `replication_ns_1@10.6.2.37`.
      [ns_server:info,2012-11-26T4:34:42.526,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:process_upstream:922]Got vbucket filter change completion message. Silencing upstream sender
      [ns_server:info,2012-11-26T4:34:42.526,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:handle_info:350]Got reply from upstream silencing request. Completing state transition to a new ebucketmigrator.
      [ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:complete_native_vb_filter_change:168]Proceeding with reading unread binaries
      [ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_downstream:195]Going to confirm reception downstream messages
      [ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_sent_messages:710]Going to wait for reception of opaque message ack
      [ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29405.5201>:ebucketmigrator_srv:confirm_sent_messages:705]Sending opaque message to confirm downstream reception
      [ns_server:debug,2012-11-26T4:34:42.527,ns_1@10.6.2.37:<0.29405.5201>:ebucketmigrator_srv:confirm_sent_messages:707]Sent fine
      [rebalance:info,2012-11-26T4:34:42.530,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:do_confirm_sent_messages:684]Got close ack!

      [ns_server:debug,2012-11-26T4:34:42.533,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:confirm_downstream:199]Confirmed upstream messages are feeded to kernel
      [ns_server:debug,2012-11-26T4:34:42.534,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:reply_and_die:208]Passed old state to caller
      [ns_server:debug,2012-11-26T4:34:42.534,ns_1@10.6.2.37:<0.29269.5201>:ebucketmigrator_srv:reply_and_die:211]Sent out state. Preparing to die
      [ns_server:debug,2012-11-26T4:34:42.536,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change:125]Got old state from previous ebucketmigrator: <0.29269.5201>
      [ns_server:debug,2012-11-26T4:34:42.537,ns_1@10.6.2.37:<0.29381.5201>:ns_vbm_new_sup:perform_vbucket_filter_change_loop:166]Sent old state to new instance
      [ns_server:info,2012-11-26T4:34:42.537,ns_1@10.6.2.37:<0.29423.5201>:ns_vbm_new_sup:mk_old_state_retriever:83]Got vbucket filter change old state. Proceeding vbucket filter change operation
      [ns_server:debug,2012-11-26T4:34:42.538,ns_1@10.6.2.37:<0.29423.5201>:ebucketmigrator_srv:init:452]Got old ebucketmigrator state from <0.29269.5201>:
      {state,#Port<0.34659745>,#Port<0.34659734>,#Port<0.34659747>,
      #Port<0.34659737>,<0.29271.5201>,<<"cut off">>,<<"cut off">>,[],10,
      false,false,0,
      {1353,933282,526685},
      completed,
      {<0.29381.5201>,#Ref<0.0.47906.210588>},
      <<"replication_ns_1@10.6.2.37">>,<0.29269.5201>,
      {had_backfill,false,undefined,[]},
      false}.
      [ns_server:debug,2012-11-26T4:34:42.538,ns_1@10.6.2.37:<0.772.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.29381.5201>,{#Ref<0.0.47906.210536>,<0.29423.5201>}}
      [error_logger:info,2012-11-26T4:34:42.538,ns_1@10.6.2.37:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
      =========================PROGRESS REPORT=========================
      supervisor: {local,'ns_vbm_new_sup-saslbucket'}
      started: [{pid,<0.29423.5201>},
      {name,{new_child_id,"]",'ns_1@10.6.2.40'}},
      {mfargs,
      {ebucketmigrator_srv,start_link,
      [{"10.6.2.40",11209},
      {"10.6.2.37",11209},
      [{old_state_retriever, #Fun<ns_vbm_new_sup.0.113560106>},
      {on_not_ready_vbuckets, #Fun<tap_replication_manager.2.88495417>},
      {username,"saslbucket"}

      ,

      {password,"password"}

      ,

      {vbuckets,"]"}

      ,

      {takeover,false}

      ,

      {suffix,"ns_1@10.6.2.37"}

      ]]}},

      {restart_type,temporary}

      ,

      {shutdown,60000}

      ,

      {child_type,worker}

      ]

      [user:info,2012-11-26T4:34:42.540,ns_1@10.6.2.37:ns_config<0.762.0>:ns_janitor:maybe_stop_rebalance_status:147]Resetting rebalance status since it's not really running
      [ns_server:debug,2012-11-26T4:34:42.543,ns_1@10.6.2.37:capi_set_view_manager-saslbucket<0.25094.5201>:capi_set_view_manager:handle_info:349]doing replicate_newnodes_docs
      [couchdb:error,2012-11-26T4:34:42.556,ns_1@10.6.2.37:<0.29267.5201>:couch_log:error:42]Uncaught error in HTTP request: {exit,{open_db_failed,

      {not_found,no_db_file}

      }}

      Stacktrace: [

      {capi_utils,must_open_vbucket,2}

      ,

      {capi_frontend,with_subdb,3}

      ,

      {couch_httpd_db,handle_design_req,2}

      ,

      {couch_httpd,handle_request,6}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Show
          andreibaranouski Andrei Baranouski added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.37-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.38-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.39-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.40-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.42-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7261/10.6.2.45-8091-diag.txt.gz
          Hide
          andreibaranouski Andrei Baranouski added a comment -

          passed after restart

          Show
          andreibaranouski Andrei Baranouski added a comment - passed after restart
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -
          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - MB-7182

            People

            • Assignee:
              alkondratenko Aleksey Kondratenko (Inactive)
              Reporter:
              andreibaranouski Andrei Baranouski
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes