Rebalance fails due to timeout error. Rebalancing in 3 nodes on a 1 node cluster ( with 6 buckets).

1. Create 6 buckets on a single node cluster
2. Enable auto-failover
3. Load around 1.5M items on 4 buckets; 2 empty bucket
4. Add 3 nodes to this node, and Rebalance

Error
---------
Rebalance successful on 4 buckets, fails on the buckets - b1 and b2.

• rebalance took over 4 hours to rebalance ( 6M items on the nodes, with 6 buckets- 4 with data, 2 empty)

1. Rebalance exited with reason {timeout,
{gen_server,call,
[

{'ns_memcached-b3','ns_1@10.3.121.32'}, {get_vbucket,690},
60000]}}
2. Node is auto-failed over.
3.Lost data in "b3" for
.Data has been lost for 100% of vbuckets in bucket "b1".
Data has been lost for 100% of vbuckets in bucket "b2".

Logs at https://s3.amazonaws.com/bugdb/jira/rebl-2.0/rebal.tar


2012-08-16 18:53:39.585 ns_memcached:1:info:message(ns_1@10.3.121.34) - Bucket "b5" loaded on node 'ns_1@10.3.121.34' in 0 seconds.
2012-08-16 18:53:39.613 ns_memcached:1:info:message(ns_1@10.3.121.37) - Bucket "b5" loaded on node 'ns_1@10.3.121.37' in 0 seconds.
2012-08-16 18:53:40.576 ns_memcached:1:info:message(ns_1@10.3.121.33) - Bucket "b5" loaded on node 'ns_1@10.3.121.33' in 0 seconds.
2012-08-16 18:54:04.048 ns_rebalancer:0:info:message(ns_1@10.3.121.32) - Started rebalancing bucket b5
2012-08-16 18:54:05.411 ns_vbucket_mover:0:info:message(ns_1@10.3.121.32) - Bucket "b5" rebalance does not seem to be swap rebalance
2012-08-16 19:14:01.573 ns_memcached:1:info:message(ns_1@10.3.121.34) - Bucket "b4" loaded on node 'ns_1@10.3.121.34' in 0 seconds.
2012-08-16 19:14:01.960 ns_memcached:1:info:message(ns_1@10.3.121.37) - Bucket "b4" loaded on node 'ns_1@10.3.121.37' in 0 seconds.
2012-08-16 19:14:02.625 ns_memcached:1:info:message(ns_1@10.3.121.33) - Bucket "b4" loaded on node 'ns_1@10.3.121.33' in 0 seconds.
2012-08-16 19:14:25.918 ns_rebalancer:0:info:message(ns_1@10.3.121.32) - Started rebalancing bucket b4
2012-08-16 19:14:29.813 ns_vbucket_mover:0:info:message(ns_1@10.3.121.32) - Bucket "b4" rebalance does not seem to be swap rebalance
2012-08-16 20:37:10.746 ns_memcached:1:info:message(ns_1@10.3.121.34) - Bucket "b3" loaded on node 'ns_1@10.3.121.34' in 0 seconds.
2012-08-16 20:37:10.821 ns_memcached:1:info:message(ns_1@10.3.121.37) - Bucket "b3" loaded on node 'ns_1@10.3.121.37' in 0 seconds.
2012-08-16 20:37:11.753 ns_memcached:1:info:message(ns_1@10.3.121.33) - Bucket "b3" loaded on node 'ns_1@10.3.121.33' in 0 seconds.
2012-08-16 20:37:34.981 ns_rebalancer:0:info:message(ns_1@10.3.121.32) - Started rebalancing bucket b3
2012-08-16 20:37:38.097 ns_vbucket_mover:0:info:message(ns_1@10.3.121.32) - Bucket "b3" rebalance does not seem to be swap rebalance
2012-08-16 21:45:54.546 auto_failover:0:info:message(ns_1@10.3.121.32) - Could not automatically failover node 'ns_1@10.3.121.32' because I think rebalance is running
2012-08-16 21:47:00.321 ns_memcached:1:info:message(ns_1@10.3.121.37) - Bucket "b1" loaded on node 'ns_1@10.3.121.37' in 0 seconds.
2012-08-16 21:47:00.643 ns_memcached:1:info:message(ns_1@10.3.121.34) - Bucket "b1" loaded on node 'ns_1@10.3.121.34' in 0 seconds.
2012-08-16 21:47:01.551 ns_memcached:1:info:message(ns_1@10.3.121.33) - Bucket "b1" loaded on node 'ns_1@10.3.121.33' in 0 seconds.
2012-08-16 21:47:12.697 ns_memcached:1:info:message(ns_1@10.3.121.37) - Bucket "b2" loaded on node 'ns_1@10.3.121.37' in 0 seconds.
2012-08-16 21:47:13.213 ns_memcached:1:info:message(ns_1@10.3.121.34) - Bucket "b2" loaded on node 'ns_1@10.3.121.34' in 0 seconds.
2012-08-16 21:47:13.642 ns_memcached:1:info:message(ns_1@10.3.121.33) - Bucket "b2" loaded on node 'ns_1@10.3.121.33' in 0 seconds.
2012-08-16 21:47:15.164 ns_vbucket_mover:0:critical:message(ns_1@10.3.121.32) - <0.24045.64> exited with {timeout,
{gen_server,call,
[{'ns_memcached-b3','ns_1@10.3.121.32'}

,

{get_vbucket,690},
60000]}}
2012-08-16 21:47:15.444 ns_orchestrator:2:info:message(ns_1@10.3.121.32) - Rebalance exited with reason {timeout,
{gen_server,call,
[{'ns_memcached-b3','ns_1@10.3.121.32'},
{get_vbucket,690}

,
60000]}}

2012-08-16 21:47:15.702 ns_orchestrator:0:info:message(ns_1@10.3.121.32) - Starting failing over 'ns_1@10.3.121.32'
2012-08-16 21:47:15.715 ns_rebalancer:0:critical:message(ns_1@10.3.121.32) - Lost data in "b3" for [690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]
2012-08-16 21:47:15.937 ns_rebalancer:1:info:message(ns_1@10.3.121.32) - Data has been lost for 32% of vbuckets in bucket "b3".
2012-08-16 21:47:17.646 ns_memcached:2:info:message(ns_1@10.3.121.32) - Shutting down bucket "b3" on 'ns_1@10.3.121.32' for deletion
2012-08-16 21:47:20.626 ns_rebalancer:0:critical:message(ns_1@10.3.121.32) - Lost data in "b1" for [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]
2012-08-16 21:47:20.629 ns_rebalancer:1:info:message(ns_1@10.3.121.32) - Data has been lost for 100% of vbuckets in bucket "b1".
2012-08-16 21:47:36.279 ns_rebalancer:0:critical:message(ns_1@10.3.121.32) - Lost data in "b2" for [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]
2012-08-16 21:47:36.280 ns_rebalancer:1:info:message(ns_1@10.3.121.32) - Data has been lost for 100% of vbuckets in bucket "b2".
2012-08-16 21:47:50.662 ns_orchestrator:6:info:message(ns_1@10.3.121.32) - Failed over 'ns_1@10.3.121.32': ok
2012-08-16 21:47:50.700 auto_failover:1:info:message(ns_1@10.3.121.32) - Node ('ns_1@10.3.121.32') was automatically failovered.
[{last_heard,{1345,178869,557311}},
{outgoing_replications_safeness_level,[

{"b2",green}

,

{"b1",green}

]},
{incoming_replications_conf_hashes,[

{"b2",[]}

,

{"b1",[]}

]},

{active_buckets,["b4","b3","b6","b2","b1","b5"]}

,

{ready_buckets,["b4","b6","b2","b1","b5"]}

,
{local_tasks,[[

{pid,<<"<0.5402.57>">>}

,

{bucket,<<"b5">>}

,

{original_target,bucket}

,

{progress,30}

,

{started_on,1345173907}

,

{total_vbuckets,513}

,

{trigger_type,scheduled}

,

{type,bucket_compaction}

,

{updated_on,1345178862}

,

{vbuckets_done,159}

]]},
{memory,[

{total,1244930200}

,

{processes,1078159104}

,

{processes_used,1073223288}

,

{system,166771096}

,

{atom,1469433}

,

{atom_used,1448062}

,

{binary,72656368}

,

{code,14396890}

,

{ets,64356624}

]},
{system_memory_data,[

{system_total_memory,8393789440}

,

{free_swap,6506700800}

,

{total_swap,6585049088}

,

{cached_memory,1809096704}

,

{buffered_memory,34918400}

,

{free_memory,128155648}

,

{total_memory,8393789440}

]},
{node_storage_conf,[

{db_path,"/opt/couchbase/var/lib/couchbase/data/"}

,

{index_path,"/opt/couchbase/var/lib/couchbase/data/"}

]},
{statistics,[{wall_clock,{18103426,5031}},
{context_switches,{487778177,0}},
{garbage_collection,{40684904,342672558536,0}},
{io,{{input,18282049685},

{output,19258350837}

}},
{reductions,{129576423507,31338119}},

{run_queue,1}

,
{runtime,{19386420,6100}}]},
{system_stats,[

{cpu_utilization_rate,62.254901960784316}

,

{swap_total,6585049088}

,

{swap_used,78356480}

]},
{interesting_stats,[

{curr_items,0}

,

{curr_items_tot,0}

,

{vb_replica_curr_items,0}

]},

{cluster_compatibility_version,131072}

,
{version,[

{public_key,"0.13"}

,

{lhttpc,"1.3.0"}

,

{ale,"8cffe61"}

,

{os_mon,"2.2.7"}

,

{couch_set_view,"1.2.0a-1a64800-git"}

,

{mnesia,"4.5"}

,

{inets,"5.7.1"}

,

{couch,"1.2.0a-1a64800-git"}

,

{mapreduce,"1.0.0"}

,

{couch_index_merger,"1.2.0a-1a64800-git"}

,

{kernel,"2.14.5"}

,

{crypto,"2.0.4"}

,

{ssl,"4.1.6"}

,

{sasl,"2.1.10"}

,

{couch_view_parser,"1.0.0"}

,

{ns_server,"2.0.0-1594-rel-enterprise"}

,

{mochiweb,"1.4.1"}

,

{oauth,"7d85d3ef"}

,

{stdlib,"1.17.5"}

]},

{supported_compat_version,[2,0]}

,

{system_arch,"x86_64-unknown-linux-gnu"}

,

{wall_clock,18103}

,
{memory_data,{8393789440,8282431488,

{<0.434.0>,37864600}

}},
{disk_data,[

{"/",19223252,36}

,

{"/dev",4093592,1}

,

{"/dev/shm",4098528,0}

,

{"/var/run",4098528,1}

,

{"/var/lock",4098528,0}

,

{"/lib/init/rw",4098528,0}

,

{"/var/lib/ureadahead/debugfs",19223252,36}

,

{"/boot",240972,11}

]},

{meminfo,<<"MemTotal: 8197060 kB\nMemFree: 134608 kB\nBuffers: 33948 kB\nCached: 1771456 kB\nSwapCached: 13368 kB\nActive: 5827960 kB\nInactive: 1897712 kB\nActive(anon): 4946172 kB\nInactive(anon): 974072 kB\nActive(file): 881788 kB\nInactive(file): 923640 kB\nUnevictable: 0 kB\nMlocked: 0 kB\nSwapTotal: 6430712 kB\nSwapFree: 6354192 kB\nDirty: 8808 kB\nWriteback: 0 kB\nAnonPages: 5908568 kB\nMapped: 7220 kB\nShmem: 16 kB\nSlab: 145184 kB\nSReclaimable: 131124 kB\nSUnreclaim: 14060 kB\nKernelStack: 1512 kB\nPageTables: 14412 kB\nNFS_Unstable: 0 kB\nBounce: 0 kB\nWritebackTmp: 0 kB\nCommitLimit: 10529240 kB\nCommitted_AS: 6743872 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed: 291676 kB\nVmallocChunk: 34359442900 kB\nHardwareCorrupted: 0 kB\nHugePages_Total: 0\nHugePages_Free: 0\nHugePages_Rsvd: 0\nHugePages_Surp: 0\nHugepagesize: 2048 kB\nDirectMap4k: 10240 kB\nDirectMap2M: 8378368 kB\n">>}

]
2012-08-16 21:47:52.418 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {{badmatch,
{error,
closed}},
[

{mc_client_binary, cmd_binary_vocal_recv, 5},
{mc_client_binary, select_bucket, 2},
{ns_memcached, ensure_bucket, 2},
{ns_memcached, handle_info, 2},
{gen_server, handle_msg, 5},
{proc_lib, init_p_do_apply, 3}]}
2012-08-16 21:47:52.424 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {badmatch,
{error,
closed}}
2012-08-16 21:48:09.059 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {badmatch,
{error,
closed}} (repeated 2 times)
2012-08-16 21:48:09.059 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {{badmatch,
{error,
closed}},
[{mc_client_binary, cmd_binary_vocal_recv, 5}

,

{mc_client_binary, select_bucket, 2}

,

{ns_memcached, ensure_bucket, 2}

,

{ns_memcached, handle_info, 2},
{gen_server, handle_msg, 5},
{proc_lib, init_p_do_apply, 3}]} (repeated 1 times)
2012-08-16 21:48:41.607 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary, stats_recv, 4},
{mc_client_binary, stats, 4},
{ns_memcached, has_started, 1},
{ns_memcached, handle_info, 2}

,

{gen_server, handle_msg, 5},
{proc_lib, init_p_do_apply, 3}]}
2012-08-16 21:52:47.984 ns_memcached:1:info:message(ns_1@10.3.121.32) - Bucket "b6" loaded on node 'ns_1@10.3.121.32' in 0 seconds.
2012-08-16 21:53:17.972 ns_memcached:4:info:message(ns_1@10.3.121.32) - Control connection to memcached on 'ns_1@10.3.121.32' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary, stats_recv, 4},
{mc_client_binary, stats, 4},
{ns_memcached, has_started, 1},
{ns_memcached, handle_info, 2},
{gen_server, handle_msg, 5}

,

{proc_lib, init_p_do_apply, 3}

]}
2012-08-16 21:53:24.599 ns_memcached:2:info:message(ns_1@10.3.121.32) - Shutting down bucket "b5" on 'ns_1@10.3.121.32' for deletion
2012-08-16 22:22:21.971 ns_memcached:2:info:message(ns_1@10.3.121.32) - Shutting down bucket "b2" on 'ns_1@10.3.121.32' for server shutdown
2012-08-16 22:39:49.318 ns_memcached:2:info:message(ns_1@10.3.121.32) - Shutting down bucket "b6" on 'ns_1@10.3.121.32' for deletion
2012-08-16 22:41:27.614 ns_memcached:1:info:message(ns_1@10.3.121.32) - Bucket "b1" loaded on node 'ns_1@10.3.121.32' in 2883 seconds.
2012-08-16 22:42:33.388 ns_memcached:1:info:message(ns_1@10.3.121.32) - Bucket "b2" loaded on node 'ns_1@10.3.121.32' in 159 seconds.