memcached crashed during rebalance-in after offline upgrade from 2.2.0 - 3.0.0-747-rel

Jenkins - test# 24
http://qa.hq.northscale.net/job/ubuntu_x64--36_01--XDCR_upgrade-P1/9/consoleFull

[Test Case]

./testrunner i ubuntu_x6436_01-XDCR_upgrade-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,initial_vbuckets=128,upgrade_version=3.0.0-747-rel -t xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.2.0-837-rel,sdata=False,bucket_topology=default:1>2;bucket0:1><2,upgrade_nodes=src,post-upgrade-actions=src-rebalancein;dest-rebalanceout;dest-create_index

[Test Error]

[2014-05-28 09:56:53,497] - [task:283] INFO - adding node 10.3.3.199:8091 to cluster
[2014-05-28 09:56:53,497] - [rest_client:930] INFO - adding remote node @10.3.3.199:8091 to this cluster @10.3.3.240:8091
[2014-05-28 09:57:06,311] - [rest_client:1084] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.3.199%2Cns_1%4010.3.3.240%2Cns_1%4010.3.3.218
[2014-05-28 09:57:06,334] - [rest_client:1088] INFO - rebalance operation started
[2014-05-28 09:57:06,340] - [rest_client:1197] INFO - rebalance percentage : 0 %
[2014-05-28 09:57:16,362] - [rest_client:1197] INFO - rebalance percentage : 8.39622465713 %
[2014-05-28 09:57:26,400] - [rest_client:1197] INFO - rebalance percentage : 29.9672033393 %
[2014-05-28 09:57:36,533] - [rest_client:1197] INFO - rebalance percentage : 52.9696817233 %
[2014-05-28 09:57:46,585] - [rest_client:1197] INFO - rebalance percentage : 87.3377869708 %
[2014-05-28 09:57:56,603] - [rest_client:1181] ERROR -

{u'status': u'none', u'errorMessage': u'UPR upgrade failed. See logs for detailed reason.'}

- rebalance failed
[2014-05-28 09:57:56,627] - [rest_client:1951] INFO - Latest logs from UI on 10.3.3.240:
[2014-05-28 09:57:56,628] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@10.3.3.218\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:52.938Z', u'module': u'ns_memcached', u'tstamp': 1401296332938, u'type': u'info'}

[2014-05-28 09:57:56,629] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.218\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:52.340Z', u'module': u'ns_memcached', u'tstamp': 1401296332340, u'type': u'info'}

[2014-05-28 09:57:56,630] - [rest_client:1952] ERROR - {u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@10.3.3.218' disconnected: {badmatch,\n {error,\n einval}}", u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:51.724Z', u'module': u'ns_memcached', u'tstamp': 1401296331724, u'type': u'info'}
[2014-05-28 09:57:56,630] - [rest_client:1952] ERROR - {u'node': u'ns_1@10.3.3.218', u'code': 2, u'text': u'UPR upgrade exited with reason {{badmatch,\n {error,

{failed_nodes,[\'ns_1@10.3.3.218\']}

}},\n [{upr_upgrade,handle_call,3,\n [

{file,"src/upr_upgrade.erl"}

,

{line,65}

]},\n {gen_server,handle_msg,5,\n [

{file,"gen_server.erl"}

,

{line,585}

]},\n {proc_lib,init_p_do_apply,3,\n [

{file,"proc_lib.erl"}

,

{line,239}

]}]}\n', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:51.366Z', u'module': u'ns_orchestrator', u'tstamp': 1401296331366, u'type': u'info'}
[2014-05-28 09:57:56,631] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u"Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139. Restarting. Messages: Wed May 28 09:58:50.855754 PDT 3: (bucket0) TAP (Producer) eq_tapq:rebalance_21 - Clear the tap queues by force\nWed May 28 09:58:50.886021 PDT 3: (bucket0) Deletion of vbucket 21 was completed.\nWed May 28 09:58:50.887642 PDT 3: (bucket0) TAP (Producer) eq_tapq:replication_building_21_'ns_1@10.3.3.199' - disconnected, keep alive for 300 seconds\nWed May 28 09:58:50.891012 PDT 3: (bucket0) TAP (Producer) eq_tapq:replication_building_21_'ns_1@10.3.3.199' - Connection is closed by force\nWed May 28 09:58:51.123816 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@10.3.3.199->ns_1@10.3.3.218:default - (vb 42) Attempting to add stream with start seqno 14, end seqno 18446744073709551615, vbucket uuid 249397048172087, snap start seqno 14, and snap end seqno 14", u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:51.314Z', u'module': u'ns_log', u'tstamp': 1401296331314, u'type': u'info'}

[2014-05-28 09:57:56,632] - [rest_client:1952] ERROR - {u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@10.3.3.218' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:51.309Z', u'module': u'ns_memcached', u'tstamp': 1401296331309, u'type': u'info'}
[2014-05-28 09:57:56,633] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 1, u'text': u'Rebalance completed successfully.\n', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:50.995Z', u'module': u'ns_orchestrator', u'tstamp': 1401296330995, u'type': u'info'}

[2014-05-28 09:57:56,633] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u'Bucket "bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:35.808Z', u'module': u'ns_vbucket_mover', u'tstamp': 1401296315808, u'type': u'info'}

[2014-05-28 09:57:56,634] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@10.3.3.199\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:35.383Z', u'module': u'ns_memcached', u'tstamp': 1401296315383, u'type': u'info'}

[2014-05-28 09:57:56,635] - [rest_client:1952] ERROR -

{u'node': u'ns_1@10.3.3.218', u'code': 0, u'text': u'Started rebalancing bucket bucket0', u'shortText': u'message', u'serverTime': u'2014-05-28T09:58:34.743Z', u'module': u'ns_rebalancer', u'tstamp': 1401296314743, u'type': u'info'}

ERROR