GoXDCR: No keys replicated (encrypted xdcr) after offline upgrade to Sherlock [err: dial tcp 10.1.2.19:11215: connection refused]

Build

4.0.0-1689

Testcase
--------
./testrunner -i INI_FILE.ini -p upgrade_version=4.0.0-1869,initial_vbuckets=1024 -t "xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.5.0-1059-rel,bucket_topology=default:1>2;sasl_bucket_1:1><2;standard_bucket_1:1<2,upgrade_nodes=dest;src,demand_encryption=1"

test#10 in http://qa.hq.northscale.net/view/3.0%20By%20COMPONENT/job/cen006-p1-xxdcr-vset05-02-goxdcr-backward-compatibility-and-upgrade/

Steps

1. Install 2.5.0 on C1[.11,.16] and C2 [.19,.20]
2. C1.default --> C2.default
C1.sasl <---> C2.sasl
C1.standard <--- C2.standard
3. Load 1000 keys into C1.default, C1.sasl, C2.standard. Expect 1000 keys in all buckets.
4. Offline upgrade C1 and C2 to 4.0.0-1869
5. Load 1000 keys each into C1.sasl, C2.sasl and C1.default.
6. Expect 3000 keys in sasl buckets, 2000 in default, 1000 in standard buckets.

C2.sasl and C2.default are missing 1000 keys loaded in step 5.

[2015-04-16 20:34:44,015] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 2000 == 2000 expected on C1, default bucket
[2015-04-16 20:34:44,113] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 3000 == 3000 expected on C1, sasl_bucket_1 bucket
[2015-04-16 20:34:44,195] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 1000 == 1000 expected on C1, standard_bucket_1 bucket
[2015-04-16 20:34:45,800] - [xdcrnewbasetests:1996] WARNING - Not Ready: vb_active_curr_items 1000 == 2000 expected on C2, default bucket

[2015-04-16 20:37:49,894] - [xdcrnewbasetests:1996] WARNING - Not Ready: vb_active_curr_items 2000 == 3000 expected on C2, sasl_bucket_1 bucket

Default bucket:

[2015-04-16 20:41:55,119] - [xdcrnewbasetests:2763] ERROR - 1000 keys not found on 10.1.2.19, printing first 100 keys: [('key: loadTwo216', 'vbucket: 403'), ('key: loadTwo510', 'vbucket: 447'), ('key: loadTwo351', 'vbucket: 601'), ('key: loadTwo753', 'vbucket: 862'), ('key: loadTwo457', 'vbucket: 629'), ('key: loadTwo268', 'vbucket: 362'), ('key: loadTwo906', 'vbucket: 221'), ('key: loadTwo614', 'vbucket: 148'), ('key: loadTwo318', 'vbucket: 1001'), ('key: loadTwo820', 'vbucket: 586'), ('key: loadTwo75', 'vbucket: 373'), ('key: loadTwo538', 'vbucket: 338'), ('key: loadTwo623', 'vbucket: 477'), ('key: loadTwo883', 'vbucket: 428'), ('key: loadTwo931', 'vbucket: 404'), ('key: loadTwo584', 'vbucket: 784'), ('key: loadTwo429', 'vbucket: 652'), ('key: loadTwo186', 'vbucket: 535'), ('key: loadTwo42', 'vbucket: 60'), ('key: loadTwo791', 'vbucket: 485'), ('key: loadTwo876', 'vbucket: 606'), ('key: loadTwo125', 'vbucket: 497'), ('key: loadTwo330', 'vbucket: 772'), ('key: loadTwo221', 'vbucket: 218'), ('key: loadTwo436', 'vbucket: 808'), ('key: loadTwo732', 'vbucket: 515'), ('key: loadTwo808', 'vbucket: 679'), ('key: loadTwo527', 'vbucket: 246'), ('key: loadTwo240', 'vbucket: 391'), ('key: loadTwo401', 'vbucket: 609'), ('key: loadTwo642', 'vbucket: 128'), ('key: loadTwo817', 'vbucket: 771'), ('key: loadTwo950', 'vbucket: 201'), ('key: loadTwo23', 'vbucket: 353'), ('key: loadTwo495', 'vbucket: 206'), ('key: loadTwo559', 'vbucket: 15'), ('key: loadTwo992', 'vbucket: 626'), ('key: loadTwo393', 'vbucket: 226'), ('key: loadTwo144', 'vbucket: 172'), ('key: loadTwo869', 'vbucket: 1018'), ('key: loadTwo307', 'vbucket: 589'), ('key: loadTwo199', 'vbucket: 947'), ('key: loadTwo80', 'vbucket: 647'), ('key: loadTwo282', 'vbucket: 828'), ('key: loadTwo173', 'vbucket: 485'), ('key: loadTwo705', 'vbucket: 842'), ('key: loadTwo460', 'vbucket: 828'), ('key: loadTwo978', 'vbucket: 36'), ('key: loadTwo571', 'vbucket: 226'), ('key: loadTwo277', 'vbucket: 206'), ('key: loadTwo675', 'vbucket: 457'), ('key: loadTwo379', 'vbucket: 692'), ('key: loadTwo841', 'vbucket: 791'), ('key: loadTwo14', 'vbucket: 40'), ('key: loadTwo967', 'vbucket: 384'), ('key: loadTwo546', 'vbucket: 427'), ('key: loadTwo209', 'vbucket: 55'), ('key: loadTwo680', 'vbucket: 571'), ('key: loadTwo112', 'vbucket: 184'), ('key: loadTwo366', 'vbucket: 784'), ('key: loadTwo764', 'vbucket: 535'), ('key: loadTwo448', 'vbucket: 977'), ('key: loadTwo919', 'vbucket: 377'), ('key: loadTwo219', 'vbucket: 300'), ('key: loadTwo690', 'vbucket: 800'), ('key: loadTwo102', 'vbucket: 419'), ('key: loadTwo909', 'vbucket: 98'), ('key: loadTwo292', 'vbucket: 551'), ('key: loadTwo458', 'vbucket: 714'), ('key: loadTwo594', 'vbucket: 523'), ('key: loadTwo196', 'vbucket: 780'), ('key: loadTwo341', 'vbucket: 834'), ('key: loadTwo135', 'vbucket: 234'), ('key: loadTwo500', 'vbucket: 164'), ('key: loadTwo743', 'vbucket: 581'), ('key: loadTwo426', 'vbucket: 563'), ('key: loadTwo830', 'vbucket: 849'), ('key: loadTwo893', 'vbucket: 183'), ('key: loadTwo231', 'vbucket: 449'), ('key: loadTwo633', 'vbucket: 198'), ('key: loadTwo921', 'vbucket: 143'), ('key: loadTwo439', 'vbucket: 919'), ('key: loadTwo52', 'vbucket: 295'), ('key: loadTwo807', 'vbucket: 536'), ('key: loadTwo189', 'vbucket: 680'), ('key: loadTwo528', 'vbucket: 73'), ('key: loadTwo485', 'vbucket: 469'), ('key: loadTwo320', 'vbucket: 543'), ('key: loadTwo154', 'vbucket: 439'), ('key: loadTwo818', 'vbucket: 956'), ('key: loadTwo722', 'vbucket: 792'), ('key: loadTwo866', 'vbucket: 837'), ('key: loadTwo317', 'vbucket: 854'), ('key: loadTwo556', 'vbucket: 176'), ('key: loadTwo250', 'vbucket: 156'), ('key: loadTwo411', 'vbucket: 890'), ('key: loadTwo652', 'vbucket: 411'), ('key: loadTwo715', 'vbucket: 593'), ('key: loadTwo940', 'vbucket: 466'), ('key: loadTwo33', 'vbucket: 122')]

Sasl bucket

[2015-04-16 20:42:00,299] - [xdcrnewbasetests:2763] ERROR - 1000 keys not found on 10.1.2.19, printing first 100 keys: [('key: loadThree508', 'vbucket: 282'), ('key: loadThree810', 'vbucket: 514'), ('key: loadThree698', 'vbucket: 670'), ('key: loadThree754', 'vbucket: 607'), ('key: loadThree957', 'vbucket: 456'), ('key: loadThree349', 'vbucket: 764'), ('key: loadThree613', 'vbucket: 405'), ('key: loadThree576', 'vbucket: 483'), ('key: loadThree672', 'vbucket: 200'), ('key: loadThree71', 'vbucket: 270'), ('key: loadThree827', 'vbucket: 843'), ('key: loadThree948', 'vbucket: 108'), ('key: loadThree583', 'vbucket: 529'), ('key: loadThree270', 'vbucket: 463'), ('key: loadThree181', 'vbucket: 790'), ('key: loadThree467', 'vbucket: 573'), ('key: loadThree406', 'vbucket: 864'), ('key: loadThree838', 'vbucket: 751'), ('key: loadThree936', 'vbucket: 149'), ('key: loadThree174', 'vbucket: 228'), ('key: loadThree478', 'vbucket: 921'), ('key: loadThree361', 'vbucket: 529'), ('key: loadThree796', 'vbucket: 228'), ('key: loadThree419', 'vbucket: 708'), ('key: loadThree10', 'vbucket: 83'), ('key: loadThree258', 'vbucket: 290'), ('key: loadThree763', 'vbucket: 790'), ('key: loadThree929', 'vbucket: 305'), ('key: loadThree84', 'vbucket: 764'), ('key: loadThree995', 'vbucket: 883'), ('key: loadThree624', 'vbucket: 220'), ('key: loadThree789', 'vbucket: 320'), ('key: loadThree115', 'vbucket: 441'), ('key: loadThree27', 'vbucket: 282'), ('key: loadThree859', 'vbucket: 946'), ('key: loadThree394', 'vbucket: 483'), ('key: loadThree300', 'vbucket: 844'), ('key: loadThree492', 'vbucket: 463'), ('key: loadThree226', 'vbucket: 475'), ('key: loadThree569', 'vbucket: 71'), ('key: loadThree431', 'vbucket: 553'), ('key: loadThree38', 'vbucket: 190'), ('key: loadThree901', 'vbucket: 476'), ('key: loadThree645', 'vbucket: 385'), ('key: loadThree337', 'vbucket: 517'), ('key: loadThree143', 'vbucket: 429'), ('key: loadThree702', 'vbucket: 587'), ('key: loadThree239', 'vbucket: 127'), ('key: loadThree541', 'vbucket: 170'), ('key: loadThree846', 'vbucket: 534'), ('key: loadThree520', 'vbucket: 503'), ('key: loadThree46', 'vbucket: 71'), ('key: loadThree328', 'vbucket: 929'), ('key: loadThree871', 'vbucket: 863'), ('key: loadThree285', 'vbucket: 573'), ('key: loadThree735', 'vbucket: 770'), ('key: loadThree450', 'vbucket: 884'), ('key: loadThree884', 'vbucket: 173'), ('key: loadThree517', 'vbucket: 190'), ('key: loadThree59', 'vbucket: 483'), ('key: loadThree356', 'vbucket: 856'), ('key: loadThree687', 'vbucket: 826'), ('key: loadThree122', 'vbucket: 240'), ('key: loadThree960', 'vbucket: 129'), ('key: loadThree211', 'vbucket: 146'), ('key: loadThree247', 'vbucket: 134'), ('key: loadThree507', 'vbucket: 421'), ('key: loadThree662', 'vbucket: 467'), ('key: loadThree894', 'vbucket: 438'), ('key: loadThree61', 'vbucket: 21'), ('key: loadThree958', 'vbucket: 375'), ('key: loadThree697', 'vbucket: 545'), ('key: loadThree260', 'vbucket: 212'), ('key: loadThree593', 'vbucket: 778'), ('key: loadThree786', 'vbucket: 511'), ('key: loadThree191', 'vbucket: 525'), ('key: loadThree579', 'vbucket: 348'), ('key: loadThree828', 'vbucket: 1012'), ('key: loadThree947', 'vbucket: 211'), ('key: loadThree105', 'vbucket: 162'), ('key: loadThree744', 'vbucket: 836'), ('key: loadThree468', 'vbucket: 642'), ('key: loadThree371', 'vbucket: 778'), ('key: loadThree409', 'vbucket: 991'), ('key: loadThree837', 'vbucket: 592'), ('key: loadThree773', 'vbucket: 525'), ('key: loadThree248', 'vbucket: 57'), ('key: loadThree939', 'vbucket: 42'), ('key: loadThree688', 'vbucket: 901'), ('key: loadThree384', 'vbucket: 248'), ('key: loadThree477', 'vbucket: 806'), ('key: loadThree655', 'vbucket: 154'), ('key: loadThree799', 'vbucket: 91'), ('key: loadThree416', 'vbucket: 635'), ('key: loadThree926', 'vbucket: 398'), ('key: loadThree257', 'vbucket: 413'), ('key: loadThree310', 'vbucket: 599'), ('key: loadThree164', 'vbucket: 511'), ('key: loadThree482', 'vbucket: 212'), ('key: loadThree28', 'vbucket: 421')]

Attached logs from C1 and C2.

In goxdcr log from C1, I see -

sasl_bucket_1_10.1.2.19:11210_26:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_12:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_10:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_19:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_7:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_21:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_3:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_25:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_0:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_31:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_2:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_1:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_20:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_18:dial tcp 10.1.2.19:11215: con
nection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_5:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_9:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_30:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_22:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_17:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_16:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_28:dial tcp 10.1.2.19:11215: connection refused]\n"}]}, progress=

]