Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
4.0.0
-
Security Level: Public
-
None
-
centOS 6.x
-
Untriaged
-
-
Unknown
Description
Build
4.0.0-1689
Testcase
--------
./testrunner -i INI_FILE.ini -p upgrade_version=4.0.0-1869,initial_vbuckets=1024 -t "xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.5.0-1059-rel,bucket_topology=default:1>2;sasl_bucket_1:1><2;standard_bucket_1:1<2,upgrade_nodes=dest;src,demand_encryption=1"
Steps
1. Install 2.5.0 on C1[.11,.16] and C2 [.19,.20]
2. C1.default --> C2.default
C1.sasl <---> C2.sasl
C1.standard <--- C2.standard
3. Load 1000 keys into C1.default, C1.sasl, C2.standard. Expect 1000 keys in all buckets.
4. Offline upgrade C1 and C2 to 4.0.0-1869
5. Load 1000 keys each into C1.sasl, C2.sasl and C1.default.
6. Expect 3000 keys in sasl buckets, 2000 in default, 1000 in standard buckets.
C2.sasl and C2.default are missing 1000 keys loaded in step 5.
[2015-04-16 20:34:44,015] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 2000 == 2000 expected on C1, default bucket
[2015-04-16 20:34:44,113] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 3000 == 3000 expected on C1, sasl_bucket_1 bucket
[2015-04-16 20:34:44,195] - [xdcrnewbasetests:2007] INFO - Saw: vb_active_curr_items 1000 == 1000 expected on C1, standard_bucket_1 bucket
[2015-04-16 20:34:45,800] - [xdcrnewbasetests:1996] WARNING - Not Ready: vb_active_curr_items 1000 == 2000 expected on C2, default bucket
[2015-04-16 20:37:49,894] - [xdcrnewbasetests:1996] WARNING - Not Ready: vb_active_curr_items 2000 == 3000 expected on C2, sasl_bucket_1 bucket
Default bucket:
[2015-04-16 20:41:55,119] - [xdcrnewbasetests:2763] ERROR - 1000 keys not found on 10.1.2.19, printing first 100 keys: [('key: loadTwo216', 'vbucket: 403'), ('key: loadTwo510', 'vbucket: 447'), ('key: loadTwo351', 'vbucket: 601'), ('key: loadTwo753', 'vbucket: 862'), ('key: loadTwo457', 'vbucket: 629'), ('key: loadTwo268', 'vbucket: 362'), ('key: loadTwo906', 'vbucket: 221'), ('key: loadTwo614', 'vbucket: 148'), ('key: loadTwo318', 'vbucket: 1001'), ('key: loadTwo820', 'vbucket: 586'), ('key: loadTwo75', 'vbucket: 373'), ('key: loadTwo538', 'vbucket: 338'), ('key: loadTwo623', 'vbucket: 477'), ('key: loadTwo883', 'vbucket: 428'), ('key: loadTwo931', 'vbucket: 404'), ('key: loadTwo584', 'vbucket: 784'), ('key: loadTwo429', 'vbucket: 652'), ('key: loadTwo186', 'vbucket: 535'), ('key: loadTwo42', 'vbucket: 60'), ('key: loadTwo791', 'vbucket: 485'), ('key: loadTwo876', 'vbucket: 606'), ('key: loadTwo125', 'vbucket: 497'), ('key: loadTwo330', 'vbucket: 772'), ('key: loadTwo221', 'vbucket: 218'), ('key: loadTwo436', 'vbucket: 808'), ('key: loadTwo732', 'vbucket: 515'), ('key: loadTwo808', 'vbucket: 679'), ('key: loadTwo527', 'vbucket: 246'), ('key: loadTwo240', 'vbucket: 391'), ('key: loadTwo401', 'vbucket: 609'), ('key: loadTwo642', 'vbucket: 128'), ('key: loadTwo817', 'vbucket: 771'), ('key: loadTwo950', 'vbucket: 201'), ('key: loadTwo23', 'vbucket: 353'), ('key: loadTwo495', 'vbucket: 206'), ('key: loadTwo559', 'vbucket: 15'), ('key: loadTwo992', 'vbucket: 626'), ('key: loadTwo393', 'vbucket: 226'), ('key: loadTwo144', 'vbucket: 172'), ('key: loadTwo869', 'vbucket: 1018'), ('key: loadTwo307', 'vbucket: 589'), ('key: loadTwo199', 'vbucket: 947'), ('key: loadTwo80', 'vbucket: 647'), ('key: loadTwo282', 'vbucket: 828'), ('key: loadTwo173', 'vbucket: 485'), ('key: loadTwo705', 'vbucket: 842'), ('key: loadTwo460', 'vbucket: 828'), ('key: loadTwo978', 'vbucket: 36'), ('key: loadTwo571', 'vbucket: 226'), ('key: loadTwo277', 'vbucket: 206'), ('key: loadTwo675', 'vbucket: 457'), ('key: loadTwo379', 'vbucket: 692'), ('key: loadTwo841', 'vbucket: 791'), ('key: loadTwo14', 'vbucket: 40'), ('key: loadTwo967', 'vbucket: 384'), ('key: loadTwo546', 'vbucket: 427'), ('key: loadTwo209', 'vbucket: 55'), ('key: loadTwo680', 'vbucket: 571'), ('key: loadTwo112', 'vbucket: 184'), ('key: loadTwo366', 'vbucket: 784'), ('key: loadTwo764', 'vbucket: 535'), ('key: loadTwo448', 'vbucket: 977'), ('key: loadTwo919', 'vbucket: 377'), ('key: loadTwo219', 'vbucket: 300'), ('key: loadTwo690', 'vbucket: 800'), ('key: loadTwo102', 'vbucket: 419'), ('key: loadTwo909', 'vbucket: 98'), ('key: loadTwo292', 'vbucket: 551'), ('key: loadTwo458', 'vbucket: 714'), ('key: loadTwo594', 'vbucket: 523'), ('key: loadTwo196', 'vbucket: 780'), ('key: loadTwo341', 'vbucket: 834'), ('key: loadTwo135', 'vbucket: 234'), ('key: loadTwo500', 'vbucket: 164'), ('key: loadTwo743', 'vbucket: 581'), ('key: loadTwo426', 'vbucket: 563'), ('key: loadTwo830', 'vbucket: 849'), ('key: loadTwo893', 'vbucket: 183'), ('key: loadTwo231', 'vbucket: 449'), ('key: loadTwo633', 'vbucket: 198'), ('key: loadTwo921', 'vbucket: 143'), ('key: loadTwo439', 'vbucket: 919'), ('key: loadTwo52', 'vbucket: 295'), ('key: loadTwo807', 'vbucket: 536'), ('key: loadTwo189', 'vbucket: 680'), ('key: loadTwo528', 'vbucket: 73'), ('key: loadTwo485', 'vbucket: 469'), ('key: loadTwo320', 'vbucket: 543'), ('key: loadTwo154', 'vbucket: 439'), ('key: loadTwo818', 'vbucket: 956'), ('key: loadTwo722', 'vbucket: 792'), ('key: loadTwo866', 'vbucket: 837'), ('key: loadTwo317', 'vbucket: 854'), ('key: loadTwo556', 'vbucket: 176'), ('key: loadTwo250', 'vbucket: 156'), ('key: loadTwo411', 'vbucket: 890'), ('key: loadTwo652', 'vbucket: 411'), ('key: loadTwo715', 'vbucket: 593'), ('key: loadTwo940', 'vbucket: 466'), ('key: loadTwo33', 'vbucket: 122')]
Sasl bucket
[2015-04-16 20:42:00,299] - [xdcrnewbasetests:2763] ERROR - 1000 keys not found on 10.1.2.19, printing first 100 keys: [('key: loadThree508', 'vbucket: 282'), ('key: loadThree810', 'vbucket: 514'), ('key: loadThree698', 'vbucket: 670'), ('key: loadThree754', 'vbucket: 607'), ('key: loadThree957', 'vbucket: 456'), ('key: loadThree349', 'vbucket: 764'), ('key: loadThree613', 'vbucket: 405'), ('key: loadThree576', 'vbucket: 483'), ('key: loadThree672', 'vbucket: 200'), ('key: loadThree71', 'vbucket: 270'), ('key: loadThree827', 'vbucket: 843'), ('key: loadThree948', 'vbucket: 108'), ('key: loadThree583', 'vbucket: 529'), ('key: loadThree270', 'vbucket: 463'), ('key: loadThree181', 'vbucket: 790'), ('key: loadThree467', 'vbucket: 573'), ('key: loadThree406', 'vbucket: 864'), ('key: loadThree838', 'vbucket: 751'), ('key: loadThree936', 'vbucket: 149'), ('key: loadThree174', 'vbucket: 228'), ('key: loadThree478', 'vbucket: 921'), ('key: loadThree361', 'vbucket: 529'), ('key: loadThree796', 'vbucket: 228'), ('key: loadThree419', 'vbucket: 708'), ('key: loadThree10', 'vbucket: 83'), ('key: loadThree258', 'vbucket: 290'), ('key: loadThree763', 'vbucket: 790'), ('key: loadThree929', 'vbucket: 305'), ('key: loadThree84', 'vbucket: 764'), ('key: loadThree995', 'vbucket: 883'), ('key: loadThree624', 'vbucket: 220'), ('key: loadThree789', 'vbucket: 320'), ('key: loadThree115', 'vbucket: 441'), ('key: loadThree27', 'vbucket: 282'), ('key: loadThree859', 'vbucket: 946'), ('key: loadThree394', 'vbucket: 483'), ('key: loadThree300', 'vbucket: 844'), ('key: loadThree492', 'vbucket: 463'), ('key: loadThree226', 'vbucket: 475'), ('key: loadThree569', 'vbucket: 71'), ('key: loadThree431', 'vbucket: 553'), ('key: loadThree38', 'vbucket: 190'), ('key: loadThree901', 'vbucket: 476'), ('key: loadThree645', 'vbucket: 385'), ('key: loadThree337', 'vbucket: 517'), ('key: loadThree143', 'vbucket: 429'), ('key: loadThree702', 'vbucket: 587'), ('key: loadThree239', 'vbucket: 127'), ('key: loadThree541', 'vbucket: 170'), ('key: loadThree846', 'vbucket: 534'), ('key: loadThree520', 'vbucket: 503'), ('key: loadThree46', 'vbucket: 71'), ('key: loadThree328', 'vbucket: 929'), ('key: loadThree871', 'vbucket: 863'), ('key: loadThree285', 'vbucket: 573'), ('key: loadThree735', 'vbucket: 770'), ('key: loadThree450', 'vbucket: 884'), ('key: loadThree884', 'vbucket: 173'), ('key: loadThree517', 'vbucket: 190'), ('key: loadThree59', 'vbucket: 483'), ('key: loadThree356', 'vbucket: 856'), ('key: loadThree687', 'vbucket: 826'), ('key: loadThree122', 'vbucket: 240'), ('key: loadThree960', 'vbucket: 129'), ('key: loadThree211', 'vbucket: 146'), ('key: loadThree247', 'vbucket: 134'), ('key: loadThree507', 'vbucket: 421'), ('key: loadThree662', 'vbucket: 467'), ('key: loadThree894', 'vbucket: 438'), ('key: loadThree61', 'vbucket: 21'), ('key: loadThree958', 'vbucket: 375'), ('key: loadThree697', 'vbucket: 545'), ('key: loadThree260', 'vbucket: 212'), ('key: loadThree593', 'vbucket: 778'), ('key: loadThree786', 'vbucket: 511'), ('key: loadThree191', 'vbucket: 525'), ('key: loadThree579', 'vbucket: 348'), ('key: loadThree828', 'vbucket: 1012'), ('key: loadThree947', 'vbucket: 211'), ('key: loadThree105', 'vbucket: 162'), ('key: loadThree744', 'vbucket: 836'), ('key: loadThree468', 'vbucket: 642'), ('key: loadThree371', 'vbucket: 778'), ('key: loadThree409', 'vbucket: 991'), ('key: loadThree837', 'vbucket: 592'), ('key: loadThree773', 'vbucket: 525'), ('key: loadThree248', 'vbucket: 57'), ('key: loadThree939', 'vbucket: 42'), ('key: loadThree688', 'vbucket: 901'), ('key: loadThree384', 'vbucket: 248'), ('key: loadThree477', 'vbucket: 806'), ('key: loadThree655', 'vbucket: 154'), ('key: loadThree799', 'vbucket: 91'), ('key: loadThree416', 'vbucket: 635'), ('key: loadThree926', 'vbucket: 398'), ('key: loadThree257', 'vbucket: 413'), ('key: loadThree310', 'vbucket: 599'), ('key: loadThree164', 'vbucket: 511'), ('key: loadThree482', 'vbucket: 212'), ('key: loadThree28', 'vbucket: 421')]
Attached logs from C1 and C2.
In goxdcr log from C1, I see -
sasl_bucket_1_10.1.2.19:11210_26:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_12:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_10:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_19:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_7:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_21:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_3:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_25:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_0:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_31:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_2:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_1:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_20:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_18:dial tcp 10.1.2.19:11215: con
nection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_5:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_9:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_30:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_22:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_17:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_16:dial tcp 10.1.2.19:11215: connection refused xmem_b58c5f4927e19f6bee1a426f151f4622/sasl_bucket_1/sasl_bucket_1_10.1.2.19:11210_28:dial tcp 10.1.2.19:11215: connection refused]\n"}]}, progress=
]
Attachments
Issue Links
- relates to
-
MB-14231 GoXDCR: Unable to create encrypted remote cluster ref to older versions (cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs)
- Closed
For Gerrit Dashboard: MB-14600 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
49886,13 | MB-14600 GoXDCR: No keys replicated (encrypted xdcr) after offline upgrade to Sherlock [err: dial tcp 10.1.2.19:11215: connection refused] -fix a panic seen in test #12 in dev sanity panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x916c2] | master | goxdcr | Status: MERGED | +2 | +1 |
50104,3 | MB-14600 GoXDCR: No keys replicated (encrypted xdcr) after offline upgrade to Sherlock [err: dial tcp 10.1.2.19:11215: connection refused] | master | goxdcr | Status: MERGED | +2 | +1 |
50212,3 | MB-14600 GoXDCR: No keys replicated (encrypted xdcr) after offline upgrade to Sherlock [err: dial tcp 10.1.2.19:11215: connection refused] | master | goxdcr | Status: MERGED | +2 | +1 |
50231,6 | MB-14600 - add timeout to dial to tcp or tls to avoid hang when the ip is not reachable | master | goxdcr | Status: MERGED | +2 | +1 |