Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Build
4.0.0-3321
Testcase
--------
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,upgrade_version=4.0.0-3321,initial_vbuckets=1024 -t "xdcr.upgradeXDCR.UpgradeTests.incremental_offline_upgrade,initial_version=2.5.0-1059-rel,replication_type=xmem,bucket_topology=default:1>2;sasl_bucket_1:1><2,upgrade_seq=src><dest"
Steps
1. C1 [.11, .16] <--> C2 [.19,.20]
C1.default --> C2.default
C1.sasl <--> C2.sasl
3. Install 2.5.0 in C1 and C2. Add some keys in every source bucket, verify replication.
4. Offline upgrade .11 in C1 to 4.0.0-3321, add 1000 keys on C1.sasl and C1.default
5. Offline upgrade .16 in C1 to 4.0.0-3321, add 1000 keys on C1.sasl and C1.default
6. Same as above for .19 and .20, one by one, add 1000 keys on C1.sasl and C1.default every time.
5. verify keys and revids.
[2015-06-30 11:35:46,702] - [xdcrnewbasetests:2190] INFO - Waiting for Outbound mutation to be zero on cluster node: 10.1.2.11
|
[2015-06-30 11:35:46,733] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:35:46,758] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:35:51,787] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:35:51,813] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:35:56,842] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:35:56,873] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:01,903] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:01,928] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:06,960] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:06,985] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:12,014] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:12,038] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:17,088] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:17,128] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:22,158] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:22,204] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:27,234] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:27,260] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:32,286] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:32,312] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:37,343] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:37,372] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:42,401] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:42,428] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:47,462] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:47,484] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:52,515] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:52,545] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:36:57,573] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:36:57,599] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:02,633] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:02,665] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:07,694] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:07,722] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:12,752] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:12,777] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:17,805] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:17,830] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:22,866] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:22,893] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:27,921] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:27,951] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:32,982] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:33,008] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:38,037] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:38,061] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:43,093] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:43,121] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:48,151] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:48,192] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:53,242] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:53,278] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:37:58,306] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:37:58,328] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:03,364] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:03,396] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:08,425] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:08,450] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:13,483] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:13,514] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:18,543] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:18,568] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:23,602] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:23,631] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:28,659] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:28,681] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:33,713] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:33,738] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:38,765] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:38,788] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:43,821] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 10
|
[2015-06-30 11:38:43,848] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C1 for bucket sasl_bucket_1 is 10
|
[2015-06-30 11:38:48,854] - [xdcrnewbasetests:2216] ERROR - Timeout occurs while waiting for mutations to be replicated
|
[2015-06-30 11:38:48,856] - [xdcrnewbasetests:2190] INFO - Waiting for Outbound mutation to be zero on cluster node: 10.1.2.19
|
[2015-06-30 11:38:48,872] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0
|
[2015-06-30 11:38:48,898] - [xdcrnewbasetests:2204] INFO - Current Outbound mutations on cluster node: C2 for bucket sasl_bucket_1 is 0
|
[2015-06-30 11:38:48,968] - [xdcrnewbasetests:2064] INFO - Saw: vb_active_curr_items 5000 == 5000 expected on C1, default bucket
|
[2015-06-30 11:38:49,031] - [xdcrnewbasetests:2064] INFO - Saw: vb_active_curr_items 6000 == 6000 expected on C1, sasl_bucket_1 bucket
|
[2015-06-30 11:38:49,091] - [xdcrnewbasetests:2098] INFO - Saw: vb_replica_curr_items 5000 == 5000 expected on C1, default bucket
|
[2015-06-30 11:38:49,150] - [xdcrnewbasetests:2098] INFO - Saw: vb_replica_curr_items 6000 == 6000 expected on C1, sasl_bucket_1 bucket
|
[2015-06-30 11:38:49,228] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:38:54,287] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:38:59,348] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:04,414] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:09,477] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:14,537] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:19,598] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:24,655] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:29,716] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:34,781] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:39,843] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:44,909] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:49,971] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:39:55,034] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:00,096] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:05,162] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:10,227] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:15,290] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:20,353] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:25,423] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:30,485] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:35,547] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:40,607] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:45,668] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:50,746] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:40:55,806] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:00,866] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:05,928] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:10,992] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:16,054] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:21,117] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:26,178] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:31,242] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:36,308] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:41,370] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:46,451] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:51,515] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:41:56,588] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:01,652] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:06,715] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:11,781] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:16,844] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:21,911] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:26,971] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:32,029] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:37,098] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:42,158] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:47,218] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:52,279] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:42:57,335] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:02,390] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:07,457] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:12,518] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:17,582] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:22,649] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:27,710] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:32,765] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:37,823] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:42,897] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:47,958] - [xdcrnewbasetests:2053] WARNING - Not Ready: vb_active_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:52,965] - [xdcrnewbasetests:2057] ERROR - ERROR: Timed-out waiting for active item count to match
|
[2015-06-30 11:43:53,040] - [xdcrnewbasetests:2064] INFO - Saw: vb_active_curr_items 6000 == 6000 expected on C2, sasl_bucket_1 bucket
|
[2015-06-30 11:43:53,104] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:56,159] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:43:59,212] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:02,277] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:05,341] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:08,404] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:11,467] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:14,525] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:17,584] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:20,654] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:23,712] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:26,768] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:29,839] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:32,896] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:35,952] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:39,008] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:42,067] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:45,129] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:48,188] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:51,247] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:54,301] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:44:57,357] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:00,419] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:03,481] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:06,552] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:09,609] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:12,672] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:15,730] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:18,784] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:21,840] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:24,908] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:27,961] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:31,026] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:34,084] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:37,145] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:40,200] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:43,259] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:46,316] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:49,375] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:52,434] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:55,488] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:45:58,543] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:01,603] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:04,668] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:07,723] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:10,778] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:13,833] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:16,890] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:19,947] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:23,008] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:26,063] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:29,122] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:32,184] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:35,254] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:38,316] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:41,375] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:44,438] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:47,500] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:50,561] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:53,617] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:56,676] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:46:59,735] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:02,804] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:05,864] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:08,922] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:11,979] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:15,038] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:18,105] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:21,165] - [xdcrnewbasetests:2086] WARNING - Not Ready: vb_replica_curr_items 4995 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:24,246] - [xdcrnewbasetests:2098] INFO - Saw: vb_replica_curr_items 5000 == 5000 expected on C2, default bucket
|
[2015-06-30 11:47:24,309] - [xdcrnewbasetests:2098] INFO - Saw: vb_replica_curr_items 6000 == 6000 expected on C2, sasl_bucket_1 bucket
|
On .16, I see goxdcr taking ~13 mins to detect xmem is stuck. Immediately it fixes the pipeline and replicates mutations across though.
|
StatisticsManager 2015-06-30T11:47:55.202-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2000, changes_left=500
|
StatisticsManager 2015-06-30T11:47:56.223-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2490, changes_left=10
|
StatisticsManager 2015-06-30T11:47:57.204-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2490, changes_left=10
|
StatisticsManager 2015-06-30T11:47:58.203-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2490, changes_left=10
|
:
|
:
|
StatisticsManager 2015-06-30T12:00:30.204-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2490, changes_left=10
|
StatisticsManager 2015-06-30T12:00:31.209-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2490, changes_left=10
|
StatisticsManager 2015-06-30T12:00:41.739-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:42.725-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:43.728-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:44.731-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:45.729-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:46.747-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:47.729-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=0, changes_left=2500
|
StatisticsManager 2015-06-30T12:00:48.728-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2500, changes_left=0
|
StatisticsManager 2015-06-30T12:00:49.727-07:00 [INFO] 39e729c88e82e50b50a6279d443ad16d/default/default total_docs=2500, docs_processed=2500, changes_left=0
|
|
|
XmemNozzle 2015-06-30T12:00:31.219-07:00 [ERROR] Xmem hasn't sent any item out for 124 ticks, 0 data in queue, flowcontrol=true, con_retry_limit=0, backoff_factor for client_setMeta is 0, backoff_factor for client_getMeta is 0
|
StatisticsManager 2015-06-30T12:00:31.220-07:00 [INFO] Xmem xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_52 state =2 connType=MemConn received 34 items, sent 34 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, getMeta's backoff_factor is 0, setMeta's backoff_factor is 0
|
XmemNozzle 2015-06-30T12:00:31.219-07:00 [INFO] xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_5 open=true checking..., 0 item unsent, received 49 items, sent 49 items, 10 items waiting for response, 0 batches ready, current batch timeout at 0001-01-01 00:00:00 +0000 UTC, current batch expire_ch size is 0
|
StatisticsManager 2015-06-30T12:00:31.220-07:00 [INFO] Xmem xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_0 state =2 connType=MemConn received 41 items, sent 41 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, getMeta's backoff_factor is 0, setMeta's backoff_factor is 0
|
StatisticsManager 2015-06-30T12:00:31.220-07:00 [INFO] Xmem xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_35 state =2 connType=MemConn received 41 items, sent 41 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, getMeta's backoff_factor is 0, setMeta's backoff_factor is 0
|
StatisticsManager 2015-06-30T12:00:31.220-07:00 [INFO] Xmem xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_46 state =2 connType=MemConn received 39 items, sent 39 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, getMeta's backoff_factor is 0, setMeta's backoff_factor is 0
|
StatisticsManager 2015-06-30T12:00:31.220-07:00 [INFO] Xmem xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_8 state =2 connType=MemConn received 48 items, sent 48 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, getMeta's backoff_factor is 0, setMeta's backoff_factor is 0
|
GenericSupervisor 2015-06-30T12:00:31.220-07:00 [ERROR] Received error report : map[xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_5:Xmem is stuck]
|
ReplicationManager 2015-06-30T12:00:31.220-07:00 [INFO] Supervisor PipelineSupervisor_39e729c88e82e50b50a6279d443ad16d/default/default of type *supervisor.GenericSupervisor reported errors map[xmem_39e729c88e82e50b50a6279d443ad16d/default/default_10.1.2.20:11210_5:Xmem is stuck]
|
Pls note, remote node is ready after warmup(from offline upgrade) by 11:33:05 itself.
[2015-06-30 11:33:03,937] - [rest_client:40] WARNING - server 10.1.2.20:8091 status is warmup
|
[2015-06-30 11:33:04,947] - [rest_client:40] WARNING - server 10.1.2.20:8091 status is warmup
|
[2015-06-30 11:33:05,957] - [cluster_helper:80] INFO - ns_server @ 10.1.2.20:8091 is running
|