Details
-
Bug
-
Resolution: Duplicate
-
Blocker
-
4.0.0
-
Security Level: Public
-
None
-
Untriaged
-
-
Unknown
Description
Build
4.0.0-3321
Testcase
---------
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,fail_on_errors=1,GROUP=P1 -t xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume,items=30000,rdirection=bidirection,ctopology=chain,replication_type=xmem,rebalance_in=C1-C2,pause_wait=10,pause=C1-C2,GROUP=P1
Steps
1. C1[.3,.4] <--> C2 [.20,.67]
2. Add 30k keys to C1.
3. Pause replication, start rebalance-in of .102 to C1 and .53 to C2.
4. Resume replication.
5. verify keys
Rebalance completed at 13:00 on C1 and C2.
[2015-06-30 13:00:54,317] - [task] [140508638734080] - INFO - rebalancing was completed with progress: 100% in 40.2104668617 sec
|
[2015-06-30 13:00:54,422] - [task] [140508630341376] - INFO - rebalancing was completed with progress: 100% in 40.201515913 sec
|
Test code shows keys missing until 13:20:32, 1800 keys from C2.
[2015-06-30 13:17:57,974] - [xdcrnewbasetests] [140508725962496] - WARNING - Not Ready: vb_replica_curr_items 58200 == 60000 expected on C1, default bucket
|
[2015-06-30 13:18:01,047] - [xdcrnewbasetests] [140508725962496] - WARNING - Not Ready: vb_replica_curr_items 58200 == 60000 expected on C1, default bucket
|
[2015-06-30 13:18:04,107] - [xdcrnewbasetests] [140508725962496] - WARNING - Not Ready: vb_replica_curr_items 58200 == 60000 expected on C1, default bucket
|
[2015-06-30 13:18:07,111] - [xdcrnewbasetests] [140508725962496] - ERROR - ERROR: Timed-out waiting for replica item count to match
|
[2015-06-30 13:18:07,111] - [remote_util] [140508725962496] - INFO - connecting to 172.23.105.3 with username : root password : couchbase ssh_key:
|
[2015-06-30 13:18:07,359] - [remote_util] [140508725962496] - INFO - Connected to 172.23.105.3
|
[2015-06-30 13:18:07,360] - [xdcrnewbasetests] [140508725962496] - INFO - Executing cbvdiff for bucket default
|
[2015-06-30 13:18:07,527] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: cat /proc/cpuinfo
|
[2015-06-30 13:18:07,610] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:07,610] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: df -Th
|
[2015-06-30 13:18:07,692] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:07,693] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: cat /proc/meminfo
|
[2015-06-30 13:18:07,775] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:07,775] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: hostname
|
[2015-06-30 13:18:07,857] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:07,857] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: hostname -d
|
[2015-06-30 13:18:07,939] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:07,940] - [remote_util] [140508725962496] - INFO - running command.raw on 172.23.105.3: /opt/couchbase/bin/cbvdiff -b default 172.23.105.3:11210,172.23.105.4:11210,172.23.105.102:11210
|
[2015-06-30 13:18:10,122] - [remote_util] [140508725962496] - INFO - command executed successfully
|
[2015-06-30 13:18:10,124] - [remote_util] [140508725962496] - INFO - Active item count = 58200
|
[2015-06-30 13:18:10,124] - [remote_util] [140508725962496] - INFO -
|
[2015-06-30 13:18:10,275] - [xdcrnewbasetests] [140508725962496] - INFO - Saw: vb_active_curr_items 60000 == 60000 expected on C2, default bucket
|
[2015-06-30 13:18:10,438] - [xdcrnewbasetests] [140508725962496] - INFO - Saw: vb_replica_curr_items 60000 == 60000 expected on C2, default bucket
|
[2015-06-30 13:18:10,637] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.3:11210 default
|
[2015-06-30 13:18:10,858] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.4:11210 default
|
[2015-06-30 13:18:12,013] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.102:11210 default
|
[2015-06-30 13:18:12,219] - [task] [140508725962496] - INFO - 60000 items will be verified on default bucket
|
[2015-06-30 13:18:12,378] - [task] [140508621948672] - INFO - 0 items were verified
|
[2015-06-30 13:18:12,626] - [xdcrnewbasetests] [140508725962496] - ERROR - Not able to get values for following keys - set(['C2-key-23041', 'C2-key-21239', 'C2-key-19815', 'C2-key-29573', 'C2-key-21884', 'C2-key-21960', 'C2-key-23803', 'C2-key-28963', 'C2-key-27761', 'C2-key-21247', 'C2-key-29146', 'C2-key-23474', 'C2-key-17515', 'C2-key-27873', 'C2-key-26842', 'C2-key-23372', 'C2-key-29391', 'C2-key-20494', 'C2-key-20392', 'C2-key-17670', 'C2-key-25519', 'C2-key-20570', 'C2-key-14818', 'C2-key-23069', 'C2-key-20951', 'C2-key-21948', 'C2-key-24921', 'C2-key-24163', 'C2-key-23590', 'C2-key-23296', 'C2-key-21569', 'C2-key-22058', 'C2-key-28244', 'C2-key-29138', 'C2-key-24135', 'C2-key-27825'])
|
[2015-06-30 13:18:12,627] - [xdcrnewbasetests] [140508725962496] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.105.3 -> Couchbase Cluster: C2, Master Ip: 172.23.105.20, bucket default
|
[2015-06-30 13:18:12,741] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.3:11210 default
|
[2015-06-30 13:18:12,826] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.4:11210 default
|
[2015-06-30 13:18:12,907] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.102:11210 default
|
[2015-06-30 13:18:13,063] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.3:11210 default
|
[2015-06-30 13:18:13,153] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.4:11210 default
|
[2015-06-30 13:18:13,239] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.102:11210 default
|
[2015-06-30 13:18:13,369] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.20:11210 default
|
[2015-06-30 13:18:13,508] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.105.67:11210 default
|
[2015-06-30 13:18:13,605] - [data_helper] [140508725962496] - INFO - creating direct client 172.23.107.53:11210 default
|
[2015-06-30 13:18:13,924] - [task] [140508725962496] - INFO - RevID verification: in progress for default ...
|
[2015-06-30 13:20:08,627] - [task] [140508621948672] - INFO - 50000 items have been verified
|
[2015-06-30 13:20:32,430] - [task] [140508621948672] - INFO - RevId Verification : 60000 existing items have been verified
|
[2015-06-30 13:20:32,431] - [task] [140508621948672] - INFO - RevId Verification : 0 deleted items have been verified
|
[2015-06-30 13:20:32,431] - [task] [140508621948672] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket
|
[2015-06-30 13:20:32,518] - [xdcrnewbasetests] [140508725962496] - ERROR - 1800 keys not found on 172.23.105.3, printing first 100 keys: [('key: C2-key-21247', 'vbucket: 380'), ('key: C2-key-17515', 'vbucket: 385'), ('key: C2-key-21239', 'vbucket: 389'), ('key: C2-key-20570', 'vbucket: 454'), ('key: C2-key-24921', 'vbucket: 462'), ('key: C2-key-22058', 'vbucket: 489'), ('key: C2-key-21884', 'vbucket: 343'), ('key: C2-key-27825', 'vbucket: 468'), ('key: C2-key-26842', 'vbucket: 342'), ('key: C2-key-17670', 'vbucket: 503'), ('key: C2-key-24135', 'vbucket: 427'), ('key: C2-key-23296', 'vbucket: 476'), ('key: C2-key-21960', 'vbucket: 379'), ('key: C2-key-23803', 'vbucket: 483'), ('key: C2-key-23041', 'vbucket: 402'), ('key: C2-key-23069', 'vbucket: 383'), ('key: C2-key-29573', 'vbucket: 455'), ('key: C2-key-27761', 'vbucket: 393'), ('key: C2-key-20392', 'vbucket: 454'), ('key: C2-key-29146', 'vbucket: 393'), ('key: C2-key-23474', 'vbucket: 476'), ('key: C2-key-20951', 'vbucket: 493'), ('key: C2-key-27873', 'vbucket: 448'), ('key: C2-key-28244', 'vbucket: 381'), ('key: C2-key-28963', 'vbucket: 378'), ('key: C2-key-24163', 'vbucket: 447'), ('key: C2-key-21569', 'vbucket: 445'), ('key: C2-key-19815', 'vbucket: 390'), ('key: C2-key-25519', 'vbucket: 414'), ('key: C2-key-29391', 'vbucket: 455'), ('key: C2-key-20494', 'vbucket: 490'), ('key: C2-key-21948', 'vbucket: 406'), ('key: C2-key-23590', 'vbucket: 496'), ('key: C2-key-29138', 'vbucket: 368'), ('key: C2-key-23372', 'vbucket: 496'), ('key: C2-key-14818', 'vbucket: 349'), ('key: C2-key-27749', 'vbucket: 356'), ('key: C2-key-26435', 'vbucket: 361'), ('key: C2-key-28212', 'vbucket: 361'), ('key: C2-key-27067', 'vbucket: 421'), ('key: C2-key-24250', 'vbucket: 477'), ('key: C2-key-29497', 'vbucket: 491'), ('key: C2-key-27354', 'vbucket: 455'), ('key: C2-key-22413', 'vbucket: 350'), ('key: C2-key-22708', 'vbucket: 465'), ('key: C2-key-20526', 'vbucket: 466'), ('key: C2-key-17092', 'vbucket: 503'), ('key: C2-key-20145', 'vbucket: 392'), ('key: C2-key-29223', 'vbucket: 511'), ('key: C2-key-23711', 'vbucket: 426'), ('key: C2-key-29952', 'vbucket: 492'), ('key: C2-key-22315', 'vbucket: 370'), ('key: C2-key-24633', 'vbucket: 391'), ('key: C2-key-27737', 'vbucket: 413'), ('key: C2-key-20643', 'vbucket: 420'), ('key: C2-key-26463', 'vbucket: 381'), ('key: C2-key-22343', 'vbucket: 358'), ('key: C2-key-26281', 'vbucket: 381'), ('key: C2-key-22589', 'vbucket: 395'), ('key: C2-key-22832', 'vbucket: 373'), ('key: C2-key-29110', 'vbucket: 413'), ('key: C2-key-20113', 'vbucket: 412'), ('key: C2-key-25531', 'vbucket: 371'), ('key: C2-key-29616', 'vbucket: 433'), ('key: C2-key-28887', 'vbucket: 342'), ('key: C2-key-29275', 'vbucket: 491'), ('key: C2-key-29904', 'vbucket: 504'), ('key: C2-key-24500', 'vbucket: 485'), ('key: C2-key-23747', 'vbucket: 446'), ('key: C2-key-24665', 'vbucket: 403'), ('key: C2-key-19057', 'vbucket: 503'), ('key: C2-key-28659', 'vbucket: 478'), ('key: C2-key-25385', 'vbucket: 359'), ('key: C2-key-21211', 'vbucket: 360'), ('key: C2-key-25938', 'vbucket: 437'), ('key: C2-key-20276', 'vbucket: 490'), ('key: C2-key-25249', 'vbucket: 422'), ('key: C2-key-24977', 'vbucket: 474'), ('key: C2-key-25798', 'vbucket: 452'), ('key: C2-key-28388', 'vbucket: 444'), ('key: C2-key-21936', 'vbucket: 367'), ('key: C2-key-27183', 'vbucket: 393'), ('key: C2-key-22864', 'vbucket: 353'), ('key: C2-key-25237', 'vbucket: 351'), ('key: C2-key-25567', 'vbucket: 359'), ('key: C2-key-23855', 'vbucket: 503'), ('key: C2-key-23324', 'vbucket: 484'), ('key: C2-key-23422', 'vbucket: 456'), ('key: C2-key-27997', 'vbucket: 492'), ('key: C2-key-26028', 'vbucket: 458'), ('key: C2-key-27685', 'vbucket: 421'), ('key: C2-key-29525', 'vbucket: 467'), ('key: C2-key-27404', 'vbucket: 511'), ('key: C2-key-24556', 'vbucket: 497'), ('key: C2-key-14439', 'vbucket: 374'), ('key: C2-key-28935', 'vbucket: 366'), ('key: C2-key-27452', 'vbucket: 491'), ('key: C2-key-27031', 'vbucket: 433'), ('key: C2-key-19332', 'vbucket: 385'), ('key: C2-key-24893', 'vbucket: 502')]
|
However around this time, there are no mutations waiting to be sent on any of C2 nodes -
On. 20-
|
StatisticsManager 2015-06-30T13:08:52.648-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20385, docs_processed=17842, changes_left=2543
|
StatisticsManager 2015-06-30T13:08:53.640-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20385, docs_processed=17842, changes_left=2543
|
StatisticsManager 2015-06-30T13:08:54.640-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20385, docs_processed=20385, changes_left=0
|
|
|
|
|
On .67
|
StatisticsManager 2015-06-30T13:01:24.469-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20848, docs_processed=19987, changes_left=861
|
StatisticsManager 2015-06-30T13:01:25.459-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20848, docs_processed=20848, changes_left=0
|
|
|
On. .53, replication is slower-
|
StatisticsManager 2015-06-30T13:19:29.697-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20254, docs_processed=16050, changes_left=4204
|
StatisticsManager 2015-06-30T13:19:30.711-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20254, docs_processed=17065, changes_left=3189
|
StatisticsManager 2015-06-30T13:19:31.696-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20254, docs_processed=20254, changes_left=0
|
StatisticsManager 2015-06-30T13:19:32.695-07:00 [INFO] 27544327e7b7568c13f0676ec1cef61e/default/default total_docs=20254, docs_processed=20254, changes_left=0
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-