0 vb_replica_curr_items after offline upgrade 2.2.0-837-rel -> 3.0.0-791-rel

http://qa.hq.northscale.net/job/centos_x64--104_01--XDCR_upgrade-P1/7/consoleFull

[Test]
./testrunner i centos_x64104_01-XDCR_upgrade-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,upgrade_version=3.0.0-791-rel,initial_vbuckets=1024 -t xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.2.0-837-rel,sdata=False,bucket_topology=default:1>2;bucket0:1><2,upgrade_nodes=src,post-upgrade-actions=src-rebalancein;dest-rebalanceout;dest-create_index

[Test Error]
[2014-06-09 11:39:15,608] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 bucket0
[2014-06-09 11:39:15,701] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 bucket0
[2014-06-09 11:39:15,782] - [task:442] INFO - Saw curr_items 3000 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091',bucket0 bucket
[2014-06-09 11:39:15,794] - [data_helper:289] INFO - creating direct client 10.3.3.126:11210 bucket0
[2014-06-09 11:39:15,890] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 bucket0
[2014-06-09 11:39:15,993] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 bucket0
[2014-06-09 11:39:16,082] - [task:442] INFO - Saw vb_active_curr_items 3000 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091',bucket0 bucket
[2014-06-09 11:39:16,096] - [data_helper:289] INFO - creating direct client 10.3.3.126:11210 bucket0
[2014-06-09 11:39:16,201] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 bucket0
[2014-06-09 11:39:16,302] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 bucket0
[2014-06-09 11:39:16,384] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:16,398] - [data_helper:289] INFO - creating direct client 10.3.3.126:11210 default
[2014-06-09 11:39:16,492] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 default
[2014-06-09 11:39:16,582] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 default
[2014-06-09 11:39:16,664] - [task:442] INFO - Saw curr_items 2000 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091',default bucket
[2014-06-09 11:39:16,677] - [data_helper:289] INFO - creating direct client 10.3.3.126:11210 default
[2014-06-09 11:39:16,773] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 default
[2014-06-09 11:39:16,865] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 default
[2014-06-09 11:39:16,949] - [task:442] INFO - Saw vb_active_curr_items 2000 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091',default bucket
[2014-06-09 11:39:16,963] - [data_helper:289] INFO - creating direct client 10.3.3.126:11210 default
[2014-06-09 11:39:17,067] - [data_helper:289] INFO - creating direct client 10.3.5.61:11210 default
[2014-06-09 11:39:17,164] - [data_helper:289] INFO - creating direct client 10.3.5.60:11210 default
[2014-06-09 11:39:17,246] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:22,274] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:22,298] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:27,327] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:27,349] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:32,379] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:32,404] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:37,433] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:37,457] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:42,486] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:42,511] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:47,540] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:47,562] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:52,593] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:52,618] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:39:57,648] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:39:57,671] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:02,700] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:02,722] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:07,754] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:07,777] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:12,811] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:12,833] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:17,865] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:17,890] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:22,919] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:22,945] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:27,975] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:27,998] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:33,027] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:33,053] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:38,084] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:38,115] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:43,144] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:43,167] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:48,205] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:48,231] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:53,260] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:53,283] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:40:58,312] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:40:58,336] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:03,365] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:03,391] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:08,426] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:08,450] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:13,481] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:13,503] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:18,533] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:18,556] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:23,589] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:23,621] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:28,652] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:28,675] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:33,702] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:33,724] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:38,753] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:38,777] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:43,806] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:43,828] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:48,861] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:48,884] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:53,916] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:53,939] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:41:58,968] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:41:58,990] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:04,018] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:04,040] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:09,067] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:09,090] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:14,120] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:14,143] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:19,174] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:19,198] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:24,229] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:24,252] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:29,280] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:29,302] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
[2014-06-09 11:42:34,331] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 3000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', bucket0 bucket
[2014-06-09 11:42:34,352] - [task:438] WARNING - Not Ready: vb_replica_curr_items 0 == 2000 expected on '10.3.3.126:8091''10.3.5.61:8091''10.3.5.60:8091', default bucket
ERROR
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('./testrunner', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/xdcr/upgradeXDCR.py', 211, 'offline_cluster_upgrade', 'self.verify_xdcr_stats(self.src_nodes, self.dest_nodes, True)'), ('pytests/xdcr/xdcrbasetests.py', 1381, 'verify_xdcr_stats', 'self._verify_item_count(src_nodes, timeout=timeout)'), ('pytests/xdcr/xdcrbasetests.py', 1319, '_verify_item_count', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Mon Jun 9 11:42:36 2014
[2014-06-09 11:42:36,110] - [xdcrbasetests:154] WARNING - CLEANUP WAS SKIPPED DUE TO FAILURES IN UPGRADE TEST

======================================================================
ERROR: offline_cluster_upgrade (xdcr.upgradeXDCR.UpgradeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "pytests/xdcr/upgradeXDCR.py", line 211, in offline_cluster_upgrade
self.verify_xdcr_stats(self.src_nodes, self.dest_nodes, True)
File "pytests/xdcr/xdcrbasetests.py", line 1381, in verify_xdcr_stats
self._verify_item_count(src_nodes, timeout=timeout)
File "pytests/xdcr/xdcrbasetests.py", line 1327, in _verify_item_count
raise XDCRException("Item count verification was not completed in

seconds".format(timeout))
XDCRException: Item count verification was not completed in 200 seconds

----------------------------------------------------------------------
Ran 1 test in 1899.258s

[Test Steps]
1. Install all nodes with 2.2.0-837
2. Setup 2-2 nodes Source and Destination Cluster.
3. CAPI Mode bucket0 <----> bucket0 (BiXDCR), and default --> default (UniXDCR) replication.
4. Load 1000 items on each bucket.
5. Offline upgrade Source Nodes i.e. '10.3.3.126:8091, 10.3.5.61:8091''
6. Add one node 10.3.5.60 to Source node.
7. Remove node 10.3.121.99 from cluster 10.3.5.11.
8. Verify items. --> curr_items and vb_active_curr_items are fine, while replica count is 0.