Details
Description
Replica not working wherein replica items remains 0 when replica_count >=1
Jenkins Link:
http://qa.hq.northscale.net/job/centos_x64--44_02--replica_read_tests-P0/38/consoleFull
http://qa.sc.couchbase.com/view/3.0.0/job/ubuntu_x64--47_02--replica_read-P1/29/consoleFull
Test to Reproduce:
./testrunner -i /tmp/replica_read.ini get-logs=True,wait_timeout=180,GROUP=P0,get-cbcollect-info=True,get-delays=true -t newmemcapable.GetrTests.getr_test,nodes_init=4,GROUP=P0,descr=#simple getr replica_count=1 expiration_and_flags = 0 docs_ops=create cluster_ops = None
*All other tests failed
2014-04-11 01:14:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items_tot 1000 == 2000 expected on '10.1.3.74:8091''10.1.3.75:8091''10.1.3.80:8091''10.1.2.88:8091', default bucket
2014-04-11 01:14:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 0 == 1000 expected on '10.1.3.74:8091''10.1.3.75:8091''10.1.3.80:8091''10.1.2.88:8091', default bucket
2014-04-11 01:14:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items_tot 1000 == 2000 expected on '10.1.3.74:8091''10.1.3.75:8091''10.1.3.80:8091''10.1.2.88:8091', default bucket
2014-04-11 01:14:30 | ERROR | MainProcess | MainThread | [basetestcase._verify_stats_all_buckets] unable to get expected stats for any node! Print taps for all nodes:
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.88:11210 default
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 10.1.2.88
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:paused true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:ack_log_size 2
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:has_queued_item true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:idle false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:backfill_completed false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:ack_seqno 101
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:ack_window_full false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno 98
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.2.88 :eq_tapq:replication_ns_1@10.1.3.80:pending_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.74:11210 default
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 10.1.3.74
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:pending_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:paused true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:ack_seqno 122
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:ack_window_full false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:has_queued_item true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:ack_log_size 21
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno 100
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:backfill_completed false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.74 :eq_tapq:replication_ns_1@10.1.3.75:idle false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.75:11210 default
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 10.1.3.75
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:has_queued_item true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:pending_disk_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:ack_log_size 60
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:idle false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:paused false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:ack_window_full false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:pending_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:ack_seqno 150
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:recv_ack_seqno 89
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.75 :eq_tapq:replication_ns_1@10.1.2.88:backfill_completed true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.80:11210 default
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 10.1.3.80
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:has_queued_item true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno 89
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:ack_seqno 150
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:paused false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:ack_window_full false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:backfill_completed true
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:pending_backfill false
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:ack_log_size 60
2014-04-11 01:14:30 | INFO | MainProcess | MainThread | [rebalance_helper.log_interesting_taps] TAP ns_1@10.1.3.80 :eq_tapq:replication_ns_1@10.1.3.74:idle false
Uploading Logs