Details
Description
Build
-------
4.0.0-1573
Adding node failed, also seeing many "Exceeded retries count trying to get consistent keys/values for config replication" errors
[2015-03-16 13:11:35,170] - [xdcrnewbasetests:1770] INFO - Starting rebalance-in nodes:[ip:10.3.2.215 port:8091 ssh_username:root] at C1 cluster 10.3.5.201
[2015-03-16 13:11:35,859] - [task:324] INFO - adding node 10.3.2.215:8091 to cluster
[2015-03-16 13:11:35,859] - [rest_client:947] INFO - adding remote node @10.3.2.215:8091 to this cluster @10.3.5.201:8091
[2015-03-16 13:11:36,272] - [rest_client:730] ERROR - http://10.3.5.201:8091/controller/addNode error 500 reason: unknown ["Unexpected server error, request logged."]
[2015-03-16 13:11:36,553] - [rest_client:2169] INFO - Latest logs from UI on 10.3.5.201:
[2015-03-16 13:11:36,554] - [rest_client:2170] ERROR -
[2015-03-16 13:11:36,554] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.496Z', u'module': u'ns_config_rep', u'tstamp': 1426536694496, u'type': u'warning'}[2015-03-16 13:11:36,555] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.308Z', u'module': u'ns_config_rep', u'tstamp': 1426536694308, u'type': u'warning'}[2015-03-16 13:11:36,555] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Replication from bucket "default" to bucket "default" on cluster "remote_cluster_C1-C2" created.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.726Z', u'module': u'xdcr', u'tstamp': 1426536632726, u'type': u'info'}[2015-03-16 13:11:36,556] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Created remote cluster reference "remote_cluster_C1-C2" via 10.3.2.202:8091.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.663Z', u'module': u'xdcr', u'tstamp': 1426536632663, u'type': u'info'}[2015-03-16 13:11:36,556] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.202\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.910Z', u'module': u'ns_memcached', u'tstamp': 1426536625910, u'type': u'info'}[2015-03-16 13:11:36,557] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.203\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.898Z', u'module': u'ns_memcached', u'tstamp': 1426536625898, u'type': u'info'}[2015-03-16 13:11:36,558] - [rest_client:2170] ERROR -
{u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.201\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.819Z', u'module': u'ns_memcached', u'tstamp': 1426536625819, u'type': u'info'}[2015-03-16 13:11:36,559] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 12, u'text': u'Created bucket "default" of type: couchbase\n[
{num_replicas,1},\n {replica_index,true},\n {ram_quota,2169503744},\n {auth_type,sasl},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only}]', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.719Z', u'module': u'menelaus_web', u'tstamp': 1426536625719, u'type': u'info'}[2015-03-16 13:11:36,559] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Reset auto-failover count', u'shortText': u'message', u'serverTime': u'2015-03-16T13:09:56.728Z', u'module': u'auto_failover', u'tstamp': 1426536596728, u'type': u'info'}
[2015-03-16 13:11:36,759] - [rest_client:2169] INFO - Latest logs from UI on 10.3.2.215:
[2015-03-16 13:11:36,760] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.684Z', u'module': u'ns_config_rep', u'tstamp': 1426536694684, u'type': u'warning'}
[2015-03-16 13:11:36,761] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.496Z', u'module': u'ns_config_rep', u'tstamp': 1426536694496, u'type': u'warning'}
[2015-03-16 13:11:36,762] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.308Z', u'module': u'ns_config_rep', u'tstamp': 1426536694308, u'type': u'warning'}
[2015-03-16 13:11:36,762] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Replication from bucket "default" to bucket "default" on cluster "remote_cluster_C1-C2" created.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.726Z', u'module': u'xdcr', u'tstamp': 1426536632726, u'type': u'info'}
[2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Created remote cluster reference "remote_cluster_C1-C2" via 10.3.2.202:8091.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.663Z', u'module': u'xdcr', u'tstamp': 1426536632663, u'type': u'info'}
[2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.202\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.910Z', u'module': u'ns_memcached', u'tstamp': 1426536625910, u'type': u'info'}
[2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.203\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.898Z', u'module': u'ns_memcached', u'tstamp': 1426536625898, u'type': u'info'}
[2015-03-16 13:11:36,766] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.201\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.819Z', u'module': u'ns_memcached', u'tstamp': 1426536625819, u'type': u'info'}
[2015-03-16 13:11:36,766] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 12, u'text': u'Created bucket "default" of type: couchbase\n[{num_replicas,1}
,\n
{replica_index,true},\n
{ram_quota,2169503744},\n
{auth_type,sasl},\n
{flush_enabled,true},\n
{num_threads,3},\n
{eviction_policy,value_only}]', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.719Z', u'module': u'menelaus_web', u'tstamp': 1426536625719, u'type': u'info'}
[2015-03-16 13:11:36,767] - [rest_client:2170] ERROR -
[2015-03-16 13:11:36,768] - [rest_client:991] ERROR - add_node error : ["Unexpected server error, request logged."]
ERROR