Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-13968

goxdcr should not update all checkpoints at the same time (was Adding node failed ("Unexpected server error, request logged"))

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.0.0
    • 4.0.0
    • XDCR
    • Security Level: Public
    • Triaged
    • Centos 64-bit
    • Yes

    Description

      Build
      -------
      4.0.0-1573

      Adding node failed, also seeing many "Exceeded retries count trying to get consistent keys/values for config replication" errors

      [2015-03-16 13:11:35,170] - [xdcrnewbasetests:1770] INFO - Starting rebalance-in nodes:[ip:10.3.2.215 port:8091 ssh_username:root] at C1 cluster 10.3.5.201
      [2015-03-16 13:11:35,859] - [task:324] INFO - adding node 10.3.2.215:8091 to cluster
      [2015-03-16 13:11:35,859] - [rest_client:947] INFO - adding remote node @10.3.2.215:8091 to this cluster @10.3.5.201:8091
      [2015-03-16 13:11:36,272] - [rest_client:730] ERROR - http://10.3.5.201:8091/controller/addNode error 500 reason: unknown ["Unexpected server error, request logged."]
      [2015-03-16 13:11:36,553] - [rest_client:2169] INFO - Latest logs from UI on 10.3.5.201:
      [2015-03-16 13:11:36,554] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.684Z', u'module': u'ns_config_rep', u'tstamp': 1426536694684, u'type': u'warning'}

      [2015-03-16 13:11:36,554] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.496Z', u'module': u'ns_config_rep', u'tstamp': 1426536694496, u'type': u'warning'}

      [2015-03-16 13:11:36,555] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.308Z', u'module': u'ns_config_rep', u'tstamp': 1426536694308, u'type': u'warning'}

      [2015-03-16 13:11:36,555] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Replication from bucket "default" to bucket "default" on cluster "remote_cluster_C1-C2" created.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.726Z', u'module': u'xdcr', u'tstamp': 1426536632726, u'type': u'info'}

      [2015-03-16 13:11:36,556] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Created remote cluster reference "remote_cluster_C1-C2" via 10.3.2.202:8091.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.663Z', u'module': u'xdcr', u'tstamp': 1426536632663, u'type': u'info'}

      [2015-03-16 13:11:36,556] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.202\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.910Z', u'module': u'ns_memcached', u'tstamp': 1426536625910, u'type': u'info'}

      [2015-03-16 13:11:36,557] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.203\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.898Z', u'module': u'ns_memcached', u'tstamp': 1426536625898, u'type': u'info'}

      [2015-03-16 13:11:36,558] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.201\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.819Z', u'module': u'ns_memcached', u'tstamp': 1426536625819, u'type': u'info'}

      [2015-03-16 13:11:36,559] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 12, u'text': u'Created bucket "default" of type: couchbase\n[

      {num_replicas,1},\n {replica_index,true},\n {ram_quota,2169503744},\n {auth_type,sasl},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only}]', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.719Z', u'module': u'menelaus_web', u'tstamp': 1426536625719, u'type': u'info'}
      [2015-03-16 13:11:36,559] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Reset auto-failover count', u'shortText': u'message', u'serverTime': u'2015-03-16T13:09:56.728Z', u'module': u'auto_failover', u'tstamp': 1426536596728, u'type': u'info'}
      [2015-03-16 13:11:36,759] - [rest_client:2169] INFO - Latest logs from UI on 10.3.2.215:
      [2015-03-16 13:11:36,760] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.684Z', u'module': u'ns_config_rep', u'tstamp': 1426536694684, u'type': u'warning'}
      [2015-03-16 13:11:36,761] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.496Z', u'module': u'ns_config_rep', u'tstamp': 1426536694496, u'type': u'warning'}
      [2015-03-16 13:11:36,762] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Exceeded retries count trying to get consistent keys/values for config replication. This is minor bug. Everything is safe, but please file bug and attach logs', u'shortText': u'message', u'serverTime': u'2015-03-16T13:11:34.308Z', u'module': u'ns_config_rep', u'tstamp': 1426536694308, u'type': u'warning'}
      [2015-03-16 13:11:36,762] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Replication from bucket "default" to bucket "default" on cluster "remote_cluster_C1-C2" created.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.726Z', u'module': u'xdcr', u'tstamp': 1426536632726, u'type': u'info'}
      [2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Created remote cluster reference "remote_cluster_C1-C2" via 10.3.2.202:8091.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:32.663Z', u'module': u'xdcr', u'tstamp': 1426536632663, u'type': u'info'}
      [2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.202', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.202\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.910Z', u'module': u'ns_memcached', u'tstamp': 1426536625910, u'type': u'info'}
      [2015-03-16 13:11:36,764] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.203', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.203\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.898Z', u'module': u'ns_memcached', u'tstamp': 1426536625898, u'type': u'info'}
      [2015-03-16 13:11:36,766] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.201\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.819Z', u'module': u'ns_memcached', u'tstamp': 1426536625819, u'type': u'info'}
      [2015-03-16 13:11:36,766] - [rest_client:2170] ERROR - {u'node': u'ns_1@10.3.5.201', u'code': 12, u'text': u'Created bucket "default" of type: couchbase\n[{num_replicas,1}

      ,\n

      {replica_index,true}

      ,\n

      {ram_quota,2169503744}

      ,\n

      {auth_type,sasl}

      ,\n

      {flush_enabled,true}

      ,\n

      {num_threads,3}

      ,\n

      {eviction_policy,value_only}

      ]', u'shortText': u'message', u'serverTime': u'2015-03-16T13:10:25.719Z', u'module': u'menelaus_web', u'tstamp': 1426536625719, u'type': u'info'}
      [2015-03-16 13:11:36,767] - [rest_client:2170] ERROR -

      {u'node': u'ns_1@10.3.5.201', u'code': 0, u'text': u'Reset auto-failover count', u'shortText': u'message', u'serverTime': u'2015-03-16T13:09:56.728Z', u'module': u'auto_failover', u'tstamp': 1426536596728, u'type': u'info'}

      [2015-03-16 13:11:36,768] - [rest_client:991] ERROR - add_node error : ["Unexpected server error, request logged."]
      ERROR

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty