Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30179

rebalance exited with reason config_sync_failed from eventing node - centos longevity

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 5.5.0
    • 5.5.0
    • eventing
    • Untriaged
    • Yes

    Description

      Following rebalance failure seen in centos longevity from an eventing node - 5.5.0-2915:

      2018-06-19T16:27:26.371-07:00, auto_failover:1:info:message(ns_1@172.23.96.206) - Could not automatically fail over nodes (['ns_1@172.23.96.209']). Rebalance is running.
      2018-06-19T16:27:45.473-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.96.206) - Rebalance exited with reason {config_sync_failed,
                                    [{'ns_1@172.23.96.209',
                                      {exit,
                                       {shutdown,
                                        {gen_server,call,
                                         [{ns_config_rep,'ns_1@172.23.96.209'},
                                          synchronize_everything,infinity]}}}}]}
      

      From the eventing log of .209, at the same time stamp seeing following errors:

      2018-06-19T16:27:45.002-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
      2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools/default/b/DISTRICT: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:45.021-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:45.021-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.785-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:45.047-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/ui/index.html: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.798-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
      2018-06-19T16:27:45.202-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
      2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:45.221-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
      2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:06.807-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
      2018-06-19T16:27:45.286-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:45.290-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:45.290-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
      2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
      2018-06-19T16:27:45.346-07:00 [Info] DynamicAuthenticator::Credentials invoked by Consumer::gocbConnectBucketCallback, authenticating endpoint: 172.23.96.212:11210 bucket: default
      2018-06-19T16:27:45.346-07:00 [Error] DynamicAuthenticator::Credentials invoked by Consumer::gocbConnectBucketCallback, failed to get auth from cbauth
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:45.353-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
      2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
      2018-06-19T16:27:45.380-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      2018-06-19T16:27:06.824-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
      

      Supportal: https://supportal.couchbase.com/snapshot/9d575cf4cbd041d49772f53fc1a5d1c3::1

      Cluster: http://172.23.96.206:8091/

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            asingh Abhishek Singh (Inactive)
            arunkumar Arunkumar Senthilnathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty