Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
5.5.0
-
Untriaged
-
Yes
Description
Following rebalance failure seen in centos longevity from an eventing node - 5.5.0-2915:
2018-06-19T16:27:26.371-07:00, auto_failover:1:info:message(ns_1@172.23.96.206) - Could not automatically fail over nodes (['ns_1@172.23.96.209']). Rebalance is running.
|
2018-06-19T16:27:45.473-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.96.206) - Rebalance exited with reason {config_sync_failed,
|
[{'ns_1@172.23.96.209',
|
{exit,
|
{shutdown,
|
{gen_server,call,
|
[{ns_config_rep,'ns_1@172.23.96.209'},
|
synchronize_everything,infinity]}}}}]}
|
From the eventing log of .209, at the same time stamp seeing following errors:
2018-06-19T16:27:45.002-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
|
2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools/default/b/DISTRICT: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:45.021-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:45.021-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.785-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:45.047-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/ui/index.html: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:06.798-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.798-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
|
2018-06-19T16:27:45.202-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
|
2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:45.221-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
|
2018-06-19T16:27:06.806-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:06.807-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
|
2018-06-19T16:27:45.286-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:45.290-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:45.290-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(3)
|
2018-06-19T16:27:06.806-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(1)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(2)
|
2018-06-19T16:27:45.346-07:00 [Info] DynamicAuthenticator::Credentials invoked by Consumer::gocbConnectBucketCallback, authenticating endpoint: 172.23.96.212:11210 bucket: default
|
2018-06-19T16:27:45.346-07:00 [Error] DynamicAuthenticator::Credentials invoked by Consumer::gocbConnectBucketCallback, failed to get auth from cbauth
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:45.353-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(4)
|
2018-06-19T16:27:06.824-07:00 [Info] Error occured during cluster info update (Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address) .. Retrying(5)
|
2018-06-19T16:27:45.380-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
2018-06-19T16:27:06.824-07:00 [Error] Consumer::getKvVbMap [worker_bucket_op_complex_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_complex_function_1.sock:24746] Failed to grab vbMap for bucket: default, err: Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: connect: cannot assign requested address
|
Supportal: https://supportal.couchbase.com/snapshot/9d575cf4cbd041d49772f53fc1a5d1c3::1
Cluster: http://172.23.96.206:8091/