Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
7.1.0-1885
The senario was simple with no special configuration. Create a 3 node cluster. 1 node kv,index,n1ql, 1 node kv,fts and 1 node kv,cbas. The rebalance hung at the index stage for at least 30 minutes before I destroyed the cluster.
In the logs I see lots of
2021-12-16T20:57:35.692+00:00 [Info] RebalanceServiceManager::GetCurrentTopology returns &{[0 0 0 0 0 0 0 145] [e0994a7139aba04106bd5b3e1bbaabb5] true []}
|
2021-12-16T20:57:35.692+00:00 [Info] RebalanceServiceManager::GetTaskList returns &{[0 0 0 0 0 0 0 145] [{[0 0 0 0 0 0 0 0] prepare/4aab91571ed60fb9974f54c9bc3fa8b3 task-prepared task-running true 0 map[] map[rebalanceId:4aab91571ed60fb9974f54c9bc3fa8b3]} {[0 0 0 0 0 0 0 139] rebalance/4aab91571ed60fb9974f54c9bc3fa8b3 task-rebalance task-running true 0.1 map[] map[rebalanceId:4aab91571ed60fb9974f54c9bc3fa8b3]}]}
|
2021-12-16T20:57:35.695+00:00 [Info] RebalanceServiceManager::GetCurrentTopology [0 0 0 0 0 0 0 145]
|
2021-12-16T20:57:35.695+00:00 [Info] RebalanceServiceManager::GetTaskList [0 0 0 0 0 0 0 145]
|
2021-12-16T20:57:37.911+00:00 [Info] Rebalancer::checkAllIndexersWarmedup Indexer 172.23.111.152:9102 State Warmup
|
2021-12-16T20:57:37.912+00:00 [Error] Rebalancer::initRebalAsync All Indexers Not Active. Waiting...
|
2021-12-16T20:57:39.271+00:00 [Info] KVSender::sendShutdownTopic Projector 172.23.111.150:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5
|
2021-12-16T20:57:39.272+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.111.150:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5. Err projector.topicMissing
|
2021-12-16T20:57:39.272+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.111.150:9999
|
2021-12-16T20:57:39.272+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
|
2021-12-16T20:57:39.272+00:00 [Info] KVSender::sendShutdownTopic Projector 172.23.111.151:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5
|
2021-12-16T20:57:39.273+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.111.151:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5. Err projector.topicMissing
|
2021-12-16T20:57:39.273+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.111.151:9999
|
2021-12-16T20:57:39.273+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
|
2021-12-16T20:57:39.273+00:00 [Info] KVSender::sendShutdownTopic Projector 172.23.111.152:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5
|
2021-12-16T20:57:39.274+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.111.152:9999 Topic MAINT_STREAM_TOPIC_e0994a7139aba04106bd5b3e1bbaabb5. Err Post "http://172.23.111.152:9999/adminport/shutdownTopicRequest": dial tcp 172.23.111.152:9999: connect: connection refused
|
2021-12-16T20:57:39.274+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post "http://172.23.111.152:9999/adminport/shutdownTopicRequest": dial tcp 172.23.111.152:9999: connect: connection refused from 172.23.111.152:9999
|
2021-12-16T20:57:40.293+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM Error from Projector Post "http://172.23.111.152:9999/adminport/shutdownTopicRequest": dial tcp 172.23.111.152:9999: connect: connection refused
|
2021-12-16T20:57:40.293+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post "http://172.23.111.152:9999/adminport/shutdownTopicRequest": dial tcp 172.23.111.152:9999: connect: connection refused. Retrying (120).
|