Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
6.6.1
-
Pre-prod
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
-
Critical
Description
dear team,
we have 3 node cluster , one of the pod went down , now we are readd back to cluster index rebalance is failing for 3rd node and it is giving below error. could you please help me how to fix this.
Err projector.topicMissing
2022-11-22T14:47:02.160+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from [240b:c0e0:104:5450:9727:2:7:b07]:9999
2022-11-22T14:47:02.160+09:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2022-11-22T14:47:03.379+09:00 [Error] KVSender::closeMutationStream, MAINT_STREAM Error from Projector Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
2022-11-22T14:47:03.380+09:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host. Retrying (49).
2022-11-22T14:47:08.380+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:47:09.160+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host
2022-11-22T14:47:09.160+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b05]:9999
2022-11-22T14:47:09.161+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:47:09.164+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
2022-11-22T14:47:09.164+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b06]:9999
2022-11-22T14:47:09.164+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:47:09.166+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err projector.topicMissing
2022-11-22T14:47:09.166+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from [240b:c0e0:104:5450:9727:2:7:b07]:9999
2022-11-22T14:47:09.167+09:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2022-11-22T14:47:10.469+09:00 [Error] KVSender::closeMutationStream, MAINT_STREAM Error from Projector Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
2022-11-22T14:47:10.469+09:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host. Retrying (50).
2022-11-22T14:47:13.013+09:00 [Info] PeriodicStats = {"cpu_utilization":0.3999733351109926,"index_not_found_errcount":0,"indexer_state":"Warmup","memory_free":17314021376,"memory_quota":15728640000,"memory_rss":88281088,"memory_total":34359738368,"memory_total_storage":0,"memory_used":0,"memory_used_queue":0,"memory_used_storage":0,"needs_restart":false,"num_connections":0,"num_cpu_core":8,"storage_mode":"not_set","timestamp":"1669096033013623850","timings/stats_response":"476 34881146 2585171853490","uptime":"8m8.122551509s"}
2022-11-22T14:47:15.470+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:47:16.170+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host
2022-11-22T14:47:16.170+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b05]:9999
2022-11-22T14:47:16.170+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:47:16.173+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
2022-11-22T14:47:16.173+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b06]:9999
2022-11-22T14:47:16.173+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
2022-11-22T14:53:31.267+09:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=[::1]:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=41f904fba96e64e9753bdb7c39514af2 -ipv6=true -isEnterprise=true]
2022-11-22T14:53:31.303+09:00 [Info] Indexer::NewIndexer Status Warmup
2022-11-22T14:53:31.577+09:00 [Error] GetSettingsConfig() failed: Get http://[::1]:8091/_metakv/indexing/settings/config: Unable to find given hostport in cbauth database: `[::1]:8091'
2022-11-22T14:53:34.577+09:00 [Error] GetSettingsConfig() failed: Get http://[::1]:8091/_metakv/indexing/settings/config: Unable to find given hostport in cbauth database: `[::1]:8091'