Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54639

indexer process is not starting both auto and manual

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • Unknown
    • Critical

    Description

      dear team,

      we have 3 node cluster  , one of the pod went down , now we are readd back to cluster index rebalance is failing for 3rd node and it is giving below error. could you please help me how to fix this.

       

       

      Err projector.topicMissing
      2022-11-22T14:47:02.160+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from [240b:c0e0:104:5450:9727:2:7:b07]:9999
      2022-11-22T14:47:02.160+09:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      2022-11-22T14:47:03.379+09:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
      2022-11-22T14:47:03.380+09:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host. Retrying (49).
      2022-11-22T14:47:08.380+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:47:09.160+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host
      2022-11-22T14:47:09.160+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b05]:9999
      2022-11-22T14:47:09.161+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:47:09.164+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
      2022-11-22T14:47:09.164+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b06]:9999
      2022-11-22T14:47:09.164+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:47:09.166+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err projector.topicMissing
      2022-11-22T14:47:09.166+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from [240b:c0e0:104:5450:9727:2:7:b07]:9999
      2022-11-22T14:47:09.167+09:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      2022-11-22T14:47:10.469+09:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
      2022-11-22T14:47:10.469+09:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host. Retrying (50).
      2022-11-22T14:47:13.013+09:00 [Info] PeriodicStats = {"cpu_utilization":0.3999733351109926,"index_not_found_errcount":0,"indexer_state":"Warmup","memory_free":17314021376,"memory_quota":15728640000,"memory_rss":88281088,"memory_total":34359738368,"memory_total_storage":0,"memory_used":0,"memory_used_queue":0,"memory_used_storage":0,"needs_restart":false,"num_connections":0,"num_cpu_core":8,"storage_mode":"not_set","timestamp":"1669096033013623850","timings/stats_response":"476 34881146 2585171853490","uptime":"8m8.122551509s"}
      2022-11-22T14:47:15.470+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:47:16.170+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b05]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host
      2022-11-22T14:47:16.170+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b05]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b05]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b05]:9999
      2022-11-22T14:47:16.170+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:47:16.173+09:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector [240b:c0e0:104:5450:9727:2:7:b06]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2. Err Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host
      2022-11-22T14:47:16.173+09:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://[240b:c0e0:104:5450:9727:2:7:b06]:9999/adminport/shutdownTopicRequest: dial tcp [240b:c0e0:104:5450:9727:2:7:b06]:9999: getsockopt: no route to host from [240b:c0e0:104:5450:9727:2:7:b06]:9999
      2022-11-22T14:47:16.173+09:00 [Info] KVSender::sendShutdownTopic Projector [240b:c0e0:104:5450:9727:2:7:b07]:9999 Topic MAINT_STREAM_TOPIC_41f904fba96e64e9753bdb7c39514af2
      2022-11-22T14:53:31.267+09:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=[::1]:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=41f904fba96e64e9753bdb7c39514af2 -ipv6=true -isEnterprise=true]
      2022-11-22T14:53:31.303+09:00 [Info] Indexer::NewIndexer Status Warmup
      2022-11-22T14:53:31.577+09:00 [Error] GetSettingsConfig() failed: Get http://[::1]:8091/_metakv/indexing/settings/config: Unable to find given hostport in cbauth database: `[::1]:8091'
      2022-11-22T14:53:34.577+09:00 [Error] GetSettingsConfig() failed: Get http://[::1]:8091/_metakv/indexing/settings/config: Unable to find given hostport in cbauth database: `[::1]:8091'

      Attachments

        Activity

          People

            varun.velamuri Varun Velamuri
            kanamani92 naveen karanam
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty