Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34937

[System Test] Fatal error in indexer log with Stream MAINT_STREAM Projector health check needed, indexer can not proceed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.5.0
    • 6.5.0
    • secondary-index
    • component cluster
    • Untriaged
    • Unknown

    Description

      Build : 6.5.0-3687
      Test : -test tests/integration/test_allFeatures_alice_timers.yml -scope tests/integration/scope_Xattrs_Alice.yml
      Scale : 3
      Iteration : 1st

      In index.los we see below errors in index nodes (172.23.104.67, 172.23.104.88). Looks like these errors have been thrown while adding index node to the cluster.

      step in system test while we see these errors in log:

      [2019-07-06T11:33:02-07:00, sequoiatools/couchbase-cli:6.5:0c29f1] server-add -c 172.23.108.103 -u Administrator -p password --server-add 172.23.104.88 --server-add-username Administrator --server-add-password password --services index
      [2019-07-06T11:33:07-07:00, sequoiatools/couchbase-cli:6.5:dad243] server-add -c 172.23.108.103 -u Administrator -p password --server-add 172.23.104.67 --server-add-username Administrator --server-add-password password --services index

       
      		 
      In 172.23.104.67
      		 
       
      		 
      2019-07-06T11:33:38.763-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from 172.23.99.21:9999
      		 
      2019-07-06T11:33:38.763-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      		 
      2019-07-06T11:33:38.763-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.25:9999 Topic MAINT_STREAM_TOPIC_99eb3104acc2959256afb68e5d2813ab
      		 
      2019-07-06T11:33:38.764-07:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.25:9999 Topic MAINT_STREAM_TOPIC_99eb3104acc2959256afb68e5d2813ab. Err projector.topicMissing
      		 
      2019-07-06T11:33:38.764-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from 172.23.99.25:9999
      		 
      2019-07-06T11:33:38.764-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      		 
      2019-07-06T11:33:39.060-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
      		 
      2019-07-06T11:33:39.764-07:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://172.23.106.100:9999/adminport/shutdownTopicRequest: dial tcp 172.23.106.100:9999: getsockopt: connection refused
      		 
      2019-07-06T11:33:39.764-07:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://172.23.106.100:9999/adminport/shutdownTopicRequest: dial tcp 172.23.106.100:9999: getsockopt: connection refused. Retrying.
      		 
       
      		 
       
      		 
      In 172.23.104.88
      		 
       
      		 
      2019-07-06T11:33:38.749-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from 172.23.99.21:9999
      		 
      2019-07-06T11:33:38.749-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      		 
      2019-07-06T11:33:38.749-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.25:9999 Topic MAINT_STREAM_TOPIC_9ee77b546e28302314c8d348933d5767
      		 
      2019-07-06T11:33:38.750-07:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.25:9999 Topic MAINT_STREAM_TOPIC_9ee77b546e28302314c8d348933d5767. Err projector.topicMissing
      		 
      2019-07-06T11:33:38.750-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received projector.topicMissing from 172.23.99.25:9999
      		 
      2019-07-06T11:33:38.750-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM  Treating projector.topicMissing As Success
      		 
      2019-07-06T11:33:39.030-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
      		 
      2019-07-06T11:33:39.750-07:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://172.23.106.100:9999/adminport/shutdownTopicRequest: dial tcp 172.23.106.100:9999: getsockopt: connection refused
      		 
      2019-07-06T11:33:39.750-07:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://172.23.106.100:9999/adminport/shutdownTopicRequest: dial tcp 172.23.106.100:9999: getsockopt: connection refused. Retrying.
      		 
      2019-07-06T11:33:40.947-07:00 [Error] getLocalMetadataFromDisk(): fail to read metadata from file /data/@2i/cache/meta/172_23_104_61_9102.  Error open /data/@2i/cache/meta/172_23_104_61_9102: no such file or directory
      		 
      2019-07-06T11:33:40.949-07:00 [Error] getLocalMetadataFromDisk(): fail to read metadata from file /data/@2i/cache/meta/172_23_104_67_9102.  Error open /data/@2i/cache/meta/172_23_104_67_9102: no such file or directory
      		 
      2019-07-06T11:33:40.951-07:00 [Error] getLocalMetadataFromDisk(): fail to read metadata from file /data/@2i/cache/meta/172_23_104_87_9102.  Error open /data/@2i/cache/meta/172_23_104_87_9102: no such file or directory
      		 
       

      https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.164.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.61.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.67.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.69.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.70.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.87.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.104.88.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.106.100.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.108.103.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.148.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.251.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.252.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.253.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.56.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.96.95.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.97.119.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.97.121.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.97.122.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.97.239.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.97.242.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.98.135.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.99.20.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.99.21.zip
      url : https://s3.amazonaws.com/bugdb/jira/systestmon-1562607614/collectinfo-2019-07-08T174023-ns_1%40172.23.99.25.zip

      Attachments

        Activity

          People

            varun.velamuri Varun Velamuri
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty