Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19348

msg': u'dial tcp 127.0.0.1:9101: connection refused from 127.0.0.1:9101,crash seen in forestdb

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 4.5.0
    • 4.5.0
    • forestdb
    • 4.5.0-2339 on centos
    • Untriaged
    • Unknown
    • ForestDB: Oct 17 - Nov 4

    Description

      Error reported:
      CBQError: CBQError: host 172.23.106.43: ERROR:{u'status': u'errors', u'errors': [

      {u'msg': u'dial tcp 127.0.0.1:9101: connection refused from 127.0.0.1:9101 - cause: dial tcp 127.0.0.1:9101: connection refused from 127.0.0.1:9101', u'code': 5000}

      ], u'results': [], u'metrics':

      {u'elapsedTime': u'35.083689ms', u'executionTime': u'35.012364ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}

      , u'requestID': u'6c089f62-62d2-462d-acc7-fd8eed2f6e5b', u'signature': None}

      Error in indexer.log:
      2016-04-24T12:36:13.068-07:00 [Error] CompactionDaemon: Index instance:17779980336642564825 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_9_#primary_6889695861393417808_0.index/data.fdb.77' crc 93 != f4 (crc in doc) keylen 772 metalen 209 bodylen 2818048 bodylen_ondisk 0 offset 802816
      2016-04-24T12:37:15.807-07:00 [Error] CompactionDaemon: Index instance:6889695861393417808 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.78' crc 56 != 85 (crc in doc) keylen 34058 metalen 96 bodylen 2818048 bodylen_ondisk 0 offset 1073152
      2016-04-24T12:37:16.388-07:00 [Error] CompactionDaemon: Index instance:4588834641880184788 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_2_#primary_4572981166022363354_0.index/data.fdb.88' crc 83 != 52 (crc in doc) keylen 19718 metalen 205 bodylen 2818048 bodylen_ondisk 0 offset 954368
      2016-04-24T12:41:02.573-07:00 [Error] CompactionDaemon: Index instance:4572981166022363354 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_1_#primary_13116844550727948074_0.index/data.fdb.91' crc 2e != 3a (crc in doc) keylen 15110 metalen 85 bodylen 2818048 bodylen_ondisk 0 offset 958464
      2016-04-24T12:43:43.876-07:00 [Error] CompactionDaemon: Index instance:13116844550727948074 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_8_#primary_12391024330846393856_0.index/data.fdb.94' crc a0 != 45 (crc in doc) keylen 17926 metalen 181 bodylen 2686976 bodylen_ondisk 0 offset 1028096
      2016-04-24T12:45:22.345-07:00 [Error] CompactionDaemon: Index instance:12391024330846393856 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_6_#primary_6999420921784126490_0.index/data.fdb.98' crc 84 != 67 (crc in doc) keylen 25350 metalen 220 bodylen 2818048 bodylen_ondisk 0 offset 978944
      2016-04-24T12:46:24.112-07:00 [Error] CompactionDaemon: Index instance:6999420921784126490 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_9_#primary_6889695861393417808_0.index/data.fdb.94' crc 1d != f7 (crc in doc) keylen 1287 metalen 211 bodylen 2818048 bodylen_ondisk 0 offset 978944
      2016-04-24T12:46:57.123-07:00 [Error] CompactionDaemon: Index instance:6889695861393417808 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_7_#primary_761779540605963233_0.index/data.fdb.99' crc 12 != 3f (crc in doc) keylen 11270 metalen 135 bodylen 2818048 bodylen_ondisk 0 offset 991232
      2016-04-24T12:48:01.570-07:00 [Error] CompactionDaemon: Index instance:761779540605963233 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_1_#primary_13116844550727948074_0.index/data.fdb.98' crc 42 != 5c (crc in doc) keylen 22534 metalen 157 bodylen 2818048 bodylen_ondisk 0 offset 962560
      2016-04-24T12:48:02.092-07:00 [Error] CompactionDaemon: Index instance:13116844550727948074 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.97' crc 82 != 27 (crc in doc) keylen 8454 metalen 161 bodylen 2818048 bodylen_ondisk 0 offset 991232
      2016-04-24T12:48:02.138-07:00 [Error] CompactionDaemon: Index instance:4588834641880184788 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_6_#primary_6999420921784126490_0.index/data.fdb.103' crc 60 != fc (crc in doc) keylen 61702 metalen 60 bodylen 2818048 bodylen_ondisk 0 offset 978944
      2016-04-24T12:49:36.883-07:00 [Error] CompactionDaemon: Index instance:6999420921784126490 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_2_#primary_4572981166022363354_0.index/data.fdb.105' crc 6c != aa (crc in doc) keylen 41990 metalen 117 bodylen 2818048 bodylen_ondisk 0 offset 978944
      2016-04-24T12:50:40.477-07:00 [Error] CompactionDaemon: Index instance:4572981166022363354 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_5_#primary_17779980336642564825_0.index/data.fdb.108' crc a3 != 0 (crc in doc) keylen 0 metalen 11008 bodylen 0 bodylen_ondisk -309459456 offset 978944
      2016-04-24T12:52:18.390-07:00 [Error] CompactionDaemon: Index instance:17779980336642564825 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_5_#primary_17779980336642564825_0.index/data.fdb.111' crc 62 != b5 (crc in doc) keylen 46857 metalen 74 bodylen 2818048 bodylen_ondisk 0 offset 1015808
      2016-04-24T12:54:26.270-07:00 [Error] CompactionDaemon: Index instance:17779980336642564825 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_7_#primary_761779540605963233_0.index/data.fdb.111' crc 99 != 16 (crc in doc) keylen 6148 metalen 57 bodylen 2818048 bodylen_ondisk 0 offset 811008
      2016-04-24T12:54:58.826-07:00 [Error] CompactionDaemon: Index instance:761779540605963233 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_6_#primary_6999420921784126490_0.index/data.fdb.113' crc d9 != a2 (crc in doc) keylen 34054 metalen 182 bodylen 2818048 bodylen_ondisk 0 offset 987136
      2016-04-24T12:55:30.532-07:00 [Error] CompactionDaemon: Index instance:6999420921784126490 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_2_#primary_4572981166022363354_0.index/data.fdb.123' crc 12 != 7e (crc in doc) keylen 25606 metalen 209 bodylen 2752512 bodylen_ondisk 0 offset 978944
      2016-04-24T13:00:52.793-07:00 [Error] CompactionDaemon: Index instance:4572981166022363354 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_9_#primary_6889695861393417808_0.index/data.fdb.126' crc c2 != 42 (crc in doc) keylen 16387 metalen 6 bodylen 2818048 bodylen_ondisk 0 offset 757760
      2016-04-24T13:04:35.701-07:00 [Error] CompactionDaemon: Index instance:6889695861393417808 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_8_#primary_12391024330846393856_0.index/data.fdb.130' crc 83 != 79 (crc in doc) keylen 24582 metalen 237 bodylen 2818048 bodylen_ondisk 0 offset 974848
      2016-04-24T13:05:10.072-07:00 [Error] CompactionDaemon: Index instance:12391024330846393856 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_4_#primary_2211223255726655258_0.index/data.fdb.134' crc e6 != 64 (crc in doc) keylen 25862 metalen 198 bodylen 2818048 bodylen_ondisk 0 offset 966656
      2016-04-24T13:06:46.445-07:00 [Error] CompactionDaemon: Index instance:2211223255726655258 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] Fatal error!!! Database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_4_#primary_2211223255726655258_0.index/data.fdb.139' is corrupted.
      [FDB ERR] Error in reading the doc metadata with offset 974848, length 19712 from a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_4_#primary_2211223255726655258_0.index/data.fdb.139'
      2016-04-24T13:10:00.255-07:00 [Error] CompactionDaemon: Index instance:2211223255726655258 Compaction failed with reason - FDB_RESULT_FILE_CORRUPTION
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_6_#primary_6999420921784126490_0.index/data.fdb.144' crc a != be (crc in doc) keylen 47619 metalen 156 bodylen 2818048 bodylen_ondisk 0 offset 802816
      2016-04-24T13:12:38.712-07:00 [Error] CompactionDaemon: Index instance:6999420921784126490 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_1_#primary_13116844550727948074_0.index/data.fdb.147' crc 89 != e4 (crc in doc) keylen 57600 metalen 210 bodylen 2818048 bodylen_ondisk 0 offset 786432
      2016-04-24T13:14:47.168-07:00 [Error] CompactionDaemon: Index instance:13116844550727948074 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_2_#primary_4572981166022363354_0.index/data.fdb.149' crc 8d != 77 (crc in doc) keylen 33030 metalen 167 bodylen 2818048 bodylen_ondisk 0 offset 1007616
      2016-04-24T13:15:19.319-07:00 [Error] CompactionDaemon: Index instance:4572981166022363354 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_4_#primary_2211223255726655258_0.index/data.fdb.148' crc e0 != 4e (crc in doc) keylen 18185 metalen 25 bodylen 2752512 bodylen_ondisk 0 offset 991232
      2016-04-24T13:15:19.676-07:00 [Error] CompactionDaemon: Index instance:2211223255726655258 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_5_#primary_17779980336642564825_0.index/data.fdb.151' crc e7 != 30 (crc in doc) keylen 14343 metalen 164 bodylen 2818048 bodylen_ondisk 0 offset 1032192
      2016-04-24T13:16:22.700-07:00 [Error] CompactionDaemon: Index instance:17779980336642564825 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_1_#primary_13116844550727948074_0.index/data.fdb.149' crc 99 != ca (crc in doc) keylen 51971 metalen 220 bodylen 2818048 bodylen_ondisk 0 offset 1032192
      2016-04-24T13:16:22.774-07:00 [Error] CompactionDaemon: Index instance:13116844550727948074 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length body checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.149' crc 7f != ab (crc in doc) keylen 44038 metalen 190 bodylen 2818048 bodylen_ondisk 0 offset 1007616
      2016-04-24T13:16:22.881-07:00 [Error] CompactionDaemon: Index instance:4588834641880184788 Compaction failed with reason - FDB_RESULT_CHECKSUM_ERROR
      [FDB ERR] doc_length key checksum mismatch error in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.150' crc 8c != 2c (crc in doc) keylen 12332 metalen 12588 bodylen 741880114 bodylen_ondisk 825633840 offset 1819145
      [FDB ERR] docio_read_doc_key error: read failure on offset 1819145 in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.150' : FDB status -15, lastbid 0x1bc, curblock 0x273, curpos 0xfaa
      0020 73 2d 6c 69 73 74 00 00 00 00 00 00 00 00 00 00 s-l[FDB ERR] Fatal error!!! Database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.150' is corrupted.
      [FDB ERR] Error in reading a doc with offset 2428928, length 142676 from a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.150'
      2016-04-24T13:17:27.901-07:00 [Error] CompactionDaemon: Index instance:4588834641880184788 Compaction failed with reason - FDB_RESULT_FILE_CORRUPTION
      2016-04-24T13:17:30.666-07:00 [Error] WatcherServer.runOnce() error : dial tcp 127.0.0.1:9100: getsockopt: connection refused
      2016-04-24T13:17:31.671-07:00 [Error] WatcherServer.runOnce() error : dial tcp 127.0.0.1:9100: getsockopt: connection refused
      2016-04-24T13:17:31.918-07:00 [Error] updateIndexerList(): Cannot locate cluster node.
      2016-04-24T13:17:34.395-07:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_95:f8:94:9a:b1:d1:60:3. Err projector.topicMissing
      2016-04-24T13:17:34.395-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
      2016-04-24T13:17:35.091-07:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_95:f8:94:9a:b1:d1:60:3. Err projector.topicMissing
      2016-04-24T13:17:35.091-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
      2016-04-24T13:17:35.290-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named copy_simple_table Nids: []
      2016-04-24T13:17:36.856-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_1 Nids: []
      2016-04-24T13:17:37.240-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_2 Nids: []
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_4_#primary_2211223255726655258_0.index/data.fdb.150', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_6_#primary_6999420921784126490_0.index/data.fdb.151', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_8_#primary_12391024330846393856_0.index/data.fdb.151', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_3_#primary_4588834641880184788_0.index/data.fdb.149', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_7_#primary_761779540605963233_0.index/data.fdb.151', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_5_#primary_17779980336642564825_0.index/data.fdb.152', errno = 2: 'No such file or directory'
      [FDB ERR] Error in OPEN on a database file '/opt/couchbase/var/lib/couchbase/data/@2i/simple_table_9_#primary_6889695861393417808_0.index/data.fdb.148', errno = 2: 'No such file or directory'
      2016-04-24T13:17:41.274-07:00 [Error] KVSender::openMutationStream MAINT_STREAM simple_table_3 Error in fetching vbuckets info No bucket named simple_table_3
      2016-04-24T13:17:41.274-07:00 [Error] Indexer::startBucketStream Stream MAINT_STREAM Bucket simple_table_3 Error from Projector No bucket named simple_table_3. Retrying.
      2016-04-24T13:17:45.202-07:00 [Error] feed.DcpGetSeqnos(): EOF
      2016-04-24T13:17:45.204-07:00 [Warn] Indexer::getCurrentKVTs error=EOF Retrying (1)
      2016-04-24T13:17:45.412-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:45.413-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:45.535-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:45.536-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:45.672-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:45.673-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:46.767-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:46.768-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:46.783-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_3 Nids: []
      2016-04-24T13:17:46.783-07:00 [Error] Indexer::startBucketStream
      2016-04-24T13:17:46.828-07:00 [Error] KVSender::sendMutationTopicRequest Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_95:f8:94:9a:b1:d1:60:3 simple_table_5
      Unexpected Error feed.clusterInfo
      2016-04-24T13:17:46.828-07:00 [Error] KVSender::openMutationStream MAINT_STREAM simple_table_5 Error Received feed.clusterInfo from 127.0.0.1:9999
      2016-04-24T13:17:46.848-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:46.849-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:46.930-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:46.931-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:46.999-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:47.000-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:47.076-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:47.077-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:47.163-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:47.165-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:47.235-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:47.236-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:47.829-07:00 [Error] KVSender::openMutationStream MAINT_STREAM simple_table_5 Error Received feed.clusterInfo
      2016-04-24T13:17:47.829-07:00 [Error] Indexer::startBucketStream Stream MAINT_STREAM Bucket simple_table_5 Error from Projector feed.clusterInfo. Retrying.
      2016-04-24T13:17:48.305-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.306-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (1)
      2016-04-24T13:17:48.365-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.366-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:48.428-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.429-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:48.512-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.513-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:48.596-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.597-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:48.678-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.680-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:48.752-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.754-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:48.831-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.832-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:48.906-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.907-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:48.971-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:48.972-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:49.041-07:00 [Error] KVSender::sendMutationTopicRequest Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_95:f8:94:9a:b1:d1:60:3 simple_table_4
      Unexpected Error feed.clusterInfo
      2016-04-24T13:17:49.041-07:00 [Error] KVSender::openMutationStream MAINT_STREAM simple_table_4 Error Received feed.clusterInfo from 127.0.0.1:9999
      2016-04-24T13:17:49.046-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:49.047-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:50.042-07:00 [Error] KVSender::openMutationStream MAINT_STREAM simple_table_4 Error Received feed.clusterInfo
      2016-04-24T13:17:50.042-07:00 [Error] Indexer::startBucketStream Stream MAINT_STREAM Bucket simple_table_4 Error from Projector feed.clusterInfo. Retrying.
      2016-04-24T13:17:50.712-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:50.713-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (1)
      2016-04-24T13:17:50.794-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:50.796-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:50.880-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:50.881-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:51.010-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.011-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:51.063-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.064-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:51.101-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_3
      2016-04-24T13:17:51.116-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.118-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:51.182-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.183-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:51.238-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.239-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:51.328-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.329-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:51.372-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.373-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:51.417-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:51.418-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:51.844-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM simple_table_6 Error in fetching cluster info No bucket named simple_table_6
      2016-04-24T13:17:51.886-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_6 Nids: []
      2016-04-24T13:17:51.887-07:00 [Error] Timekeeper::sendRestartMsg Bucket Not Found For Stream MAINT_STREAM Bucket simple_table_6
      2016-04-24T13:17:51.954-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_6
      2016-04-24T13:17:52.832-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:52.834-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (1)
      2016-04-24T13:17:52.871-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:52.872-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:52.874-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_5 Nids: []
      2016-04-24T13:17:52.874-07:00 [Error] Indexer::startBucketStream
      2016-04-24T13:17:52.913-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:52.914-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:52.925-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_5
      2016-04-24T13:17:52.973-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:52.974-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:53.017-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.018-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:53.055-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.056-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:53.082-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.084-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:53.122-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.123-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:53.158-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.159-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:53.189-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.190-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:53.235-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:53.236-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:54.634-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM simple_table_7 Error in fetching cluster info No bucket named simple_table_7
      2016-04-24T13:17:54.636-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.637-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (1)
      2016-04-24T13:17:54.663-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.665-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:54.665-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_7 Nids: []
      2016-04-24T13:17:54.666-07:00 [Error] Timekeeper::sendRestartMsg Bucket Not Found For Stream MAINT_STREAM Bucket simple_table_7
      2016-04-24T13:17:54.694-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.695-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:54.708-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_7
      2016-04-24T13:17:54.722-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.723-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:54.760-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.762-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:54.797-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.798-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:54.825-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.826-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:54.853-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.855-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:54.874-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.875-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:54.898-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.899-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:54.917-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:54.919-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:55.062-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_4 Nids: []
      2016-04-24T13:17:55.062-07:00 [Error] Indexer::startBucketStream
      2016-04-24T13:17:55.089-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_4
      2016-04-24T13:17:55.758-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM simple_table_8 Error in fetching cluster info No bucket named simple_table_8
      2016-04-24T13:17:55.784-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_8 Nids: []
      2016-04-24T13:17:55.784-07:00 [Error] Timekeeper::sendRestartMsg Bucket Not Found For Stream MAINT_STREAM Bucket simple_table_8
      2016-04-24T13:17:55.819-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_8
      2016-04-24T13:17:55.940-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:55.941-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (1)
      2016-04-24T13:17:55.963-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:55.964-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (2)
      2016-04-24T13:17:55.980-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:55.981-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (3)
      2016-04-24T13:17:55.998-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.000-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (4)
      2016-04-24T13:17:56.015-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.016-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (5)
      2016-04-24T13:17:56.032-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.033-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (6)
      2016-04-24T13:17:56.054-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.055-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (7)
      2016-04-24T13:17:56.073-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.074-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (8)
      2016-04-24T13:17:56.092-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.093-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (9)
      2016-04-24T13:17:56.107-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.108-07:00 [Warn] Indexer::getCurrentKVTs error=No bucket named simple_table_4 Retrying (10)
      2016-04-24T13:17:56.127-07:00 [Error] Unable to connect with bucket "simple_table_4"
      2016-04-24T13:17:56.128-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster No bucket named simple_table_4
      2016-04-24T13:17:57.129-07:00 [Error] Timekeeper::handleStats Error occured while obtaining KV seqnos - No bucket named simple_table_4
      2016-04-24T13:17:58.180-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM simple_table_9 Error in fetching cluster info No bucket named simple_table_9
      2016-04-24T13:17:58.187-07:00 [Fatal] Indexer::Error Fetching Bucket Info: No bucket named simple_table_9 Nids: []
      2016-04-24T13:17:58.187-07:00 [Error] Timekeeper::sendRestartMsg Bucket Not Found For Stream MAINT_STREAM Bucket simple_table_9
      2016-04-24T13:17:58.189-07:00 [Error] receiving packet: read tcp 127.0.0.1:9105->127.0.0.1:41733: use of closed network connection
      2016-04-24T13:17:58.189-07:00 [Error] DATP[->dataport ":9105"] worker "127.0.0.1:41733" exit: read tcp 127.0.0.1:9105->127.0.0.1:41733: use of closed network connection
      2016-04-24T13:17:58.208-07:00 [Error] Indexer::stopBucketStream Success Stream MAINT_STREAM Bucket simple_table_9

      Attaching core file and logs.

      Attachments

        1. core
          464 kB
        2. logs.zip.gz
          17.09 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jung-sang Jung-Sang Ahn (Inactive)
            Prerna.Manaktala Prerna Manaktala (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty