Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19023

[performance tests] Indexer stuck in fdb_commit

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • forestdb
    • 4.5.0-2016
    • Untriaged
    • Unknown
    • ForestDB: Oct 17 - Nov 4

    Description

      test http://perf.jenkins.couchbase.com/job/hades/3437/
      tests/n1ql/n1ql_lat_CI3_20M_18Kops_960Qops_gsi_ok.test

      2016-04-03T09:11:58.533-07:00 [Info] CompactionDaemon: Compacting index instance:8397725242306989303
      assertion failed [rsize && rsize <= trie->chunksize] at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/hbtrie.cc:84 ((nil) != 0x7fcffa41ec30)
      Called from:
      2016-04-03T09:11:58.552-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 22568504. Slice Id 0, IndexInstId 8397725242306989303, IndexDefnId 8397725242306989303
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebe7464]
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebce127]
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebd1c8d]
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebccb78]
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebe2815]
              /opt/couchbase/bin/../lib/libforestdb.so() [0x7fe06ebc3649]
              /opt/couchbase/bin/indexer(_cgo_816fefc71821_Cfunc_fdb_commit+0x39) [0xc70c37]
              /opt/couchbase/bin/indexer() [0x4631b0]
      Hanging process...
      2016-04-03T09:11:59.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21540000
      2016-04-03T09:12:02.843-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21550000
      2016-04-03T09:12:05.384-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21560000
      2016-04-03T09:12:08.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21570000
      2016-04-03T09:12:10.841-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21580000
      2016-04-03T09:12:10.907-07:00 [Info] Indexer::ReadMemstats Time Taken 22.115412ms
      2016-04-03T09:12:13.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21590000
      2016-04-03T09:12:16.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21600000
      2016-04-03T09:12:19.041-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21610000
      2016-04-03T09:12:21.787-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21620000
      2016-04-03T09:12:24.514-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21630000
      2016-04-03T09:12:27.241-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21640000
      2016-04-03T09:12:29.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21650000
      2016-04-03T09:12:32.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21660000
      2016-04-03T09:12:34.698-07:00 [Info] connected with 1 indexers
      2016-04-03T09:12:34.698-07:00 [Info] index 8397725242306989303 has 1 replicas
      2016-04-03T09:12:34.698-07:00 [Info] client load stats {}
      2016-04-03T09:12:35.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21670000
      

      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.45-20160403-2247-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.55-20160403-2244-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.70-20160403-2232-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.71-20160403-2235-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.72-20160403-2238-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-19023/d3cd8ace/172.23.100.73-20160403-2241-diag.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                PagerDuty