Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45925

CV Intermittent failure: test dcp replica stream one collection due to ThrowExceptionUnderflowPolicy

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Intermittent CV failure observed due to engine_testapp

      [2021-04-26T14:43:05.229Z]  27/411 Test #266: ep_testsuite_dcp.ephemeral.comp_passive .....................................................................Child aborted***Exception:  96.89 sec
      [2021-04-26T14:43:05.229Z] Running [0000/0096]: test dcp vbtakeover stat no stream...(88 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0001/0096]: test open consumer...(48 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0002/0096]: test dcp consumer flow control none...(45 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0003/0096]: test dcp consumer flow control static...(46 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0004/0096]: test dcp consumer flow control dynamic...(68 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0005/0096]: test dcp consumer flow control aggressive...(1114 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0006/0096]: test open producer...(55 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0007/0096]: test dcp noop...(67 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0008/0096]: test dcp noop failure...(64 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0009/0096]: test dcp consumer noop...(66 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0010/0096]: test dcp replica stream backfill...(85 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0011/0096]: test dcp replica stream backfill and warmup (MB-34173)...(0 ms) SKIPPED
      [2021-04-26T14:43:05.229Z] Running [0012/0096]: test dcp replica stream in-memory...(91 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0013/0096]: test dcp replica stream all...WARNING: Logging before InitGoogleLogging() is written to STDERR
      [2021-04-26T14:43:05.229Z] W0426 14:41:28.440670  9999 HazptrDomain.h:671] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object
      [2021-04-26T14:43:05.229Z] (5159 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0014/0096]: test dcp replica stream all with collections enabled stream...(1191 ms) OK
      [2021-04-26T14:43:05.229Z] Running [0015/0096]: test dcp replica stream one collection with mutations just from disk...Uncaught std::exception. what():ThrowExceptionUnderflowPolicy current:497 arg:521
      [2021-04-26T14:43:05.229Z] (21230 ms) DIED
      

      Also as part exception throw we might not have freed the mock cookie correctly:

      [2021-04-26T14:43:05.230Z] =================================================================
      [2021-04-26T14:43:05.230Z] ==8777==ERROR: LeakSanitizer: detected memory leaks
      [2021-04-26T14:43:05.230Z] 
      [2021-04-26T14:43:05.230Z] Direct leak of 336 byte(s) in 1 object(s) allocated from:
      [2021-04-26T14:43:05.230Z]     #0 0x1175acd in operator new(unsigned long) (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/kv_engine/ep_testsuite_dcp+0x1175acd)
      [2021-04-26T14:43:05.230Z]     #1 0x1364b5b in create_mock_cookie(EngineIface*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/mock_cookie.cc:34:12
      [2021-04-26T14:43:05.230Z]     #2 0x1294b12 in test_dcp_replica_stream_one_collection_on_disk(EngineIface*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/engines/ep/tests/ep_testsuite_dcp.cc:5766:34
      [2021-04-26T14:43:05.230Z]     #3 0x1210bf1 in try_run_test(std::function<test_result ()>) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:280:16
      [2021-04-26T14:43:05.230Z]     #4 0x12136e6 in execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:407:19
      [2021-04-26T14:43:05.230Z]     #5 0x1211e7b in main /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:660:37
      [2021-04-26T14:43:05.230Z]     #6 0x7f2abc683bf6 in __libc_start_main /build/glibc-S9d2JN/glibc-2.27/csu/../csu/libc-start.c:310
      [2021-04-26T14:43:05.230Z] 
      [2021-04-26T14:43:05.230Z] Direct leak of 336 byte(s) in 1 object(s) allocated from:
      [2021-04-26T14:43:05.230Z]     #0 0x1175acd in operator new(unsigned long) (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/kv_engine/ep_testsuite_dcp+0x1175acd)
      [2021-04-26T14:43:05.230Z]     #1 0x1364b5b in create_mock_cookie(EngineIface*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/mock_cookie.cc:34:12
      [2021-04-26T14:43:05.230Z]     #2 0x12940ab in test_dcp_replica_stream_one_collection_on_disk(EngineIface*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/engines/ep/tests/ep_testsuite_dcp.cc:5650:33
      [2021-04-26T14:43:05.230Z]     #3 0x1210bf1 in try_run_test(std::function<test_result ()>) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:280:16
      [2021-04-26T14:43:05.230Z]     #4 0x12136e6 in execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:407:19
      [2021-04-26T14:43:05.230Z]     #5 0x1211e7b in main /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/build/../kv_engine/programs/engine_testapp/engine_testapp.cc:660:37
      [2021-04-26T14:43:05.230Z]     #6 0x7f2abc683bf6 in __libc_start_main /build/glibc-S9d2JN/glibc-2.27/csu/../csu/libc-start.c:310
      [2021-04-26T14:43:05.230Z] 
      [2021-04-26T14:43:05.230Z] SUMMARY: AddressSanitizer: 672 byte(s) leaked in 2 allocation(s).
      [2021-04-26T14:43:05.230Z] /usr/bin/timeout: the monitored command dumped core
      

      Seen during cv job:http://cv.jenkins.couchbase.com/job/kv_engine.ASan-UBSan/job/master/17984/

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jwalker Jim Walker added a comment -

            Duping with a newer MB that has a tiny bit more information

            jwalker Jim Walker added a comment - Duping with a newer MB that has a tiny bit more information MB-46365

            People

              owend Daniel Owen
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty