[System Test] :- " FATAL ERROR: 2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007] libcouchbase experienced an unrecoverable error and terminates the program" seen in eventing logs

./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 1 -repeat 0 -log_level 0 -version 7.2.5-7595 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

/opt/couchbase/var/lib/couchbase/logs/eventing.log.4.gz:2024-04-14T13:24:33.730-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:3457464] FATAL ERROR:

/opt/couchbase/var/lib/couchbase/logs/eventing.log.5.gz:2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007] FATAL ERROR:

/opt/couchbase/var/lib/couchbase/logs/eventing.log.5.gz:2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019] FATAL ERROR:

/opt/couchbase/var/lib/couchbase/logs/eventing.log.5.gz:2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033] FATAL ERROR:

/opt/couchbase/var/lib/couchbase/logs/eventing.log.5.gz:2024-04-14T09:21:43.747-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019] FATAL ERROR:

/opt/couchbase/var/lib/couchbase/logs/eventing.log.5.gz:2024-04-14T09:45:17.625-07:00 [Info] eventing-consumer [worker_sbm_0_0:/tmp/127.0.0.1:8091_0_602275864.sock:1529507] FATAL ERROR:

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007] FATAL ERROR:

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     libcouchbase experienced an unrecoverable error and terminates the program

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     to avoid undefined behavior.

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     The program should have generated a "corefile" which may used

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     to gather more information about the problem.

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     If your system doesn't create "corefiles" I can tell you that the

2024-04-14T09:21:43.636-07:00 [Info] eventing-consumer [worker_timers_0_0:/tmp/127.0.0.1:8091_0_1120527895.sock:1524007]     assertion failed in /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mc/mcreq-flush-inl.h at line 63

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019] FATAL ERROR:

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     libcouchbase experienced an unrecoverable error and terminates the program

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     to avoid undefined behavior.

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     The program should have generated a "corefile" which may used

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     to gather more information about the problem.

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     If your system doesn't create "corefiles" I can tell you that the

2024-04-14T09:21:43.648-07:00 [Info] eventing-consumer [worker_timers_0_1:/tmp/127.0.0.1:8091_1_1120527895.sock:1524019]     assertion failed in /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mc/mcreq-flush-inl.h at line 63

2024-04-14T09:21:43.677-07:00 [Error] GetVBmap() failed: vbmap: Invalid KV node no -1 for vb 172

2024-04-14T09:21:43.677-07:00 [Info] No Location exist with id: 8

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033] FATAL ERROR:

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     libcouchbase experienced an unrecoverable error and terminates the program

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     to avoid undefined behavior.

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     The program should have generated a "corefile" which may used

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     to gather more information about the problem.

2024-04-14T09:21:43.685-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     If your system doesn't create "corefiles" I can tell you that the

2024-04-14T09:21:43.686-07:00 [Info] eventing-consumer [worker_timers_0_2:/tmp/127.0.0.1:8091_2_1120527895.sock:1524033]     assertion failed in /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mc/mcreq-flush-inl.h at line 63

(gdb) bt

#0  0x00007f8e1aca80fc in __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:70

#1  0x00007f8e1aca81cf in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78

#2  0x00007f8e1ac5a472 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26

#3  0x00007f8e1ac444b2 in __GI_abort () at ./stdlib/abort.c:79

#4  0x00007f8e1b516cd3 in mcreq__pktflush_callback (p=p@entry=0x7f8df00fd330, hint=hint@entry=177, arg=arg@entry=0x7f8dfbff9ef0)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mc/mcreq-flush-inl.h:63

#5  0x00007f8e1b4c9f1f in netbuf_end_flush2 (mgr=mgr@entry=0x7f8e0c0f2a80, nflushed=nflushed@entry=177, callback=callback@entry=0x7f8e1b516b50 <mcreq__pktflush_callback(void*, nb_SIZE, void*)>, 

    lloff=lloff@entry=8, arg=arg@entry=0x7f8dfbff9ef0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/netbuf/netbuf.c:659

#6  0x00007f8e1b51bd0b in mcreq_flush_done_ex (now=<optimized out>, expected=177, nflushed=177, pl=0x7f8e0c0f2a40)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mc/mcreq-flush-inl.h:94

#7  on_flush_done (ctx=<optimized out>, expected=177, actual=177) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:90

#8  0x00007f8e1b4d17f3 in E_put_ex (nb=177, niov=<optimized out>, iov=0x7f8dfbff9f80, ctx=0x7f8e0c21be70)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.cc:502

#9  lcbio_ctx_put_ex (ctx=ctx@entry=0x7f8e0c21be70, iov=iov@entry=0x7f8dfbff9f80, niov=<optimized out>, nb=177)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.cc:576

#10 0x00007f8e1b516af7 in on_flush_ready (ctx=0x7f8e0c21be70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:74

#11 0x00007f8e1b4d04fa in E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f8e0c21be70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.cc:293

#12 0x00007f8e1b4b7d55 in run_loop (io=0x7f8e0c08db30, is_tick=0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:309

#13 0x00007f8e1b57e1e7 in lcb_wait (instance=instance@entry=0x7f8e0c08d610, flags=flags@entry=LCB_WAIT_DEFAULT)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:108

#14 0x00000000004a1dea in LcbSet (instance=instance@entry=0x7f8e0c08d610, cmd=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/src/lcb_util.cc:307

#15 0x000000000049b642 in RetryLcbCommand<lcb_CMDSTORE_, std::pair<lcb_STATUS, Result> (&)(lcb_st*, lcb_CMDSTORE_&)> (instance=0x7f8e0c08d610, cmd=..., max_retry_count=max_retry_count@entry=0, 

    max_retry_secs=max_retry_secs@entry=58, callable=@0x4a1c40: {std::pair<lcb_STATUS, Result> (lcb_st *, lcb_CMDSTORE_ &)} 0x4a1c40 <LcbSet(lcb_st*, lcb_CMDSTORE_&)>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/../features/include/lcb_utils.h:94

#16 0x000000000049761e in LcbInstance::Execute<lcb_CMDSTORE_*, std::pair<lcb_STATUS, Result> (&)(lcb_st*, lcb_CMDSTORE_&)> (max_retry_secs=58, max_retry_count=0, callable=<optimized out>, 

    cmd=@0x7f8dfbffa888: 0x7f8df00074b0, this=<optimized out>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/../features/include/lcbinstance.h:42

#17 timer::TimerStore::Upsert (this=this@entry=0x7f8e0c089ec0, key=..., value=..., max_retry_count=max_retry_count@entry=0, max_retry_secs=max_retry_secs@entry=58)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:732

#18 0x000000000049a73b in timer::TimerStore::SetTimer (this=0x7f8e0c089ec0, timer=..., max_retry_count=0, max_retry_secs=58)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:138

#19 0x0000000000442031 in V8Worker::SetTimer (this=this@entry=0x7f8e0c016850, tinfo=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1662

#20 0x00000000004674f3 in Timer::CreateTimerImpl (this=0x7f8e0c0826e0, args=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:103

#21 0x00000000004680db in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:224

#22 0x00007f8e1d53b61d in Builtins_CallApiCallbackGeneric () from /opt/couchbase/bin/../lib/libv8.so

#23 0x00007f8dfbffb220 in ?? ()

#24 0x00007f8dfbffb258 in ?? ()

#25 0x0000000000000004 in ?? ()

#26 0x0000000000000070 in ?? ()

#27 0x0000000500000000 in ?? ()

#28 0x00007f8dfbffb1c0 in ?? ()

#29 0x0000000000000024 in ?? ()

#30 0x00007f8dfbffb2e0 in ?? ()

#31 0x00007f8e1d53989e in Builtins_InterpreterEntryTrampoline () from /opt/couchbase/bin/../lib/libv8.so

#32 0x00001fd44c4c12c9 in ?? ()

#33 0x0000000400000000 in ?? ()

#34 0x00001fd44c4c1ae9 in ?? ()

#35 0x00001fd44c4c1a39 in ?? ()

#36 0x00007f8e0c01d550 in ?? ()

#37 0x000014beb0e80061 in ?? ()