Couchbase Build Team added a comment - 22/Nov/19 11:56 AM Build couchbase-server-7.0.0-1065 contains libcouchbase commit 7c3a100 with commit message: CCBC-1118 : Avoid calling read callback after close

Suraj Naik (Inactive) added a comment - 05/Dec/19 3:38 AM - edited The issue was seen again, the stack trace seen is similar to the one seen here #0 __GI___libc_free (mem=0x7f046c44ee80) at malloc.c:2941 #1 0x00007f048c0b1bbb in lcbvb_destroy (conf=0x7f046c052b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/vbucket/vbucket.c:853 #2 0x00007f048c0e6d64 in decref (this=0x7f046c050ae0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/clconfig.h:546 #3 update (data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>, host=<optimized out>, this=0x7f04840f6e10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:207 #4 lcb::clconfig::cccp_update (provider=provider@entry=0x7f04840f6e10, host=<optimized out>, data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:175 #5 0x00007f048c127136 in lcb::Server::handle_nmv (this=this@entry=0x7f0484111550, resinfo=..., oldpkt=oldpkt@entry=0x7f048412cbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:151 #6 0x00007f048c129d49 in try_read (ior=0x7f048411ac28, ctx=0x7f048411abe0, this=0x7f0484111550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:396 #7 on_read (ctx=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:460 #8 0x00007f048c0c3d4c in invoke_read_cb (nb=10969, ctx=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:278 #9 E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:307 #10 0x00007f048c0a9852 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:323 #11 0x00007f048c13847e in lcb_wait (instance=0x7f04840f64c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:103 #12 0x0000000000463596 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> ( callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40 #13 0x0000000000464b01 in timer::TimerStore::GetCounter (this=this@entry=0x7f04840cf240, key=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:319 #14 0x00000000004664d0 in timer::TimerStore::SetTimer (this=0x7f04840cf240, timer=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:47 #15 0x000000000041f846 in V8Worker::SetTimer (this=this@entry=0x7f0484013700, tinfo=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1148 #16 0x000000000043dbdf in Timer::CreateTimerImpl (this=0x7f04840c22b0, args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98 #17 0x000000000043e286 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142 #18 0x00007f048d88c239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so #19 0x00007f048d88b738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so #20 0x00007f048d88aec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so #21 0x00007f048e0c48ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so #22 0x00002618c730816e in ?? () #23 0x0000028282c825a1 in ?? () #24 0x00002617c3a9d139 in ?? () #25 0x0000000900000000 in ?? () #26 0x0000028282c82681 in ?? () #27 0x0000065d78b0b0e9 in ?? () #28 0x0000364842a7fcd9 in ?? () #29 0x0000065d78b0af49 in ?? () #30 0x0000364842a094a9 in ?? () #31 0x00001f7c96f04679 in ?? () #32 0x0000065d78b0b0e9 in ?? () #33 0x0000364842a7fcd9 in ?? () #34 0x0000065d78b0af49 in ?? () #35 0x0000364842a094a9 in ?? () #36 0x00002617c3a9d139 in ?? () #37 0x0000065d78b0b0e9 in ?? () #38 0x0000065d78b0b0c9 in ?? () #39 0x0000065d78b0b099 in ?? () #40 0x0000065d78b0af49 in ?? () #41 0x000000a300000000 in ?? () #42 0x0000364842a09e91 in ?? () #43 0x0000364842a09441 in ?? () #44 0x00002617c3a82ad9 in ?? () #45 0x00007f0481e324c8 in ?? () #46 0x00007f048e034603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so #47 0x0000065d78b0aed9 in ?? () #48 0x0000065d78b0ac81 in ?? () #49 0x00001f7c96f04679 in ?? () #50 0x0000364842a09441 in ?? () #51 0x0000000000000020 in ?? () #52 0x00007f0481e32530 in ?? () #53 0x00002618c73040de in ?? () #54 0x0000000000000000 in ?? () Sergey Avseyev please look into this. This was seen on build 6.5.0-4917

Sergey Avseyev added a comment - 06/Dec/19 8:59 AM This crash is not the same as previous one. Could you open new issues? In this case watchers won't be confused thinking that this is a regression.

Matt Ingenthron added a comment - 06/Dec/19 9:02 AM hey Suraj Naik , I had a look at the stack provided here and it's completely different, other than the libc free top frame. It's not the same as CCBC-1118 . In the future, let's please not just reopen the same bug each time you see a crash unless you are sure it's the same cause. Any memory issues are likely to end up in some of the same files/stack frames. This time we'll fix up the issues.

Sergey Avseyev added a comment - 06/Dec/19 9:04 AM I moved the report to https://issues.couchbase.com/browse/CCBC-1130 . Suraj Naik , could you update other things and context how the issue has been reproduced?