Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1118

Libcouchbase crash observed in eventing

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.10.5
    • 2.10.6
    • None
    • None
    • 1

    Description

      Build: 6.5.0-4783, last working build was 4744

      Attachments

        1. temp.core
          272 kB
        2. test_8.zip
          75.33 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-1065 contains libcouchbase commit 7c3a100 with commit message:
            CCBC-1118: Avoid calling read callback after close

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1065 contains libcouchbase commit 7c3a100 with commit message: CCBC-1118 : Avoid calling read callback after close
            suraj.naik Suraj Naik (Inactive) added a comment - - edited

            The issue was seen again, the stack trace seen is similar to the one seen here

            #0  __GI___libc_free (mem=0x7f046c44ee80) at malloc.c:2941
            #1  0x00007f048c0b1bbb in lcbvb_destroy (conf=0x7f046c052b10)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/vbucket/vbucket.c:853
            #2  0x00007f048c0e6d64 in decref (this=0x7f046c050ae0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/clconfig.h:546
            #3  update (data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>, host=<optimized out>, this=0x7f04840f6e10)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:207
            #4  lcb::clconfig::cccp_update (provider=provider@entry=0x7f04840f6e10, host=<optimized out>, 
                data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:175
            #5  0x00007f048c127136 in lcb::Server::handle_nmv (this=this@entry=0x7f0484111550, resinfo=..., oldpkt=oldpkt@entry=0x7f048412cbd0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:151
            #6  0x00007f048c129d49 in try_read (ior=0x7f048411ac28, ctx=0x7f048411abe0, this=0x7f0484111550)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:396
            #7  on_read (ctx=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:460
            #8  0x00007f048c0c3d4c in invoke_read_cb (nb=10969, ctx=0x7f048411abe0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:278
            #9  E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f048411abe0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:307
            #10 0x00007f048c0a9852 in run_loop (io=<optimized out>, is_tick=<optimized out>)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:323
            #11 0x00007f048c13847e in lcb_wait (instance=0x7f04840f64c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:103
            #12 0x0000000000463596 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (
                callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, 
                max_retry_count=5)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #13 0x0000000000464b01 in timer::TimerStore::GetCounter (this=this@entry=0x7f04840cf240, key=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:319
            #14 0x00000000004664d0 in timer::TimerStore::SetTimer (this=0x7f04840cf240, timer=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:47
            #15 0x000000000041f846 in V8Worker::SetTimer (this=this@entry=0x7f0484013700, tinfo=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1148
            #16 0x000000000043dbdf in Timer::CreateTimerImpl (this=0x7f04840c22b0, args=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98
            #17 0x000000000043e286 in CreateTimer (args=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142
            #18 0x00007f048d88c239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so
            #19 0x00007f048d88b738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so
            #20 0x00007f048d88aec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) ()
               from /opt/couchbase/lib/libv8.so
            #21 0x00007f048e0c48ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so
            #22 0x00002618c730816e in ?? ()
            #23 0x0000028282c825a1 in ?? ()
            #24 0x00002617c3a9d139 in ?? ()
            #25 0x0000000900000000 in ?? ()
            #26 0x0000028282c82681 in ?? ()
            #27 0x0000065d78b0b0e9 in ?? ()
            #28 0x0000364842a7fcd9 in ?? ()
            #29 0x0000065d78b0af49 in ?? ()
            #30 0x0000364842a094a9 in ?? ()
            #31 0x00001f7c96f04679 in ?? ()
            #32 0x0000065d78b0b0e9 in ?? ()
            #33 0x0000364842a7fcd9 in ?? ()
            #34 0x0000065d78b0af49 in ?? ()
            #35 0x0000364842a094a9 in ?? ()
            #36 0x00002617c3a9d139 in ?? ()
            #37 0x0000065d78b0b0e9 in ?? ()
            #38 0x0000065d78b0b0c9 in ?? ()
            #39 0x0000065d78b0b099 in ?? ()
            #40 0x0000065d78b0af49 in ?? ()
            #41 0x000000a300000000 in ?? ()
            #42 0x0000364842a09e91 in ?? ()
            #43 0x0000364842a09441 in ?? ()
            #44 0x00002617c3a82ad9 in ?? ()
            #45 0x00007f0481e324c8 in ?? ()
            #46 0x00007f048e034603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so
            #47 0x0000065d78b0aed9 in ?? ()
            #48 0x0000065d78b0ac81 in ?? ()
            #49 0x00001f7c96f04679 in ?? ()
            #50 0x0000364842a09441 in ?? ()
            #51 0x0000000000000020 in ?? ()
            #52 0x00007f0481e32530 in ?? ()
            #53 0x00002618c73040de in ?? ()
            #54 0x0000000000000000 in ?? ()
            

            Sergey Avseyev please look into this. This was seen on build 6.5.0-4917

            suraj.naik Suraj Naik (Inactive) added a comment - - edited The issue was seen again, the stack trace seen is similar to the one seen here #0 __GI___libc_free (mem=0x7f046c44ee80) at malloc.c:2941 #1 0x00007f048c0b1bbb in lcbvb_destroy (conf=0x7f046c052b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/vbucket/vbucket.c:853 #2 0x00007f048c0e6d64 in decref (this=0x7f046c050ae0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/clconfig.h:546 #3 update (data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>, host=<optimized out>, this=0x7f04840f6e10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:207 #4 lcb::clconfig::cccp_update (provider=provider@entry=0x7f04840f6e10, host=<optimized out>, data=0x7f046c022db0 <Address 0x7f046c022db0 out of bounds>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/bucketconfig/bc_cccp.cc:175 #5 0x00007f048c127136 in lcb::Server::handle_nmv (this=this@entry=0x7f0484111550, resinfo=..., oldpkt=oldpkt@entry=0x7f048412cbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:151 #6 0x00007f048c129d49 in try_read (ior=0x7f048411ac28, ctx=0x7f048411abe0, this=0x7f0484111550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:396 #7 on_read (ctx=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:460 #8 0x00007f048c0c3d4c in invoke_read_cb (nb=10969, ctx=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:278 #9 E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f048411abe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/ctx.c:307 #10 0x00007f048c0a9852 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:323 #11 0x00007f048c13847e in lcb_wait (instance=0x7f04840f64c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:103 #12 0x0000000000463596 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> ( callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40 #13 0x0000000000464b01 in timer::TimerStore::GetCounter (this=this@entry=0x7f04840cf240, key=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:319 #14 0x00000000004664d0 in timer::TimerStore::SetTimer (this=0x7f04840cf240, timer=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:47 #15 0x000000000041f846 in V8Worker::SetTimer (this=this@entry=0x7f0484013700, tinfo=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1148 #16 0x000000000043dbdf in Timer::CreateTimerImpl (this=0x7f04840c22b0, args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98 #17 0x000000000043e286 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142 #18 0x00007f048d88c239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so #19 0x00007f048d88b738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so #20 0x00007f048d88aec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so #21 0x00007f048e0c48ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so #22 0x00002618c730816e in ?? () #23 0x0000028282c825a1 in ?? () #24 0x00002617c3a9d139 in ?? () #25 0x0000000900000000 in ?? () #26 0x0000028282c82681 in ?? () #27 0x0000065d78b0b0e9 in ?? () #28 0x0000364842a7fcd9 in ?? () #29 0x0000065d78b0af49 in ?? () #30 0x0000364842a094a9 in ?? () #31 0x00001f7c96f04679 in ?? () #32 0x0000065d78b0b0e9 in ?? () #33 0x0000364842a7fcd9 in ?? () #34 0x0000065d78b0af49 in ?? () #35 0x0000364842a094a9 in ?? () #36 0x00002617c3a9d139 in ?? () #37 0x0000065d78b0b0e9 in ?? () #38 0x0000065d78b0b0c9 in ?? () #39 0x0000065d78b0b099 in ?? () #40 0x0000065d78b0af49 in ?? () #41 0x000000a300000000 in ?? () #42 0x0000364842a09e91 in ?? () #43 0x0000364842a09441 in ?? () #44 0x00002617c3a82ad9 in ?? () #45 0x00007f0481e324c8 in ?? () #46 0x00007f048e034603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so #47 0x0000065d78b0aed9 in ?? () #48 0x0000065d78b0ac81 in ?? () #49 0x00001f7c96f04679 in ?? () #50 0x0000364842a09441 in ?? () #51 0x0000000000000020 in ?? () #52 0x00007f0481e32530 in ?? () #53 0x00002618c73040de in ?? () #54 0x0000000000000000 in ?? () Sergey Avseyev please look into this. This was seen on build 6.5.0-4917

            This crash is not the same as previous one. Could you open new issues? In this case watchers won't be confused thinking that this is a regression.

            avsej Sergey Avseyev added a comment - This crash is not the same as previous one. Could you open new issues? In this case watchers won't be confused thinking that this is a regression.

            hey Suraj Naik, I had a look at the stack provided here and it's completely different, other than the libc free top frame. It's not the same as CCBC-1118.

            In the future, let's please not just reopen the same bug each time you see a crash unless you are sure it's the same cause. Any memory issues are likely to end up in some of the same files/stack frames. This time we'll fix up the issues.

            ingenthr Matt Ingenthron added a comment - hey Suraj Naik , I had a look at the stack provided here and it's completely different, other than the libc free top frame. It's not the same as CCBC-1118 . In the future, let's please not just reopen the same bug each time you see a crash unless you are sure it's the same cause. Any memory issues are likely to end up in some of the same files/stack frames. This time we'll fix up the issues.

            I moved the report to https://issues.couchbase.com/browse/CCBC-1130.

            Suraj Naik, could you update other things and context how the issue has been reproduced?

            avsej Sergey Avseyev added a comment - I moved the report to https://issues.couchbase.com/browse/CCBC-1130 . Suraj Naik , could you update other things and context how the issue has been reproduced?

            People

              avsej Sergey Avseyev
              suraj.naik Suraj Naik (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty