Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1537

core dump encountered during graceful failover while performing multi-get operations

    XMLWordPrintable

Details

    • Bug
    • Status: New
    • Major
    • Resolution: Unresolved
    • 3.0.6
    • None
    • library
    • None
    • application backends in Amadeus using libcouchbase 3.0.6 towards Couchbase server 6.0.4
    • 1

    Description

      Hi Sergey, Couchbase team,

      Amadeus Availability Process Engine application is encountered some core dump on a few backends (2-5 instances over 2200 in total) while doing kernel patching on a node, basically starting with a graceful failover of the node. It appears currenlty with libcouchbase 3.0.6 and towards Couchbase server 6.0.4. Troughput for all backend is approxymately 1M req/s with elapsed time median to 800us.

      Have you already seen this kind of issues ?

      From the stack trace below, it appears that during the graceful failover, libcouchbase may rarely core while performing a multiget operation and attempting to update vbucket map with new cluster topology: 

      #1  0x00007f19f5a8064a in __GI_abort () at abort.c:89
      save_stage = 2
      act = {__sigaction_handler = {sa_handler = 0x541dcb0, sa_sigaction = 0x541dcb0}, sa_mask = {__val = {146498816, 139749460973088, 139749116680179, 0, 206158430256, 3087494, 874396, 139749460973056, 139749208745152, 167560320, 139749470174842, 139749460973120, 1, 139749460973072, 139749470174935, 139749460973072}}, sa_flags = -175702233, sa_restorer = 0x7f19f4fa9640}
      sigs = {__val = {32, 0 <repeats 15 times>}}
      #2  0x00007f19f5870665 in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) () from /ama/obe/PRD/ape/odc/pack/OSP/lib/libtcmalloc.so.4
      No symbol table info available.
      #3  0x00007f19f586780a in (anonymous namespace)::InvalidFree(void*) [clone .lto_priv.197] () from /ama/obe/PRD/ape/odc/pack/OSP/lib/libtcmalloc.so.4
      No symbol table info available.
      #4  0x00007f19e5f0c43b in lcbvb_destroy () from /ama/obe/PRD/ape/odc/pack/MDW/lib/libkvclient.so.16.1.0
      No symbol table info available.
      #5  0x00007f19e5f60276 in lcb_update_vbconfig () from /ama/obe/PRD/ape/odc/pack/MDW/lib/libkvclient.so.16.1.0
      No symbol table info available.
      #6  0x00007f19e5f3a6b4 in lcb::io::Timer<lcb::Bootstrap, &lcb::Bootstrap::timer_dispatch>::cb(void*) () from /ama/obe/PRD/ape/odc/pack/MDW/lib/libkvclient.so.16.1.0
      No symbol table info available.
      #7  0x00007f19e5f1e4ec in timer_callback () from /ama/obe/PRD/ape/odc/pack/MDW/lib/libkvclient.so.16.1.0
      No symbol table info available.
      #8  0x00007f19e073bf09 in event_base_loop () from /ama/obe/PRD/ape/odc/pack/OSP/lib/libevent-2.0.so.5
      No symbol table info available.
      #9  0x00007f19e5fad322 in lcb_wait () from /ama/obe/PRD/ape/odc/pack/MDW/lib/libkvclient.so.16.1.0

      Regards,

      Alexis Deltour.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Alexis Deltour Alexis Deltour added a comment - attached to this support ticket https://support.couchbase.com/hc/en-us/requests/44385

            Hi Alexis Deltour, I think this issue has been fixed already. Do you have any chance to upgrade to recent version of libcouchbase and try to reproduce?

            avsej Sergey Avseyev added a comment - Hi Alexis Deltour , I think this issue has been fixed already. Do you have any chance to upgrade to recent version of libcouchbase and try to reproduce?

            Hi Sergey Avseyev we are working on a story to upgrade to libcouchbase 3.2.4, i will let you know

            Regards, Alexis.

            Alexis Deltour Alexis Deltour added a comment - Hi Sergey Avseyev we are working on a story to upgrade to libcouchbase 3.2.4, i will let you know Regards, Alexis.

            People

              avsej Sergey Avseyev
              Alexis Deltour Alexis Deltour
              Sergey Avseyev
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty