Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34944

[Durability]: GSL:Precondition failure - Expects(!foundCmt) in hash_table.cc' (1) [ETA 2019/7/17]

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • No
    • KV-Engine Mad-Hatter Beta

    Description

      Build: 6.5.0-3688

      Scenario:

      1. 4 node cluster, couchbase bucket (replicas=2)
      2. Loaded 1K docs with durability=MAJORITY
      3. Killed 'beam.smp' on one of the node (172.23.105.205)
      4. Perform CRUD operation on active vbuckets not involving the killed nodes' active vBucket with same durability levels as before

      Observation:

      Memcached crashes after step #3 with prints,

      *** Error in `/opt/couchbase/bin/memcached': free(): invalid pointer: 0x00007ffa3d554000 ***{{ }}

      After restarting the server also, observing Memcached crashing continuously.

       

      Expected behavior:

      1. Memcached should not crash when other process go down unexpectedly
      2. After restarting the couchbase-server service, Memcached should not crash multiple-times

      Service 'memcached' exited with status 134. Restarting. Messages:
      2019-07-08T23:54:39.221753-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ffa34d60000+0xeae5b]
      2019-07-08T23:54:39.221761-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ffa34d60000+0xebe40]
      2019-07-08T23:54:39.221769-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ffa34d60000+0x139715]
      2019-07-08T23:54:39.221775-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ffa34d60000+0x139bdd]
      2019-07-08T23:54:39.221782-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7ffa34d60000+0x132fbf]
      2019-07-08T23:54:39.221786-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7ffa3be82000+0x8f27]
      2019-07-08T23:54:39.221792-07:00 CRITICAL /lib64/libpthread.so.0() [0x7ffa398b2000+0x7dd5]
      2019-07-08T23:54:39.221822-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7ffa394e5000+0xfdead]
      [*** LOG ERROR ***] [2019-07-08 23:54:39] [spdlog_file_logger] async log: thread pool doesn't exist anymore
      *** Error in `/opt/couchbase/bin/memcached': free(): invalid pointer: 0x00007ffa3d554000 ***
       
      After restarting the service, getting this crash prints:
      Service 'memcached' exited with status 134. Restarting. Messages:
      2019-07-08T23:58:33.440412-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0x1e79c4]
      2019-07-08T23:58:33.440419-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0x1e82f5]
      2019-07-08T23:58:33.440428-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0xeae5b]
      2019-07-08T23:58:33.440437-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0xebe40]
      2019-07-08T23:58:33.440453-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0x139715]
      2019-07-08T23:58:33.440460-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0x139bdd]
      2019-07-08T23:58:33.440466-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f5e15d60000+0x132fbf]
      2019-07-08T23:58:33.440474-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f5e1ce57000+0x8f27]
      2019-07-08T23:58:33.440484-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f5e1a887000+0x7dd5]
      2019-07-08T23:58:33.440524-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f5e1a4ba000+0xfdead]

       UI logs:

      Crash after service restarted,

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty