Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55609

[Magma-CDC] Memcached crashed in fail_fast_throw ( message=0xcac1f8 "GSL: Precondition failure: 'lsmStats.HistoryLogicalDiskSize >= historyLogicalGarbageSize'

    XMLWordPrintable

Details

    Description

      Steps To Recreate:

      1. Create a 3 node cluster
      2. Create a magma buckets (history params not set) (Vbuckets=1024, replicas=2)
      3. Create fifty collections(total collections including default are fifty one)
      4. Create 100K docs in each of the collections
      5. Update 100K docs in each of the collections while update is going on disable replicas and rebalance
      6. Update bucket params with history_retention_seconds=86400, history_retention_bytes=96000000000
      7. Start upserting the docs again
      8. Observed Memcached crashes in fail_fast_throw ( message=0xcac1f8 "GSL: Precondition failure: 'lsmStats.HistoryLogicalDiskSize >= historyLogicalGarbageSize'

      Below Core Dump is on node 172.23.107.239

      BackTrace:

      (gdb) bt full
      #0  0x00007f5edcb58387 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007f5edcb59a78 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x00007f5edd4a363c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
              terminating = false
              t = <optimized out>
      #3  0x0000000000b4273b in backtrace_terminate_handler() ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
      No locals.
      #4  0x00007f5edd4ae8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      No locals.
      #5  0x00007f5edd4ae961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      No locals.
      #6  0x00007f5edd4aebf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f5ed4004ba0, tinfo=tinfo@entry=0xc54a08 <typeinfo for gsl::fail_fast>,
          dest=dest@entry=0x598ea0 <gsl::fail_fast::~fail_fast()>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
              globals = <optimized out>
              header = 0x7f5ed4004b20
      #7  0x000000000050f05c in fail_fast_throw (
          message=0xcac1f8 "GSL: Precondition failure: 'lsmStats.HistoryLogicalDiskSize >= historyLogicalGarbageSize' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:139")
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/gsl-lite.hpp:1769
      No locals.
      #8  magma::LSMTree::GetStats (this=0x7f5e24887010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:139
              dataLevel = <optimized out>
              logicalGarbageSize = <optimized out>
              historyLogicalGarbageSize = <optimized out>
              lsmStats = {UID = {static npos = 18446744073709551615,
                  _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
                    _M_p = 0x7f5edb5f5c40 <Address 0x7f5edb5f5c40 out of bounds>}, _M_string_length = 52, {
                    _M_local_buf = "4\000\000\000\000\000\000\000=\232}\336^\177\000", _M_allocated_capacity = 52}}, NItemsPut = 15068, NItemsDel = 0,
                NItemsGet = 0, BytesIncoming = 9226086, BytesOutgoing = 0, NWriteBytes = 19359790, NReadBytes = 6875895, NSyncs = 316, NReadIOs = 3305,
                NWriteIOs = 1433, NItemsGarbageCollected = 4344, NItemsMerged = 4344, NItemsTransformed = 0, NItemsDropped = 0, NTablesCreated = 1112,
                NTablesDeleted = 538, NTableFiles = 34, NCompacts = 179, NTTLCompacts = 0, NFileCountCompacts = 21, NNonL0LocalCompacts = 0,
      

      QE-TEST:

      git fetch https://review.couchbase.org/TAF refs/changes/62/186462/22 && git checkout -b change-186462 FETCH_HEAD
       
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.cdc_steady_state.SteadyStateTests.test_history_retention_for_n_update_iterations,nodes_init=3,skip_cleanup=True,num_items=100000,doc_size=1024,batch_size=100,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=51,ops_rate=20000,key_type=RandomKey,vbuckets=1024,replicas=2,test_itr=50,bucket_history_retention_seconds=0,bucket_history_retention_bytes=0,standard_buckets=1,magma_buckets=1,num_scopes=1 -m rest'
      

      Attachments

        For Gerrit Dashboard: MB-55609
        # Subject Branch Project Status CR V

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty