Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49839

[NexusKVStore] Memcached aborted in NexusKVStoreSecondaryGetAllKeysCallback::callback(DiskDocKey const&) ()

    XMLWordPrintable

Details

    Description

      Steps to repro:

      1. Create a 3 node cluster(172.23.121.139, 172.23.121.140, 172.23.121.141(bucket_ram_quota = 2056 MB/node), replicas=1)
      2. Create 100 non default collections in default scope(collection count is 101(including default collection)
      3. Start Load 300k docs in each of 100 non default collections
      4. Trigger full compaction
      5. Create two scopes(scope1 and scope2) and 5 collections in each of these scopes
      6. While data loading is going on remove node 172.23.121.141 and change replica to 0
      7. Triggered Rebalance
      8. Drop one scope (scope2) and two collections in scope1
      9. Rebalance was successful
      10. Again change replica to 1 and rebalance
      11. Recreate a new scope (scope2) and two collections in scope1
      12. Drop the scope2
      13. Observed Memcached aborted in NexusKVStoreSecondaryGetAllKeysCallback::callback(DiskDocKey const&) ()
        Core found on node : 172.23.121.140

      BackTrace:

      (gdb) bt full
      #0  0x00007fdc1420d387 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007fdc1420ea78 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x0000000000497316 in cb::handleError(spdlog::logger&, spdlog::level::level_enum, std::basic_string_view<char, std::char_traits<char> >, cb::ErrorHandlingMethod) [clone .cold] ()
      No symbol table info available.
      #3  0x0000000000902769 in NexusKVStoreSecondaryGetAllKeysCallback::callback(DiskDocKey const&) ()
      No symbol table info available.
      #4  0x000000000086e8e1 in MagmaKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const::{lambda(magma::Slice&, magma::Slice&, magma::Slice&)#1}::operator()(magma::Slice&, magma::Slice&, magma::Slice&) const [clone .constprop.0] ()
      No symbol table info available.
      #5  0x000000000088f216 in std::_Function_handler<void (magma::Slice&, magma::Slice&, magma::Slice&), MagmaMemoryTrackingProxy::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long)::{lambda(magma::Slice&, magma::Slice&, magma::Slice&)#1}>::_M_invoke(std::_Any_data const&, magma::Slice&, magma::Slice&, magma::Slice&) ()
      No symbol table info available.
      #6  0x000000000098557e in magma::KVStore::IterateByKey(std::shared_ptr<magma::KVStoreSnapshot>&, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
      No symbol table info available.
      #7  0x00000000009284c5 in magma::Magma::GetRange(magma::Magma::Snapshot&, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
      No symbol table info available.
      #8  0x000000000092ab62 in magma::Magma::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
      No symbol table info available.
      #9  0x00000000008902c2 in MagmaMemoryTrackingProxy::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
      No symbol table info available.
      #10 0x0000000000875867 in MagmaKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const
          ()
      No symbol table info available.
      #11 0x00000000008ed3f2 in NexusKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const
          ()
      No symbol table info available.
      #12 0x00000000006d8e23 in FetchAllKeysTask::run() ()
      

      QE-TEST:

      git fetch "https://review.couchbase.org/TAF" refs/changes/88/166488/2 && git checkout FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/qe_r.ini -p bucket_storage=couchstore,rerun=false,bucket_eviction_policy=fullEviction,init_loading=False -t storage.magma.magma_rebalance.MagmaRebalance.test_data_load_collections_with_rebalance_in,num_items=300000,doc_size=256,nodes_init=3,nodes_in=1,standard_buckets=1,magma_buckets=0,bucket_storage=couchstore,data_load_stage=during,sdk_timeout=60,vbuckets=1024,key_size=12,replicas=1,infra_log_level=debug,log_level=debug,skip_cleaup=True,randomize_value=True,bucket_eviction_policy=fullEviction,infra_log_level=debug,log_level=debug,init_loading=False,fragmentation=50,skip_cleanup=True,autoCompactionDefined=true,iterations=1,enable_dp=True,num_collections=101,num_scopes=1,bucket_ram_quota=2056,skip_cleanup=True,sdk_client_pool=False,ops_rate=30000,doc_ops=create,create_perc=0,delete_perc=50,update_perc=50,num_collections_to_drop=50,get-cbcollect-info=True -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1847 contains kv_engine commit 5bdf3ba with commit message:
            MB-49839: Return early in CouchKVStore::getAllKeys

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1847 contains kv_engine commit 5bdf3ba with commit message: MB-49839 : Return early in CouchKVStore::getAllKeys

            Build couchbase-server-7.1.0-1867 contains kv_engine commit 89d2d69 with commit message:
            MB-49839: Remove unused variable

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1867 contains kv_engine commit 89d2d69 with commit message: MB-49839 : Remove unused variable

            Build couchbase-server-7.1.0-1876 contains kv_engine commit 6119cd1 with commit message:
            MB-49839: Adapt to GetRange API change

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1876 contains kv_engine commit 6119cd1 with commit message: MB-49839 : Adapt to GetRange API change

            Build couchbase-server-7.1.0-1876 contains magma commit f755764 with commit message:
            MB-49839 magma: Add bool return to GetRange callback function

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1876 contains magma commit f755764 with commit message: MB-49839 magma: Add bool return to GetRange callback function

            Build couchbase-server-7.1.0-1877 contains kv_engine commit cb93fa7 with commit message:
            MB-49839: [Magma] Cancel getAllKeys in same way that CouchKVStore does

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1877 contains kv_engine commit cb93fa7 with commit message: MB-49839 : [Magma] Cancel getAllKeys in same way that CouchKVStore does

            People

              ankush.sharma Ankush Sharma
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty