Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59472

Memcached crashed at cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.2.3
    • couchbase-bucket
    • 7.2.3-6705

    Description

      Steps to Recreate:
      Steps:

      1. Create a 6 node cluster (3 kv nodes, 2 index , 1 index:n1ql)
      2. Create 1 couchstore, 1 ephemeral , 1 magma buckets)
      3. Set oso_dcp_backfill
      4. Create few indexes and run few queries
      5. SIG Kill memcached on all the kv nodes one by one

      This issue looks similar to MB-58893

      Core was found on node 172.23.106.100(1fb73785-a4e5-4bec-b6d8a5a0-73ad28b9.dmp)

      bt full:
      https://gist.github.com/ankushsharma29/fb9ee32f6fb40330e7226742e0d1bef4

      BackTrace:

      (gdb) bt
      #0  0x00000000007d5a74 in std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed, this=<error reading variable: Cannot access memory at address 0x0>)
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/atomic_base.h:426
      #1  cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:235
      #2  cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::operator unsigned long (
          this=<error reading variable: Cannot access memory at address 0x0>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:231
      #3  Collections::VB::ManifestEntry::getItemCount (this=<error reading variable: Cannot access memory at address 0x0>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_entry.h:110
      #4  Collections::VB::StatsReadHandle::getItemCount (this=this@entry=0x7f55df7eb450)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_handles.cc:39
      #5  0x00000000008e2e7d in operator() (__closure=<optimized out>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1915
      #6  ActiveStream::tryAndScheduleOSOBackfill (this=0x7f5556486090, producer=..., vb=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1916
      #7  0x00000000008e572c in ActiveStream::scheduleBackfill_UNLOCKED (this=<optimized out>, producer=..., reschedule=<optimized out>)
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:1324
      #8  0x00000000008e6452 in ActiveStream::transitionState (this=0x7f5556486090, newState=ActiveStream::StreamState::Backfilling)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:2238
      #9  0x00000000008f1b1d in ActiveStream::setActive (this=0x7f5556486090)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.h:138
      #10 0x000000000080c5a3 in DcpProducer::streamRequest(unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=this@entry=0x7f5560ae4280, flags=flags@entry=16, opaque=opaque@entry=2768372744, vbucket=vbucket@entry=...,
          start_seqno=start_seqno@entry=0, end_seqno=end_seqno@entry=18446744073709551615, vbucket_uuid=<optimized out>, snap_start_seqno=<optimized out>,
          snap_end_seqno=<optimized out>, rollback_seqno=<optimized out>, callback=..., json=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:577
      #11 0x00000000006bacd6 in EventuallyPersistentEngine::stream_req(CookieIface const&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=<optimized out>, cookie=..., flags=16, opaque=2768372744, vbucket=..., startSeqno=0,
          endSeqno=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7f55df7ec028, callback=..., json=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1310
       
      #12 0x000000000059e65b in dcpStreamReq(Cookie&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (cookie=..., flags=flags@entry=16, opaque=2768372744, vbucket=vbucket@entry=..., startSeqno=startSeqno@entry=0,
          endSeqno=endSeqno@entry=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7f55df7ec028, callback=..., json=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:724
      #13 0x00000000006411d8 in dcp_stream_req_executor (cookie=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/dcp_stream_req_executor.cc:59
      #14 0x000000000061361b in Cookie::doExecute (this=0x7f55eccd0080)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/mcbp/protocol/header.h:110
      #15 0x00000000006136b0 in Cookie::execute (this=this@entry=0x7f55eccd0080, useStartTime=useStartTime@entry=true)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:168
      #16 0x00000000005f4bd3 in Connection::executeCommandPipeline (this=0x7f55f0c4d580)
      --Type <RET> for more, q to quit, c to continue without paging--
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:577
      #17 0x00000000005f4e82 in Connection::executeCommandsCallback (this=0x7f55f0c4d580)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:801
      #18 0x00000000005f550b in Connection::rw_callback (bev=<optimized out>, ctx=<optimized out>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:950
      #19 0x00007f55f31efd7e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7f55ecdde000)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208
      #20 0x00007f55f31f8cf1 in event_process_active_single_queue (base=base@entry=0x7f55f0c66c00, max_to_process=max_to_process@entry=2147483647,
          endtime=endtime@entry=0x0, activeq=<optimized out>) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726
      #21 0x00007f55f31f953f in event_process_active (base=0x7f55f0c66c00)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789
      #22 event_base_loop (base=0x7f55f0c66c00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012
      #23 0x0000000000c2f3f7 in folly::EventBase::loopBody (this=this@entry=0x7f55f0c3ac40, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397
      #24 0x0000000000c2f8c6 in folly::EventBase::loop (this=this@entry=0x7f55f0c3ac40)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315
      #25 0x0000000000c31286 in folly::EventBase::loopForever (this=this@entry=0x7f55f0c3ac40)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538
      #26 0x00000000005c16f9 in worker_libevent (arg=0x7f55f0c3ac30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257
      #27 0x0000000000bb3ed9 in CouchbaseThread::run (this=0x7f55f0c220a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51
      #28 platform_thread_wrap (arg=0x7f55f0c220a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64
      #29 0x00007f55f3d28ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #30 0x00007f55f1f8aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-collections-vset00-00-dcp_oso_backfill_7.2_P1_majority/testexec.11104.ini GROUP=OSO_DCP_BACKFILL_P0,get-cbcollect-info=True,infra_log_level=info,log_level=info,bucket_storage=couchstore,upgrade_version=7.2.3-6705,sirius_url=http://172.23.120.103:4000 -t bucket_collections.oso_backfill_test.KvOsoBackfillTests.test_dcp_backfill_config,nodes_init=6,replicas=1,num_items=5000,load_collections_exponentially=True,services_init=kv-kv-kv-index-index-index:n1ql,bucket_spec=multi_bucket.buckets_with_similar_hierarchy,oso_dcp_backfill=auto,simulate_error=kill_memcached,kv_quota_percent=100,index_quota_percent=100,err_itrs=20,GROUP=OSO_DCP_BACKFILL_P0'
      

      Job: http://qa.sc.couchbase.com/job/test_suite_executor-TAF/282453/consoleFull

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty