Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55507

CDC: memcached throws "Ckpt::queueDirty() status:failure:duplicate item when vbstate is non-active:2"

    XMLWordPrintable

Details

    Description

       

      Build: 7.2.0-5150

      Scenario:

      • 4 node cluster
      • Magma bucket with replica=3, retention_bytes=1000000000,seconds=86400
      • Perform cont. dedupe mutations and trigger rebalance_in for 2 nodes in parallel 

        +----------------+----------+-----------------------+---------------+--------------+
        		 
        | Nodes          | Services | Version               | CPU           | Status       |
        		 
        +----------------+----------+-----------------------+---------------+--------------+
        		 
        | 172.23.105.212 | kv       | 7.2.0-5150-enterprise | 96.6901059859 | Cluster node |
        		 
        | 172.23.105.155 | kv       | 7.2.0-5150-enterprise | 94.4061862745 | Cluster node |
        		 
        | 172.23.105.213 | kv       | 7.2.0-5150-enterprise | 94.4647571316 | Cluster node |
        		 
        | 172.23.105.211 | kv       | 7.2.0-5150-enterprise | 98.2583104595 | Cluster node |
        		 
        | 172.23.105.244 | None     |                       |               | <--- IN ---  |
        		 
        | 172.23.105.245 | None     |                       |               | <--- IN ---  |
        		 
        +----------------+----------+-----------------------+---------------+--------------+

      Observation:

      Memcached crashed on node .212 resulting in rebalance failure.

      Rebalance exited with reason {{badmatch,
      		 
      {error,
      		 
      {failed_nodes,['ns_1@172.23.105.212']}}},
      		 
      [{ns_janitor,cleanup_apply_config_body,4,
      		 
      [{file,"src/ns_janitor.erl"},{line,295}]},
      		 
      {ns_janitor,'-cleanup_apply_config/4-fun-0-',
      		 
      4,
      		 
      [{file,"src/ns_janitor.erl"},{line,215}]},
      		 
      {async,'-async_init/4-fun-1-',3,
      		 
      [{file,"src/async.erl"},{line,191}]}]}.
      		 
      Rebalance Operation Id = 2d98d03e8da97bd0d1df294eacf3474e
      		 
       
      		 
      Node :: ns_1@172.23.105.212
      		 
      Service 'memcached' exited with status 134. Restarting. Messages:
      		 
      #16 /opt/couchbase/bin/memcached() [0x400000+0x331384]
      		 
      #17 /opt/couchbase/bin/memcached() [0x400000+0x334052]
      		 
      #18 /opt/couchbase/bin/memcached() [0x400000+0x2fdfb8]
      		 
      #19 /opt/couchbase/bin/memcached() [0x400000+0x4eb5c8]
      		 
      #20 /opt/couchbase/bin/memcached() [0x400000+0x4ebe85]
      		 
      #21 /opt/couchbase/bin/memcached() [0x400000+0x4dcf72]
      		 
      #22 /opt/couchbase/bin/memcached() [0x400000+0x4e3517]
      		 
      #23 /opt/couchbase/bin/memcached() [0x400000+0x4e6a45]
      		 
      #24 /opt/couchbase/bin/memcached() [0x400000+0x6a3279]
      		 
      #25 /opt/couchbase/bin/memcached() [0x400000+0x69c93a]
      		 
      #26 /opt/couchbase/bin/memcached() [0x400000+0x6a462e]
      		 
      #27 /opt/couchbase/bin/memcached() [0x400000+0x801c10]
      		 
      #28 /opt/couchbase/bin/memcached() [0x400000+0x7ec6aa]
      		 
      #29 /opt/couchbase/bin/memcached() [0x400000+0x804bc9]
      		 
      #30 /opt/couchbase/bin/memcached() [0x400000+0x69c634]
      		 
      #31 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f894672c000+0xcdd40]
      		 
      #32 /lib64/libpthread.so.0() [0x7f89485fa000+0x7dd5]
      		 
      #33 /lib64/libc.so.6(clone+0x6d) [0x7f8945e45000+0xfdead] 

      Memcached gdb backtrace: memcahced_bt.txt

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-55582 CDC: Graceful failover+delta recovery rebalance failed with error 'sync_shutdown_many_i_am_trapping_exits'

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-55610 [CDC] Rebalance exited with wait_seqno_persisted_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-55507
          # Subject Branch Project Status CR V
          191909,2 Merge commit neo/d39926349 into master master kv_engine Status: NEW +2 -1

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty