Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55456

[CDC] ns_server allows historyRetentionBytes/Seconds to be set to values that memcached cannot handle

    XMLWordPrintable

Details

    Description

      Build: 7.2.0-5144

      Steps to repro:

      1. Create a three node cluster
      2. Create a magma bucket with bucket_history_retention_seconds=86400,bucket_history_retention_bytes=102400000000000000000000
      3. Observed node turned in to amber/pending state, and remains in the same state

      Below crash report found on node 172.23.107.240
      Logs:

      [error_logger:error,2023-02-06T23:57:57.386-08:00,ns_1@172.23.107.240:janitor_agent-default<0.29109.39>:ale_error_logger_handler:do_log:101]
      =========================CRASH REPORT=========================
        crasher:
          initial call: janitor_agent:init/1
          pid: <0.29109.39>
          registered_name: 'janitor_agent-default'
          exception exit: {{bad_return_value,
                            {stop,
                             {error,
                              {bucket_create_error,
                               {memcached_error,not_stored,undefined}}}}},
                           {gen_server,call,
                            [{'ns_memcached-default','ns_1@172.23.107.240'},
                             {raw_stats,<<"dcp">>,#Fun<dcp_replicator.4.11572224>,
                              []},
                             300000]}}
            in function  gen_server:call/3 (gen_server.erl, line 247)
            in call from ns_memcached:do_call/4 (src/ns_memcached.erl, line 1446)
            in call from dcp_replicator:get_connections/1 (src/dcp_replicator.erl, line 319)
            in call from dcp_sup:nuke/1 (src/dcp_sup.erl, line 118)
            in call from janitor_agent:init/1 (src/janitor_agent.erl, line 556)
          ancestors: ['janitor_agent_sup-default','single_bucket_kv_sup-default',
                        ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
                        ns_server_nodes_sup,<0.271.0>,ns_server_cluster_sup,
                        root_sup,<0.145.0>]
          message_queue_len: 0
          messages: []
          links: [<0.29200.39>]
          dictionary: []
          trap_exit: false
          status: running
          heap_size: 2586
      

      QE-TEST:

      git fetch https://review.couchbase.org/TAF refs/changes/72/186272/2 && git checkout -b change-186272 FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol_3.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_magma_failures_with_CDC,nodes_init=3,skip_cleanup=True,num_items=1000000,doc_size=1024,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=5,ops_rate=20000,key_type=SimpleKey,vbuckets=1024,replicas=2,doc_ops=update:delete,test_itr=50,bucket_history_retention_seconds=86400,bucket_history_retention_bytes=102400000000000000000000,standard_buckets=1,magma_buckets=1,num_scopes=1,induce_failures=True -m rest'
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty