Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
Untriaged
-
Centos 64-bit
-
-
0
-
No
Description
Build: 7.2.0-5144
Steps to repro:
- Create a three node cluster
- Create a magma bucket with bucket_history_retention_seconds=86400,bucket_history_retention_bytes=102400000000000000000000
- Observed node turned in to amber/pending state, and remains in the same state
Below crash report found on node 172.23.107.240
Logs:
[error_logger:error,2023-02-06T23:57:57.386-08:00,ns_1@172.23.107.240:janitor_agent-default<0.29109.39>:ale_error_logger_handler:do_log:101]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: janitor_agent:init/1
|
pid: <0.29109.39>
|
registered_name: 'janitor_agent-default'
|
exception exit: {{bad_return_value,
|
{stop,
|
{error,
|
{bucket_create_error,
|
{memcached_error,not_stored,undefined}}}}},
|
{gen_server,call,
|
[{'ns_memcached-default','ns_1@172.23.107.240'},
|
{raw_stats,<<"dcp">>,#Fun<dcp_replicator.4.11572224>,
|
[]},
|
300000]}}
|
in function gen_server:call/3 (gen_server.erl, line 247)
|
in call from ns_memcached:do_call/4 (src/ns_memcached.erl, line 1446)
|
in call from dcp_replicator:get_connections/1 (src/dcp_replicator.erl, line 319)
|
in call from dcp_sup:nuke/1 (src/dcp_sup.erl, line 118)
|
in call from janitor_agent:init/1 (src/janitor_agent.erl, line 556)
|
ancestors: ['janitor_agent_sup-default','single_bucket_kv_sup-default',
|
ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
|
ns_server_nodes_sup,<0.271.0>,ns_server_cluster_sup,
|
root_sup,<0.145.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.29200.39>]
|
dictionary: []
|
trap_exit: false
|
status: running
|
heap_size: 2586
|
QE-TEST:
git fetch https://review.couchbase.org/TAF refs/changes/72/186272/2 && git checkout -b change-186272 FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol_3.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_magma_failures_with_CDC,nodes_init=3,skip_cleanup=True,num_items=1000000,doc_size=1024,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=5,ops_rate=20000,key_type=SimpleKey,vbuckets=1024,replicas=2,doc_ops=update:delete,test_itr=50,bucket_history_retention_seconds=86400,bucket_history_retention_bytes=102400000000000000000000,standard_buckets=1,magma_buckets=1,num_scopes=1,induce_failures=True -m rest'
|