Details
-
Bug
-
Resolution: Unresolved
-
Major
-
Elixir
-
7.5.0-3380
-
Untriaged
-
Centos 64-bit
-
0
-
Unknown
Description
STEPS TO RECREATE:
- Create a 3 node cluster {KV Nodes : [172.23.107.76 , 172.23.107.126, 172.23.107.237]}
- Create 20 magma buckets
- Create 10 collections in each bucket
- Create 300 items in each collections of every bucket
- Update all the items 300 items in each collection of every bucket
- Rebalance in one node (Node in : 172.23.120.97)
- SIGKill memcached 20 times (sleep between each sigkill is 60 to 120 seconds)
- Start Rebalancing out one node(172.23.120.97)
- Abort rebalance when rabalance progress reached 20% (by killing memcached(SIGKILL)
- Restart rebalance after killing at 20%
- Repeat above two steps (when rebalance reaches 40%, 60%, 80%)
- After node 172.23.120.97 rebalances out sleep for 60 seconds
- Start killing memcached multiple times(SIGKILL) (sleep between each sigkill is 60 to 120 seconds) while data loading is going on
- After 16th iteration of sigkill, observed below error on node (172.23.120.76)
Error
2022-12-09T22:34:48.078019-08:00 ERROR 454: Exception occurred during packet execution. Closing connection [ {"ip":"172.23.107.126","port":44854} - {"ip":"172.23.107.76","port":11206} (not authenticated) ]: Connection::isPacketAvailable(): Invalid packet header detected: (mcbp::header: magic:0x16, opcode:0x3, keylen:769, extlen:8, datatype:0x1, specific:1, bodylen:67306250, opaque:0xe3676bc9) totalRecv:0. Cookies: []
|
QE-TEST
git fetch https://review.couchbase.org/TAF refs/changes/27/182527/35 && git checkout -b change-182527 FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_vol.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=150,num_buckets=20,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=22,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=10,maxttl=10,num_indexes=1,pc=1000,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,ramQuota=5120,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=20000,key_type=RandomKey,vbuckets=64 -m rest'
|