Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
7.1.0-1420
-
Untriaged
-
-
1
-
Unknown
-
KV-Engine-Sept-21
Description
Steps:
1. Create a 3 node cluster
2. Create buckets and 50 collections.
3. Create 40000000 items in each collection:
Read Start: 0
|
Read End: 0
|
Update Start: 0
|
Update End: 0
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 0
|
Delete End: 0
|
Create Start: 0
|
Create End: 40000000
|
Final Start: 0
|
Final End: 40000000
|
4. Update 40000000 keys to create 50 percent fragmentation:
Read Start: 0
|
Read End: 0
|
Update Start: 0
|
Update End: 40000000
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 0
|
Delete End: 0
|
Create Start: 0
|
Create End: 0
|
Final Start: 0
|
Final End: 40000000
|
5. Create another 40000000 items:
Read Start: 0
|
Read End: 0
|
Update Start: 0
|
Update End: 0
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 0
|
Delete End: 0
|
Create Start: 40000000
|
Create End: 80000000
|
Final Start: 40000000
|
Final End: 80000000
|
6. Update 40000000 keys (created in step 5) to maintain 50 percent fragmentation
Read Start: 0
|
Read End: 0
|
Update Start: 40000000
|
Update End: 80000000
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 0
|
Delete End: 0
|
Create Start: 0
|
Create End: 0
|
Final Start: 40000000
|
Final End: 80000000
|
7. Start ASYNC load:
Read Start: 0
|
Read End: 40000000
|
Update Start: 0
|
Update End: 40000000
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 40000000
|
Delete End: 80000000
|
Create Start: 80000000
|
Create End: 120000000
|
Final Start: 80000000
|
Final End: 120000000
|
8. Rebalance IN with Loading of docs in step 7
9. Rebalance OUT with Loading of docs in step 7
10. Rebalance SWAP with Loading of docs in step 7
11. Rebalance IN/OUT with Loading of docs in step 7
12. Rebalance OUT/IN with Loading of docs in step 7
13. Validate all docs mutated in step 7. All is well until here.
14. Repeat the test from step 7.
Read Start: 0
|
Read End: 40000000
|
Update Start: 0
|
Update End: 40000000
|
Expiry Start: 0
|
Expiry End: 0
|
Delete Start: 80000000
|
Delete End: 120000000
|
Create Start: 120000000
|
Create End: 160000000
|
Final Start: 120000000
|
Final End: 160000000
|
15. Repeat step 8. Rebalance failed in step 8.
{u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1633824149930L, u'shortText': u'message', u'serverTime': u'2021-10-09T17:02:29.930Z', u'text': u'Worker <0.8562.558> (for action {move,{580,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.64\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.8543.558>,\n {{{{nocatch,\n {error,\n timeout}},\n [{mc_binary,\n recv_with_data,\n 4,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 43}]},\n {mc_binary,\n quick_active_recv,\n 3,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 48}]},\n {mc_binary,\n quick_stats_loop_enter,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 100}]},\n {mc_binary,\n quick_stats,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 85}]},\n {mc_client_binary,\n get_dcp_docs_estimate,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 754}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 571}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 224}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 249}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 580,\n "replication:ns_1@172.23.110.66->ns_1@172.23.110.65:GleamBookUsers0"},\n 180000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.66\'},\n {if_rebalance,\n <0.5292.547>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\'],\n 580}},\n infinity]}}}}'}
|
