Details
-
Bug
-
Resolution: Incomplete
-
Critical
-
7.1.0
-
Triaged
-
-
1
-
Unknown
Description
Build : 7.1.0-2284
Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Scale : 3
Iteration : 8th
Rebalance to add a new Eventing node to the cluster was in progress when there was an internal server error on the orchestrator node 172.23.108.103 :
[2022-02-16T11:38:13-08:00, sequoiatools/couchbase-cli:7.1:f55f65] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.5 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
[2022-02-16T11:38:28-08:00, sequoiatools/couchbase-cli:7.1:ae76cf] rebalance -c 172.23.108.103:8091 --server-remove 172.23.99.20 -u Administrator -p password
→
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.99.20 -u Administrator -p password]
docker logs ae76cf
docker start ae76cf
*Unable to display progress bar on this os
8ERROR: Internal server error, please retry your request
[2022-02-16T11:56:27-08:00, sequoiatools/cmd:909fd5] 60
In the error.log on 172.23.108.103, the following errors can be seen around this time :
[ns_server:error,2022-02-16T11:56:21.414-08:00,ns_1@172.23.108.103:<0.4418.3660>:menelaus_util:reply_server_error_before_close:210]Server error during processing: ["web request failed",
|
{path,"/pools/default/tasks"},
|
{method,'GET'},
|
{type,exit},
|
{what,
|
{timeout,
|
{gen_server,call,[ns_doctor,get_nodes]}}},
|
{trace,
|
[{gen_server,call,2,
|
[{file,"gen_server.erl"},{line,239}]},
|
{ns_doctor,build_tasks_list,2,
|
[{file,"src/ns_doctor.erl"},{line,221}]},
|
{menelaus_web_misc,do_handle_tasks,3,
|
[{file,"src/menelaus_web_misc.erl"},
|
{line,63}]},
|
{request_tracker,request,2,
|
[{file,"src/request_tracker.erl"},
|
{line,40}]},
|
{menelaus_util,handle_request,2,
|
[{file,"src/menelaus_util.erl"},
|
{line,221}]},
|
{mochiweb_http,headers,6,
|
[{file,
|
"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
|
{line,153}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,226}]}]}]
|
The following can be seen in the debug.log on 172.23.108.103:
[error_logger:error,2022-02-16T11:56:21.268-08:00,ns_1@172.23.108.103:menelaus_web_cache<0.22155.3218>:ale_error_logger_handler:do_log:101]
|
=========================ERROR REPORT=========================
|
** Generic server menelaus_web_cache terminating
|
** Last message in was #Fun<menelaus_web_cache.2.9693504>
|
** When Server state == []
|
** Reason for termination ==
|
** {{timeout,{gen_server,call,[ns_doctor,get_tasks_version]}},
|
[{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},
|
{menelaus_web_pools,do_build_pool_info,4,
|
[{file,"src/menelaus_web_pools.erl"},{line,209}]},
|
{menelaus_web_pools,'-build_pool_info/6-fun-1-',5,
|
[{file,"src/menelaus_web_pools.erl"},{line,183}]},
|
{menelaus_web_cache,do_compute_with_expiration,3,
|
[{file,"src/menelaus_web_cache.erl"},{line,96}]},
|
{work_queue,handle_call,3,[{file,"src/work_queue.erl"},{line,51}]},
|
{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},
|
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},
|
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}
|
** Client <0.17685.3221> stacktrace
|
** [{gen,do_call,4,[{file,"gen.erl"},{line,214}]},
|
{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},
|
{menelaus_web_pools,pool_info,6,
|
[{file,"src/menelaus_web_pools.erl"},{line,106}]},
|
{menelaus_util,'-handle_streaming/2-fun-0-',3,
|
[{file,"src/menelaus_util.erl"},{line,598}]},
|
{menelaus_util,streaming_inner,5,
|
[{file,"src/menelaus_util.erl"},{line,636}]},
|
{menelaus_util,handle_streaming,5,
|
[{file,"src/menelaus_util.erl"},{line,627}]},
|
{menelaus_util,handle_request,2,
|
[{file,"src/menelaus_util.erl"},{line,221}]}]
|
|
[ns_server:debug,2022-02-16T11:56:21.276-08:00,ns_1@172.23.108.103:<0.30062.3223>:ns_memcached:verify_report_long_call:357]call verify_warmup took too long: 895706 us
|
[error_logger:error,2022-02-16T11:56:21.277-08:00,ns_1@172.23.108.103:menelaus_web_cache<0.22155.3218>:ale_error_logger_handler:do_log:101]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: work_queue:init/1
|
pid: <0.22155.3218>
|
registered_name: menelaus_web_cache
|
exception exit: {timeout,{gen_server,call,[ns_doctor,get_tasks_version]}}
|
in function gen_server:call/2 (gen_server.erl, line 239)
|
in call from menelaus_web_pools:do_build_pool_info/4 (src/menelaus_web_pools.erl, line 209)
|
in call from menelaus_web_pools:'-build_pool_info/6-fun-1-'/5 (src/menelaus_web_pools.erl, line 183)
|
in call from menelaus_web_cache:do_compute_with_expiration/3 (src/menelaus_web_cache.erl, line 96)
|
in call from work_queue:handle_call/3 (src/work_queue.erl, line 51)
|
in call from gen_server:try_handle_call/4 (gen_server.erl, line 721)
|
in call from gen_server:handle_msg/6 (gen_server.erl, line 750)
|
ancestors: [menelaus_sup,ns_server_sup,ns_server_nodes_sup,
|
<0.24031.3218>,ns_server_cluster_sup,root_sup,<0.144.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.18587.3218>]
|
dictionary: []
|
trap_exit: false
|
status: running
|
heap_size: 196650
|
stack_size: 29
|
reductions: 43911578313
|
neighbours:
|
|
[ns_server:debug,2022-02-16T11:56:21.290-08:00,ns_1@172.23.108.103:<0.2465.3660>:ns_memcached:perform_very_long_call_with_timing:902]ensure_bucket took too long: 4408585 us
|