Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.1.0-2543
Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Iteration : 1st
Scale : 3
Rebalance operation to add 2 KV nodes in the cluster failed
[2022-04-02T19:30:22-07:00, sequoiatools/couchbase-cli:7.1:699a08] server-add -c 172.23.108.103:8091 --server-add https://172.23.97.239 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2022-04-02T19:31:03-07:00, sequoiatools/couchbase-cli:7.1:f6d291] server-add -c 172.23.108.103:8091 --server-add https://172.23.105.107 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2022-04-02T19:31:18-07:00, sequoiatools/couchbase-cli:7.1:c17acb] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
|
|
|
docker logs c17acb
|
docker start c17acb
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2022-04-02T19:35:57-07:00, sequoiatools/cmd:7cef2c] 60
|
The error seen in the error.log on 172.23.108.103 is the following :
[ns_server:error,2022-04-02T19:35:48.399-07:00,ns_1@172.23.108.103:<0.25183.723>:ns_single_vbucket_mover:spawn_and_wait:79]Got unexpected exit signal {'EXIT',<0.8590.723>,
|
{unexpected_exit,
|
{'EXIT',<0.24108.723>,
|
{{error,
|
{badrpc,
|
{'EXIT',
|
{{noproc,
|
{gen_server,call,
|
[<16189.2936.0>,
|
{monitor_partition_update,103,
|
#Ref<16189.930313933.3236429829.52818>,
|
<16189.32343.102>},
|
infinity]}},
|
{gen_server,call,
|
['capi_set_view_manager-default',
|
{wait_index_updated,103},
|
infinity]}}}}},
|
{gen_server,call,
|
[{'janitor_agent-default','ns_1@172.23.104.5'},
|
{if_rebalance,<0.8590.723>,
|
{wait_index_updated,103}},
|
infinity]}}}}}
|
[user:error,2022-04-02T19:35:48.471-07:00,ns_1@172.23.108.103:<0.25066.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.24108.723>,
|
{{error,
|
{badrpc,
|
{'EXIT',
|
{{noproc,
|
{gen_server,call,
|
[<16189.2936.0>,
|
{monitor_partition_update,103,
|
#Ref<16189.930313933.3236429829.52818>,
|
<16189.32343.102>},
|
infinity]}},
|
{gen_server,call,
|
['capi_set_view_manager-default',
|
{wait_index_updated,103},
|
infinity]}}}}},
|
{gen_server,call,
|
[{'janitor_agent-default',
|
'ns_1@172.23.104.5'},
|
{if_rebalance,<0.8590.723>,
|
{wait_index_updated,103}},
|
infinity]}}}}}.
|
Rebalance Operation Id = 1d376ca8c7b62bb71785ae1c698098f3
|
On 172.23.104.5, the following error is seen in the couchdb.log file at the same time.
[couchdb:info,2022-04-02T19:35:46.609-07:00,couchdb_ns_1@cb.local:<0.2936.0>:couch_log:info:30]Set view `default`, main (prod) group `_design/scale`, signature `3187a7d527597477d663a16e5e99ebe5`, terminating with reason: <ud>{dcp_client_died,<0.2944.0>,noproc}</ud>
|
[couchdb:info,2022-04-02T19:35:46.609-07:00,couchdb_ns_1@cb.local:<0.2946.0>:couch_log:info:30]Set view `default`, replica (prod) group `_design/scale`, signature `3187a7d527597477d663a16e5e99ebe5`, terminating with reason: <ud>shutdown</ud>
|
[couchdb:error,2022-04-02T19:35:48.358-07:00,couchdb_ns_1@cb.local:<0.31163.102>:couch_log:error:33]Uncaught error in HTTP request: {exit,
|
{{dcp_client_died,<0.2944.0>,noproc},
|
{gen_server,call,
|
[<0.2936.0>,
|
{set_view_group_req,update_after,true,
|
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
|
16,17,18,19,20,27,28,29,30,31,32,33,34,
|
35,36,37,38,39,40,41,42,43,44,45,46,47,
|
48,49,50,51,52,53,54,55,59,60,61,62,63,
|
64,65,66,67,68,69,70,71,72,73,74,75,76,
|
77,78,79,80,81,82,83,84,85,86,92,262,
|
434,435,436,462,463,521,529,539,540,606,
|
689,690,691,733,734,749],
|
false,main,prod},
|
infinity]}}}
|
|
|
Stacktrace: <ud>[{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},
|
{couch_set_view_group,request_group,3,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},
|
{line,160}]},
|
{couch_set_view,get_group,4,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view.erl"},
|
{line,94}]},
|
{couch_set_view,get_map_view,4,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view.erl"},
|
{line,739}]},
|
{couch_view_merger,simple_set_view_query,3,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_index_merger/src/couch_view_merger.erl"},
|
{line,1237}]},
|
{couch_httpd,handle_request,7,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_httpd.erl"},
|
{line,228}]},
|
{mochiweb_http,headers,6,
|
[{file,"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
|
{line,153}]},
|
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]</ud>
|
[couchdb:info,2022-04-02T19:35:48.359-07:00,couchdb_ns_1@cb.local:<0.31163.102>:couch_log:info:30]<ud>"172.23.97.122"</ud> -- POST <ud>/default/_design/scale/_view/padd</ud> 500
|
Marking it as a regression since this issue hasn't been seen until RC3.