Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.0
-
component cluster
-
Untriaged
-
Unknown
-
KV-Engine Mad-Hatter GA
Description
Build : 6.5.0-4471
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st
Step:
[2019-10-04T09:44:15-07:00, sequoiatools/couchbase-cli:6.5:6a4039] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data |
[2019-10-04T09:44:38-07:00, sequoiatools/couchbase-cli:6.5:b4a767] rebalance -c 172.23.108.103:8091 -u Administrator -p password |
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password] |
|
docker logs b4a767
|
docker start b4a767
|
|
*Unable to display progress bar on this os |
JERROR: Rebalance failed. See logs for detailed reason. You can try again. |
[2019-10-04T09:55:07-07:00, sequoiatools/cmd:5a4414] 60 |
[2019-10-04T09:56:16-07:00, sequoiatools/cmd:348ec6] 300 |
Error: (diag.log in 172.23.108.103)
2019-10-04T09:54:52.724-07:00, compaction_daemon:0:warning:message(ns_1@172.23.97.119) - Compactor for view `mapreduce_view/default/_design/scale/replica` (pid [{type, |
view},
|
{important,
|
true}, |
{name,
|
<<"mapreduce_view/default/_design/scale/replica">>}, |
{fa,
|
{#Fun<compaction_daemon.23.62703019>, |
[<<"default">>, |
<<"_design/scale">>, |
mapreduce_view,
|
replica,
|
{config,
|
{30, |
undefined},
|
{30, |
undefined},
|
undefined,
|
false, |
false, |
{daemon_config,
|
30, |
131072, |
20971520}}, |
true, |
{[{type,
|
bucket}]}]}}]) terminated unexpectedly: {updater_died,
|
{updater_error,
|
timeout}}
|
2019-10-04T09:54:52.930-07:00, ns_rebalancer:2:info:message(ns_1@172.23.108.103) - Bad replicators after rebalance: |
Missing = [{'ns_1@172.23.108.103','ns_1@172.23.97.119',113}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',114}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',115}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',116}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',117}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',118}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',119}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',120}, |
{'ns_1@172.23.108.103','ns_1@172.23.97.119',121}] |
Extras = []
|
2019-10-04T09:54:52.933-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason bad_replicas. |
Rebalance Operation Id = 0b67a052f94895e5418ff35de8a74354
|
-------------------------------
|
|
|
per_node_processes('ns_1@172.23.108.103') = |
{<0.32074.207>, |
[{backtrace,
|
[<<"Program counter: 0x00007f550eb437c8 (gen_server:loop/7 + 288)">>, |
<<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>, |
<<"0x00007f538ccd6128 Return addr 0x00007f55553e6ff0 (proc_lib:init_p_do_apply/3 + 72)">>, |
<<"y(0) []">>,<<"y(1) infinity">>, |
<<"y(2) dcp_proxy">>, |
<<"(3) {state,#Port<0.236464>,{producer,\"replication:ns_1@172.23.106.100->ns_1@172.23.108.103:NEW_ORDER\",'ns_1@172.23.1">>, |
<<"y(4) <0.32074.207>">>,<<"y(5) <0.7577.208>">>,<<>>, |
<<"0x00007f538ccd6160 Return addr 0x0000000000942608 (<terminate process normally>)">>, |
<<"y(0) []">>, |
<<"y(1) Catch 0x00007f55553e7010 (proc_lib:init_p_do_apply/3 + 104)">>, |
<<>>]},
|
{messages,[]},
|
{dictionary,
|
[{'$initial_call',{dcp_proxy,init,1}}, |
{'$ancestors', |
['dcp_replicator-NEW_ORDER-ns_1@172.23.106.100', |
'dcp_sup-NEW_ORDER','single_bucket_kv_sup-NEW_ORDER', |
ns_bucket_sup,ns_bucket_worker_sup,ns_server_sup,
|
ns_server_nodes_sup,<0.205.0>,ns_server_cluster_sup,root_sup, |
<0.117.0>]}]}, |
{registered_name,[]},
|
{status,waiting},
|
{initial_call,{proc_lib,init_p,5}}, |
{error_handler,error_handler},
|
{garbage_collection,
|
[{max_heap_size,#{error_logger => true,kill => true,size => 0}}, |
{min_bin_vheap_size,46422}, |
{min_heap_size,233}, |
{fullsweep_after,512}, |
{minor_gcs,98}]}, |
{garbage_collection_info,
|
[{old_heap_block_size,6772}, |
{heap_block_size,1598}, |
{mbuf_size,0}, |
{recent_size,24}, |
{stack_size,10}, |
{old_heap_size,2339}, |
{heap_size,445}, |
|
logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.156.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.164.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.104.87.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.56.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.96.95.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1570208407/collectinfo-2019-10-04T170009-ns_1%40172.23.99.25.zip
Attachments
Issue Links
- is duplicated by
-
MB-36584 [System test]: received unexpected commit for key
-
- Closed
-