Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.0
-
Centos cluster 1
-
Untriaged
-
-
Unknown
Description
Build: 6.0.0 build 1529
Test Job: http://qa.sc.couchbase.com/job/centos-systest-launcher/1580/console
Cluster: http://172.23.108.103:8091/
We run the following steps in centos longevity
Longevity :
- Create 22 node cluster (9 KV,5 index, 2 query , 2 fts, 2 eventing, 2 cbas)
- Create 10 buckets (default bucket with Active compression)
- Create views
- Load data
- Remove kv node
- Deploy eventing functions
- Create dataset on analytics on 4 buckets
- Create index on 2 datasets
- Create 2i index
- Load more data
- Run queries on 2i
- Swap a KV node
- Run 240 queries per second on analytics
- Connect link Local
- Load more data to default bucket
- Add eventing node
- Remove eventing node
- Swap eventing node
- Disconnect link Local
- Add analytics node
- connect link Local
- Disconnect link Local
- Remove analytics node
- connect link Local
- Swap analytics node
- Kill analytics nodes
- Run views
- Create fts indexes
- Regex search on FTS
- XDCR replication
- Add rbac users
- Undeploy eventing handlers
- Load 1M doc
- Create 2i indexes
- Rebalance in index
- Rebalance out index
- Swap index node
- Rebalance in 2 index nodes
- Rebalance out 2 index nodes
- Rebalance out 1 KV
- Rebalance in 1 KV
- Failover -> Full recovery index node
- Failover -> Rebalance out index node
- Add index node
- Redeploy eventing handlers
- Run Tpcc
- Update Doc
- Add a kv node , failover kv node -> rebalance
- swap hard failover -> Add 1 KV remove 2 KV as soft and hard failover
- Multinode autofailover -> failover 3 KV nodes and rebalance
observed the following fails with the below steps
- Adding a data node back
[2018-08-18T02:50:04-07:00, sequoiatools/pillowfight:d91da7] -U couchbase://172.23.108.103/default?select_bucket=true -I 3000 -B 300 -t 4 -c 100 -P password
|
[2018-08-18T02:54:55-07:00, sequoiatools/couchbase-cli:262814] server-add -c 172.23.108.103:8091 --server-add 172.23.108.104:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2018-08-18T02:55:18-07:00, sequoiatools/couchbase-cli:4cc1a4] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
|
|
|
docker logs 4cc1a4
|
docker start 4cc1a4
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[user:error,2018-08-18T03:08:17.682-07:00,ns_1@172.23.108.103:<0.9637.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {noproc,
|
{gen_server,call,
|
[{'janitor_agent-WAREHOUSE',
|
'ns_1@172.23.96.56'},
|
{get_dcp_docs_estimate,134,
|
['ns_1@172.23.108.104']},
|
infinity]}}
|
- Removing data node
[2018-08-18T03:38:18-07:00, sequoiatools/couchbase-cli:8c9dd6] rebalance -c 172.23.108.103:8091 --server-remove 172.23.108.104:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.108.104:8091 -u Administrator -p password]
|
|
|
docker logs 8c9dd6
|
docker start 8c9dd6
|
|
|
*Unable to display progress bar on this os
|
[user:error,2018-08-18T03:55:49.367-07:00,ns_1@172.23.108.103:<0.9637.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.28031.797>,
|
{{error,{badrpc,nodedown}},
|
{gen_server,call,
|
[{'janitor_agent-DISTRICT',
|
'ns_1@172.23.96.56'},
|
{if_rebalance,<0.1608.796>,
|
{update_vbucket_state,135,active,
|
undefined,undefined}},
|
infinity]}}}}}
|
- swap of analytics node
[2018-08-18T04:24:55-07:00, sequoiatools/couchbase-cli:97ad55] server-add -c 172.23.108.103:8091 --server-add 172.23.96.148:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics
|
[2018-08-18T04:25:20-07:00, sequoiatools/couchbase-cli:d8b44d] rebalance -c 172.23.108.103:8091 --server-remove 172.23.99.25 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.99.25 -u Administrator -p password]
|
|
|
docker logs d8b44d
|
docker start d8b44d
|
|
|
*Unable to display progress bar on this os
|
[user:error,2018-08-18T04:35:38.929-07:00,ns_1@172.23.108.103:<0.9637.0>:ns_orchestrator:do_log_rebalance_completion:1117]Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.26855.819>,
|
{noproc,
|
{gen_server,call,
|
[{'janitor_agent-DISTRICT',
|
'ns_1@172.23.96.56'},
|
{if_rebalance,<0.18658.819>,
|
{inhibit_view_compaction,<0.18658.819>}},
|
infinity]}}}}}
|
Note: With alice we are able to complete first cycle first time. Hence its not regression