Details
Description
Steps to reproduce
Steps to reproduce
- Created a cluster on Couchbase Enterprise Edition build 7.1.1-3175 with the following setup
- 172.23.121.136 - cbas
- 172.23.121.174 - index, kv, n1ql
- 172.23.121.194 - index, kv, n1ql
- 172.23.121.135 - cbas
- 172.23.121.198 - cbas
- Created a bucket called "bucket-0"
- Loaded 10000 items onto it
- Created dataverses, links, datasets, synonyms, indexes
- Upgraded the whole cluster to 7.6.0-2149 by failing over a node and then upgrade and add back
- Started a rebalance post upgrade - Rebalance hangs
Response of /pools/default/tasks
[{"statusId":"2d35895d90e955e571aaa80b3fe952f9","type":"rebalance","subtype":"rebalance","recommendedRefreshPeriod":0.25,"status":"running","progress":26.66666666670339,"perNode":{"ns_1@172.23.121.194":{"progress":66.66666666666666},"ns_1@172.23.121.174":{"progress":66.66666666666666},"ns_1@172.23.121.135":{"progress":6.119999999999877e-11},"ns_1@172.23.121.136":{"progress":6.119999999999877e-11},"ns_1@172.23.121.198":{"progress":6.119999999999877e-11}},"detailedProgress":{"bucket":"bucket-2","bucketNumber":1,"bucketsCount":1,"perNode":{"ns_1@172.23.121.198":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.194":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.174":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.136":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.135":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}}}},"stageInfo":{"analytics":{"totalProgress":6.119999999999875e-11,"perNodeProgress":{"ns_1@172.23.121.135":6.119999999999876e-13,"ns_1@172.23.121.136":6.119999999999876e-13,"ns_1@172.23.121.198":6.119999999999876e-13},"startTime":"2024-02-19T03:27:36.191-08:00","completedTime":false,"timeTaken":6130726},"index":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:30.139-08:00","completedTime":"2024-02-19T03:27:36.191-08:00","timeTaken":6052},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:29.420-08:00","completedTime":"2024-02-19T03:27:30.133-08:00","timeTaken":713},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:30.133-08:00","completedTime":"2024-02-19T03:27:30.139-08:00","timeTaken":6}},"rebalanceId":"3d6830874371afe45a43f9a5afaeea70","nodesInfo":{"active_nodes":["ns_1@172.23.121.136","ns_1@172.23.121.135","ns_1@172.23.121.198","ns_1@172.23.121.174","ns_1@172.23.121.194"],"keep_nodes":["ns_1@172.23.121.136","ns_1@172.23.121.135","ns_1@172.23.121.198","ns_1@172.23.121.174","ns_1@172.23.121.194"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.121.136"}] |
The progress for nodes ns_1@172.23.121.135, ns_1@172.23.121.136 and ns_1@172.23.121.198 is weirdly stuck at 6.119999999999877e-11
"ns_1@172.23.121.135": {"progress": 6.119999999999877e-11},"ns_1@172.23.121.136": {"progress": 6.119999999999877e-11},"ns_1@172.23.121.198": {"progress": 6.119999999999877e-11} |
Marking this is a regression since this was not seen in runs for RC4 - 7.6.0-2119
TAF Script to reproduce
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-analytics-vset00-00-analytics_upgrade_with_failover_from_7.1.1_with_collections/testexec.24941.ini -p GROUP=7_1_1;failover_upgrade,kv_quota_percent=70,bucket_storage=couchstore,key=test_collections,get-cbcollect-info=True,upgrade_version=7.6.0-2151,aws_access_key=xxxxx,aws_secret_key=xxxxx,sirius_url=http://172.23.120.103:4000 -t upgrade.cbas_upgrade.UpgradeTests.test_upgrade_with_failover,upgrade_chain=7.1.1,upgrade_type=failover_delta_recovery,update_nodes=kv;cbas,nodes_init=5,services_init=kv:index:n1ql-kv:index:n1ql-cbas-cbas-cbas,pre_update_no_of_dv=2,pre_update_ds_per_dv=4,pre_update_no_of_synonym=5,pre_update_no_of_index=3,replica_num=3,override_spec_params=num_buckets;num_scopes;num_collections;replicas;num_items,num_items=10000,num_buckets=3,num_scopes=5,num_collections=5,no_of_dv=10,ds_per_dv=3,no_of_synonym=10,no_of_index=5,GROUP=7_1_1;failover_upgrade,cbas_cc_node_upgrade_sequence=first' |
Job name : debian-analytics_analytics_upgrade_with_failover_from_7.1.1_with_collections
Job ref : http://qa.sc.couchbase.com/job/test_suite_executor-TAF/313214/console
Attachments
Issue Links
- duplicates
-
MB-60840 Intermittent failure during authentication handshake failed initial rebalance
- Closed