Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
6.6.5
-
Centos 7 64 bit; CB EE 6.6.5 build 10060
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Summary:
During in rebalance in-out, observed that the response from /tasks had rebalance progress > 100%
[{"statusId":"9a9c9cd76eb8f3a4e36042663e725a03","type":"rebalance","subtype":"rebalance","recommendedRefreshPeriod":0.25,"status":"running","progress":115,"perNode":{"ns_1@172.23.120.183":{"progress":50},"ns_1@172.23.120.224":{"progress":147.5},"ns_1@172.23.120.219":{"progress":147.5}},"detailedProgress":{"bucket":"GleamBookUsers","bucketNumber":1,"bucketsCount":1,"perNode":{"ns_1@172.23.120.224":{"ingoing":{"docsTotal":13,"docsTransferred":13,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.120.219":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":6,"docsTransferred":6,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.120.183":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":7,"docsTransferred":7,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}}}},"stageInfo":{"search":{"totalProgress":195,"perNodeProgress":{"ns_1@172.23.120.224":1.95,"ns_1@172.23.120.219":1.95},"startTime":"2021-12-14T01:31:07.625-08:00","completedTime":false,"timeTaken":1849303},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.120.183":1,"ns_1@172.23.120.224":1,"ns_1@172.23.120.219":1},"startTime":"2021-12-14T01:30:09.393-08:00","completedTime":"2021-12-14T01:31:07.617-08:00","timeTaken":58224,"details":{"GleamBookUsers":{"compactionInfo":{"perNode":{"ns_1@172.23.120.219":{"averageTime":33.875},"ns_1@172.23.120.224":{"averageTime":2}}},"vbucketLevelInfo":{"move":{"averageTime":578.2958984375,"totalCount":1024,"remainingCount":0},"backfill":{"averageTime":19.9736328125},"takeover":{"averageTime":12.8984375},"persistence":{"averageTime":6.31875}},"replicationInfo":{"ns_1@172.23.120.183":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":7,"outDocsLeft":0},"ns_1@172.23.120.224":{"inDocsTotal":13,"inDocsLeft":0,"outDocsTotal":0,"outDocsLeft":0},"ns_1@172.23.120.219":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":6,"outDocsLeft":0}},"startTime":"2021-12-14T01:30:11.214-08:00","completedTime":"2021-12-14T01:31:07.614-08:00","timeTaken":56401}}},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.120.183":1},"startTime":"2021-12-14T01:31:07.625-08:00","completedTime":"2021-12-14T01:31:07.625-08:00","timeTaken":0}},"rebalanceId":"5bf084ac71120942fd783620c9842101","nodesInfo":{"active_nodes":["ns_1@172.23.120.224","ns_1@172.23.120.183","ns_1@172.23.120.219"],"keep_nodes":["ns_1@172.23.120.224","ns_1@172.23.120.183"],"eject_nodes":["ns_1@172.23.120.219"],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.120.183"}]
|
Steps
1. Create a 2 node cluster
2021-12-14 01:03:09,335 | test | INFO | pool-1-thread-5 | [table_view:display:72] Rebalance Overview
|
+----------------+------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+------------+--------------+
|
| 172.23.120.183 | kv, n1ql | Cluster node |
|
| 172.23.123.168 | ['kv,fts'] | <--- IN --- |
|
+----------------+------------+--------------+
|
2. Upload x509 certs and enforce tls
3. Create 20 fts indexes
4. Create a bucket with 10 documents
2021-12-14 01:04:22,369 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
|
+----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
|
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
|
| GleamBookUsers | membase | 1 | none | 0 | 10 | 10485760000 | 99307168 | 12786084 |
|
+----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
|
5. Rebalance-in with parallel doc loading
2021-12-14 01:04:25,279 | test | INFO | pool-1-thread-5 | [table_view:display:72] Rebalance Overview
|
+----------------+------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+------------+--------------+
|
| 172.23.123.168 | fts, kv | Cluster node |
|
| 172.23.120.183 | kv, n1ql | Cluster node |
|
| 172.23.120.219 | ['kv,fts'] | <--- IN --- |
|
+----------------+------------+--------------+
|
6. Rebalance-out with parallel doc loading
2021-12-14 01:14:38,407 | test | INFO | pool-1-thread-10 | [table_view:display:72] Rebalance Overview
|
+----------------+-----------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+-----------------+--------------+
|
| 172.23.123.168 | [u'fts', u'kv'] | --- OUT ---> |
|
| 172.23.120.183 | kv, n1ql | Cluster node |
|
| 172.23.120.219 | fts, kv | Cluster node |
|
+----------------+-----------------+--------------+
|
7. Rebalance in-out with parallel doc loading
2021-12-14 01:30:09,375 | test | INFO | pool-1-thread-3 | [table_view:display:72] Rebalance Overview
|
+----------------+-----------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+-----------------+--------------+
|
| 172.23.120.183 | kv, n1ql | Cluster node |
|
| 172.23.120.219 | [u'fts', u'kv'] | --- OUT ---> |
|
| 172.23.120.224 | ['kv,fts'] | <--- IN --- |
|
+----------------+-----------------+--------------+
|
This rebalance's progress went above 100(ranged from 100% to 116% before finishing)
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-