Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50081

Rebalance progress seems to go above 100%

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • 6.6.5
    • 7.1.0
    • fts
    • Centos 7 64 bit; CB EE 6.6.5 build 10060
    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      Summary:
      During in rebalance in-out, observed that the response from /tasks had rebalance progress > 100%

      [{"statusId":"9a9c9cd76eb8f3a4e36042663e725a03","type":"rebalance","subtype":"rebalance","recommendedRefreshPeriod":0.25,"status":"running","progress":115,"perNode":{"ns_1@172.23.120.183":{"progress":50},"ns_1@172.23.120.224":{"progress":147.5},"ns_1@172.23.120.219":{"progress":147.5}},"detailedProgress":{"bucket":"GleamBookUsers","bucketNumber":1,"bucketsCount":1,"perNode":{"ns_1@172.23.120.224":{"ingoing":{"docsTotal":13,"docsTransferred":13,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.120.219":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":6,"docsTransferred":6,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.120.183":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":7,"docsTransferred":7,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}}}},"stageInfo":{"search":{"totalProgress":195,"perNodeProgress":{"ns_1@172.23.120.224":1.95,"ns_1@172.23.120.219":1.95},"startTime":"2021-12-14T01:31:07.625-08:00","completedTime":false,"timeTaken":1849303},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.120.183":1,"ns_1@172.23.120.224":1,"ns_1@172.23.120.219":1},"startTime":"2021-12-14T01:30:09.393-08:00","completedTime":"2021-12-14T01:31:07.617-08:00","timeTaken":58224,"details":{"GleamBookUsers":{"compactionInfo":{"perNode":{"ns_1@172.23.120.219":{"averageTime":33.875},"ns_1@172.23.120.224":{"averageTime":2}}},"vbucketLevelInfo":{"move":{"averageTime":578.2958984375,"totalCount":1024,"remainingCount":0},"backfill":{"averageTime":19.9736328125},"takeover":{"averageTime":12.8984375},"persistence":{"averageTime":6.31875}},"replicationInfo":{"ns_1@172.23.120.183":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":7,"outDocsLeft":0},"ns_1@172.23.120.224":{"inDocsTotal":13,"inDocsLeft":0,"outDocsTotal":0,"outDocsLeft":0},"ns_1@172.23.120.219":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":6,"outDocsLeft":0}},"startTime":"2021-12-14T01:30:11.214-08:00","completedTime":"2021-12-14T01:31:07.614-08:00","timeTaken":56401}}},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.120.183":1},"startTime":"2021-12-14T01:31:07.625-08:00","completedTime":"2021-12-14T01:31:07.625-08:00","timeTaken":0}},"rebalanceId":"5bf084ac71120942fd783620c9842101","nodesInfo":{"active_nodes":["ns_1@172.23.120.224","ns_1@172.23.120.183","ns_1@172.23.120.219"],"keep_nodes":["ns_1@172.23.120.224","ns_1@172.23.120.183"],"eject_nodes":["ns_1@172.23.120.219"],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.120.183"}]

      Steps 
      1. Create a 2 node cluster

      2021-12-14 01:03:09,335 | test  | INFO    | pool-1-thread-5 | [table_view:display:72] Rebalance Overview
      +----------------+------------+--------------+
      | Nodes          | Services   | Status       |
      +----------------+------------+--------------+
      | 172.23.120.183 | kv, n1ql   | Cluster node |
      | 172.23.123.168 | ['kv,fts'] | <--- IN ---  |
      +----------------+------------+--------------+

      2. Upload x509 certs and enforce tls
      3. Create 20 fts indexes
      4. Create a bucket with 10 documents

      2021-12-14 01:04:22,369 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
      +----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
      | Bucket         | Type    | Replicas | Durability | TTL | Items | RAM Quota   | RAM Used | Disk Used |
      +----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
      | GleamBookUsers | membase | 1        | none       | 0   | 10    | 10485760000 | 99307168 | 12786084  |
      +----------------+---------+----------+------------+-----+-------+-------------+----------+-----------+
      

      5. Rebalance-in with parallel doc loading

      2021-12-14 01:04:25,279 | test  | INFO    | pool-1-thread-5 | [table_view:display:72] Rebalance Overview
      +----------------+------------+--------------+
      | Nodes          | Services   | Status       |
      +----------------+------------+--------------+
      | 172.23.123.168 | fts, kv    | Cluster node |
      | 172.23.120.183 | kv, n1ql   | Cluster node |
      | 172.23.120.219 | ['kv,fts'] | <--- IN ---  |
      +----------------+------------+--------------+

      6. Rebalance-out with parallel doc loading

      2021-12-14 01:14:38,407 | test  | INFO    | pool-1-thread-10 | [table_view:display:72] Rebalance Overview
      +----------------+-----------------+--------------+
      | Nodes          | Services        | Status       |
      +----------------+-----------------+--------------+
      | 172.23.123.168 | [u'fts', u'kv'] | --- OUT ---> |
      | 172.23.120.183 | kv, n1ql        | Cluster node |
      | 172.23.120.219 | fts, kv         | Cluster node |
      +----------------+-----------------+--------------+

      7. Rebalance in-out with parallel doc loading

      2021-12-14 01:30:09,375 | test  | INFO    | pool-1-thread-3 | [table_view:display:72] Rebalance Overview
      +----------------+-----------------+--------------+
      | Nodes          | Services        | Status       |
      +----------------+-----------------+--------------+
      | 172.23.120.183 | kv, n1ql        | Cluster node |
      | 172.23.120.219 | [u'fts', u'kv'] | --- OUT ---> |
      | 172.23.120.224 | ['kv,fts']      | <--- IN ---  |
      +----------------+-----------------+--------------+

      This rebalance's progress went above 100(ranged from 100% to 116% before finishing)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Given that the fix version for this is Neo, this test need to be repeated for a latest Neo build.

            abhinav Abhinav Dangeti added a comment - Given that the fix version for this is Neo, this test need to be repeated for a latest Neo build.

            Couldn't reproduce this issue on a Neo build.

            thejas.orkombu Thejas Orkombu added a comment - Couldn't reproduce this issue on a Neo build.

            People

              thejas.orkombu Thejas Orkombu
              sumedh.basarkod Sumedh Basarkod
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty