Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60789

[Rebalance] [FTS] : Rebalance out hangs during mutations to vector fields

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus
    • 7.6.0
    • fts
    • Operating System : Operating System : Debian GNU/Linux 11 (bullseye)
      Couchbase Enterprise Edition 7.6.0-2133

    Description

      Steps to reproduce

      1. Created a magma bucket named "test-bucket" with expiry timeout = 1
      2. Loaded 10k documents onto it - Had fields as follows
        1. content - a text field
        2. id - doc_id 
        3. type - was populated with "vector"
        4. vector - a vector of dimensions 1024
      3. Created a vector search index for the bucket
      4. Started mutations to the documents changing the vector dimensions from 1024 to 1526
      5. Created another vector search index for dim 1526
      6. Ran queries continuously 
      7. Removed an fts node
      8. Started a rebalance - Rebalance hangs

       

      The response from /pools/default/tasks show that the rebalance is hung at 49.89 from 50 minutes

      [{"statusId":"89e72cad218f0e16f28b69d26a41e3d8","type":"rebalance","subtype":"rebalance","recommendedRefreshPeriod":0.25,"status":"running","progress":49.89035087719298,"perNode":{"ns_1@172.23.96.220":{"progress":50},"ns_1@172.23.105.202":{"progress":16.44736842105263},"ns_1@172.23.105.192":{"progress":16.44736842105263},"ns_1@172.23.104.66":{"progress":100},"ns_1@172.23.105.199":{"progress":16.44736842105263},"ns_1@172.23.105.179":{"progress":100}},"detailedProgress":{"bucket":"test_bucket","bucketNumber":1,"bucketsCount":1,"perNode":{"ns_1@172.23.96.220":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.105.202":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.105.199":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.105.192":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.105.179":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.104.66":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}}}},"stageInfo":{"search":{"totalProgress":16.44736842105263,"perNodeProgress":{"ns_1@172.23.105.202":0.1644736842105263,"ns_1@172.23.105.192":0.1644736842105263,"ns_1@172.23.105.199":0.1644736842105263},"startTime":"2024-02-14T05:16:30.947-08:00","completedTime":false,"timeTaken":3283516},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.96.220":1,"ns_1@172.23.104.66":1,"ns_1@172.23.105.179":1},"startTime":"2024-02-14T05:16:13.614-08:00","completedTime":"2024-02-14T05:16:30.937-08:00","timeTaken":17322,"details":{"test_bucket":{"compactionInfo":{"perNode":{"ns_1@172.23.104.66":{"averageTime":1},"ns_1@172.23.105.179":{"averageTime":1},"ns_1@172.23.96.220":{"averageTime":1}}},"vbucketLevelInfo":{"move":{"averageTime":5513,"totalCount":2,"remainingCount":0},"backfill":{"averageTime":5372.5},"takeover":{"averageTime":8},"persistence":{"averageTime":1796.833333333333}},"replicationInfo":{"ns_1@172.23.96.220":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":0,"outDocsLeft":0},"ns_1@172.23.104.66":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":0,"outDocsLeft":0},"ns_1@172.23.105.179":{"inDocsTotal":0,"inDocsLeft":0,"outDocsTotal":0,"outDocsLeft":0}},"startTime":"2024-02-14T05:16:19.839-08:00","completedTime":"2024-02-14T05:16:30.935-08:00","timeTaken":11096}}},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.96.220":1},"startTime":"2024-02-14T05:16:30.937-08:00","completedTime":"2024-02-14T05:16:30.947-08:00","timeTaken":10}},"rebalanceId":"a4acefc82e935af3dba3d50f1ba8bfff","nodesInfo":{"active_nodes":["ns_1@172.23.104.66","ns_1@172.23.105.179","ns_1@172.23.105.192","ns_1@172.23.105.202","ns_1@172.23.96.220","ns_1@172.23.105.199"],"keep_nodes":["ns_1@172.23.104.66","ns_1@172.23.105.179","ns_1@172.23.105.192","ns_1@172.23.105.202","ns_1@172.23.96.220"],"eject_nodes":["ns_1@172.23.105.199"],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.105.192"}]

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            aditi.ahuja Aditi Ahuja
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty