Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58191

FTS rebalance hung during ScaleIn

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.6.0
    • 7.2.1
    • fts

    Description

      Build: Enterprise Edition 7.2.1 build 17478

      Test Steps:

      1. 15 scopes and 60 collections across 3 buckets
        • Each bucket -> 10M data 1 index on each collection -> 60 indexes (each index ~400k docs)
      2. Sleep - 15 mins
      3. Run FTS Queries FTS Flex Queries Random fashion for 2 hrs
      4. Kill CBFT
      5. sleep for 15 mins
      6. Scale out to 5 nodes
      7. Create 10 more fts indexes with following config on default scope and default index (500k docs per index) :
        • 1 Index | 1 Replica | 5 Partitions
        •  3 Indexes | 2 Replicas | 6 Partitions
        • 6 Indexes | 0 Replicas | 4 Partitions
      8. Sleep for 15 mins
      9. Run FTS Queries FTS Flex Queries Random fashion for 30 mins again
      10.  Kill CBFT
      11. Sleep for 15 mins
      12. Rebalance/Scale in to 4 nodes

      http://qe-jenkins1.sc.couchbase.com/job/cp-cli-runner/2905/console

      Seeing search rebalance hung for ~20Hrs while scaling In.

      Attachments

        1. image-2023-08-08-11-12-40-512.png
          436 kB
          Sarthak Dua
        2. image-2023-08-10-19-44-00-642.png
          1.72 MB
          Sarthak Dua
        3. image-2023-08-10-19-44-24-824.png
          1.75 MB
          Sarthak Dua
        4. image-2023-08-11-15-03-46-925.png
          1.52 MB
          Sarthak Dua
        5. image-2023-08-11-15-04-36-620.png
          360 kB
          Sarthak Dua
        6. image-2023-08-11-15-21-09-236.png
          474 kB
          Sarthak Dua
        7. image-2023-08-11-15-22-05-748.png
          293 kB
          Sarthak Dua
        8. image-2023-08-11-15-23-58-262.png
          130 kB
          Sarthak Dua
        9. image-2023-08-11-16-05-50-369.png
          123 kB
          Sarthak Dua
        10. image-2023-08-11-16-05-52-965.png
          123 kB
          Sarthak Dua
        11. image-2023-08-11-16-08-49-535.png
          119 kB
          Sarthak Dua
        12. image-2023-08-11-16-08-59-481.png
          119 kB
          Sarthak Dua
        13. image-2023-08-14-21-40-36-169.png
          239 kB
          Sarthak Dua
        14. image-2023-08-14-21-43-19-459.png
          301 kB
          Sarthak Dua
        15. Screenshot 2023-08-09 at 11.36.56 AM.png
          1.70 MB
          Sarthak Dua
        16. Screenshot 2023-08-09 at 11.37.02 AM.png
          1.79 MB
          Sarthak Dua
        17. Screenshot 2023-08-11 at 4.08.38 PM.png
          150 kB
          Sarthak Dua
        18. Screenshot 2023-08-16 at 12.35.31 PM.png
          364 kB
          Shaad Khan
        19. Screenshot 2023-08-16 at 12.36.19 PM.png
          555 kB
          Shaad Khan

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-57607 SystemTest - Provisioned - Search rebalance taking long time

          • Critical - Crashes, loss of data, severe memory leak.
          • In Progress

          Bug - A problem which impairs or prevents the functions of the product. MB-58195 [CAPELLA] Scaling failed due to FTS rebalance being stuck on 7.2.1-5904

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mohd.shaadkhan Shaad Khan
              sarthak.dua Sarthak Dua
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty