Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50561

[System Test] FTS rebalance hung for 30+ hrs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • fts

    Description

      Build : 7.1.0-2126
      Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale : 3
      Iteration : 2nd

      In the 2nd iteration of the test, the following rebalance operation is stuck due to FTS

      [2022-01-23T02:04:59-08:00, sequoiatools/couchbase-cli:7.1:cdd10d] server-add -c 172.23.108.103:8091 --server-add https://172.23.99.11 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics
      [2022-01-23T02:05:23-08:00, sequoiatools/couchbase-cli:7.1:463d16] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      

      172.23.105.111 and 172.23.96.148 are the FTS nodes in the cluster. Unfortunately, we haven't been able to collect logs since 2022-01-22 08:24:46 due to other issues in the system. The current set of logs are attached. Not sure if they will be helpful, but the rebalance was still stuck while the logs were collected.

      We will attempt a re-run of the test as well to reproduce the issue.

      This seems to be a regression as we did not encounter this issue in the previous run with 7.1.0-2097.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty