Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45429

[System Test] Rebalance exited with timeout error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • ns_server
    • Untriaged
    • 1
    • Unknown

    Description

      7.0.0-4797

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_longevity_no_eventing_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      Iteration 3

      Console:

      [2021-03-31T21:30:13-07:00, sequoiatools/couchbase-cli:7.0:64e110] rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.14:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.97.74:8091 --server-remove 172.23.96.14:8091 -u Administrator -p password]
       
      docker logs 64e110
      docker start 64e110
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      .106.134:

      2021-03-31T21:30:13.907-07:00, ns_orchestrator:0:info:message(ns_1@172.23.106.134) - Starting rebalance, KeepNodes = ['ns_1@172.23.106.134','ns_1@172.23.106.136',
                                       'ns_1@172.23.120.58','ns_1@172.23.120.73',
                                       'ns_1@172.23.120.75','ns_1@172.23.120.77',
                                       'ns_1@172.23.120.81','ns_1@172.23.120.86',
                                       'ns_1@172.23.121.77','ns_1@172.23.123.24',
                                       'ns_1@172.23.123.25','ns_1@172.23.123.26',
                                       'ns_1@172.23.123.31','ns_1@172.23.123.32',
                                       'ns_1@172.23.123.33','ns_1@172.23.96.122',
                                       'ns_1@172.23.96.243','ns_1@172.23.97.110',
                                       'ns_1@172.23.97.112','ns_1@172.23.97.148',
                                       'ns_1@172.23.97.149','ns_1@172.23.97.150',
                                       'ns_1@172.23.97.241','ns_1@172.23.97.74'], EjectNodes = ['ns_1@172.23.96.14'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 8d66187567514398e93f2679c87a8564
      2021-03-31T21:30:30.499-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.106.134) - Rebalance exited with reason {timeout,
                                       {gen,call,
                                           [{kv,'ns_1@172.23.123.31'},
                                            command,15000]}}.
      Rebalance Operation Id = 8d66187567514398e93f2679c87a8564
      

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1617257293/collectinfo-2021-04-01T060814-ns_1%40172.23.97.74.zip

      Attachments

        1. screenshot-1.png
          screenshot-1.png
          153 kB
        2. screenshot-2.png
          screenshot-2.png
          93 kB
        3. screenshot-3.png
          screenshot-3.png
          51 kB
        4. screenshot-4.png
          screenshot-4.png
          37 kB
        5. screenshot-5.png
          screenshot-5.png
          54 kB

        Issue Links

          Activity

            People

              Aliaksey Artamonau Aliaksey Artamonau (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty