Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45837

[System Test][Eventing]Rebalance after adding event node failed after ~5 hours in longevity

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      7.0.0-4955

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      Iteration 3

      .108.103 diag.log:

      2021-04-20T15:54:11.170-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket bucket9
      2021-04-20T15:54:13.436-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "bucket9" rebalance appears to be swap rebalance
      2021-04-20T15:54:15.919-07:00, ns_rebalancer:0:info:message(ns_1@172.23.108.103) - Started rebalancing bucket default
      2021-04-20T15:54:18.640-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.108.103) - Bucket "default" rebalance appears to be swap rebalance
      2021-04-20T20:06:57.821-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.5111.3209>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 1200 secs">>}}}}}.
      Rebalance Operation Id = a70c4d3da7e8b0c16e2669fdc2fde92a
      

      Console:

      [2021-04-20T15:52:38-07:00, sequoiatools/couchbase-cli:7.0:aa59e1] server-add -c 172.23.108.103:8091 --server-add https://172.23.97.239 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      [2021-04-20T15:53:01-07:00, sequoiatools/couchbase-cli:7.0:ac12a0] rebalance -c 172.23.108.103:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
      docker logs ac12a0
      docker start ac12a0
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2021-04-20T20:07:05-07:00, sequoiatools/cmd:4ee6ff] 60
      

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618974657/collectinfo-2021-04-21T031104-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritam.sharma Ritam Sharma
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty