Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37200

[System test]: Eventing rebalance hung ,failed with timeout

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 6.5.0
    • 6.5.0
    • eventing

    Description

      Build: 6.5.0-4926 , not seen on 6.5.0-4908

      Test: MH longevity

      Day: 2nd

      Cycle: 1st and 2nd

      Seeing Multiple occurrences of timeout 

      [2019-12-07T02:40:31-08:00, sequoiatools/couchbase-cli:6.5:80497e] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2019-12-07T02:41:15-08:00, sequoiatools/couchbase-cli:6.5:2767f4] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs 2767f4
      docker start 2767f4
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-07T03:06:45-08:00, sequoiatools/cmd:a79d85] 60
       
       
       
      [user:error,2019-12-07T03:06:13.881-08:00,ns_1@172.23.108.103:<0.26696.357>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.13972.1071>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 600 secs">>}}}}}. 

      [2019-12-07T03:20:52-08:00, sequoiatools/couchbase-cli:6.5:ab9dde] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.5 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2019-12-07T03:32:36-08:00, sequoiatools/couchbase-cli:6.5:72069a] server-add -c 172.23.108.103:8091 --server-add https://172.23.99.20 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2019-12-07T03:33:14-08:00, sequoiatools/couchbase-cli:6.5:2d5e1f] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs 2d5e1f
      docker start 2d5e1f
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-07T03:56:17-08:00, sequoiatools/cmd:454a13] 60
       
       
       
       
       
       
      [user:error,2019-12-07T03:55:40.784-08:00,ns_1@172.23.108.103:<0.26696.357>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.414.1104>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 600 secs">>}}}}}.
      Rebalance Operation Id = ddf558dfc73434f31e1f474c60eb81a4 

      [2019-12-07T08:13:46-08:00, sequoiatools/couchbase-cli:6.5:4b9823] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.5 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      [2019-12-07T08:14:15-08:00, sequoiatools/couchbase-cli:6.5:d82136] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs d82136
      docker start d82136
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-07T08:31:29-08:00, sequoiatools/cmd:7a8168] 60
       
       
       
       
      [user:error,2019-12-07T08:30:47.970-08:00,ns_1@172.23.108.103:<0.26696.357>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.6373.1283>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 600 secs">>}}}}}.
      Rebalance Operation Id = f59864e651935d9a6b4867f44b063d9c 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          No work has yet been logged on this issue.

          People

            satya.nand Satya Nand (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty