Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37179

[System test]: Eventing rebalance failing with Some apps are being paused

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4917, not seen on 6.5.0-4908

      Test: MH longevity 

      Cycle: 2nd

      Day: 2nd

      Seeing multiple rebalance failure for eventing 

      [user:error,2019-12-05T15:10:18.944-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.29188.1182>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = da3ae28e304d693f84567e0bdc3cab39
       
       
      [user:error,2019-12-05T15:35:01.557-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.23000.1196>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 5ee66f3b1ce6892b90e0b7af66d97beb
       
       
       
       
      [user:error,2019-12-05T16:11:57.527-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.9006.1218>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 56a74378a8d5a9737f816839f813f608 

      Rebalance is even failing when it involves only index. We should ignore other services rebalance

      [2019-12-05T15:05:15-08:00, sequoiatools/couchbase-cli:6.5:6c7db0] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.164 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
      [2019-12-05T15:05:44-08:00, sequoiatools/couchbase-cli:6.5:580796] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs 580796
      docker start 580796
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T15:10:35-08:00, sequoiatools/cmd:3ee376] 60
       
       
       
       
      [2019-12-05T15:28:52-08:00, sequoiatools/couchbase-cli:6.5:5bfb5a] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
      warning using 'json' filter:  unexpected end of JSON input []
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
       
       
      docker logs 5bfb5a
      docker start 5bfb5a
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T15:35:16-08:00, sequoiatools/cmd:a23d6f] 60
       
       
       
       
      [2019-12-05T16:07:42-08:00, sequoiatools/couchbase-cli:6.5:c0ad80] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
       
       
      docker logs c0ad80
      docker start c0ad80
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T16:12:18-08:00, sequoiatools/cmd:12bade] 60
       

      Test not recovered after this all rebalance failing 

      eventing : 3 ===== > [172.23.104.87:8091 172.23.96.148:8091 172.23.98.135:8091]

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          vikas.chaudhary Vikas Chaudhary created issue -
          jeelan.poola Jeelan Poola made changes -
          Field Original Value New Value
          Assignee Jeelan Poola [ jeelan.poola ] Ankit Prabhu [ ankit.prabhu ]
          jeelan.poola Jeelan Poola made changes -
          Assignee Ankit Prabhu [ ankit.prabhu ] Gautham Banasandra [ gautham.banasandra ]
          jeelan.poola Jeelan Poola made changes -
          Is this a Regression? Yes [ 10450 ] No [ 10451 ]
          lynn.straus Lynn Straus made changes -
          Labels system-test approved-for-mad-hatter system-test
          lynn.straus Lynn Straus made changes -
          Due Date 10/Dec/19
          lynn.straus Lynn Straus made changes -
          Link This issue blocks MB-36676 [ MB-36676 ]
          jeelan.poola Jeelan Poola made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          vikas.chaudhary Vikas Chaudhary made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            Gautham.Banasandra Gautham Banasandra (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty