Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37179

[System test]: Eventing rebalance failing with Some apps are being paused

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4917, not seen on 6.5.0-4908

      Test: MH longevity 

      Cycle: 2nd

      Day: 2nd

      Seeing multiple rebalance failure for eventing 

      [user:error,2019-12-05T15:10:18.944-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.29188.1182>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = da3ae28e304d693f84567e0bdc3cab39
       
       
      [user:error,2019-12-05T15:35:01.557-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.23000.1196>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 5ee66f3b1ce6892b90e0b7af66d97beb
       
       
       
       
      [user:error,2019-12-05T16:11:57.527-08:00,ns_1@172.23.108.103:<0.13067.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.9006.1218>,
                                      {{badmatch,
                                        {error,
                                         {unknown_error,
                                          <<"Some apps are being paused, node: 172.23.96.148:8096">>}}},
                                       [{service_rebalancer,rebalance_worker,1,
                                         [{file,"src/service_rebalancer.erl"},
                                          {line,170}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = 56a74378a8d5a9737f816839f813f608 

      Rebalance is even failing when it involves only index. We should ignore other services rebalance

      [2019-12-05T15:05:15-08:00, sequoiatools/couchbase-cli:6.5:6c7db0] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.164 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
      [2019-12-05T15:05:44-08:00, sequoiatools/couchbase-cli:6.5:580796] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs 580796
      docker start 580796
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T15:10:35-08:00, sequoiatools/cmd:3ee376] 60
       
       
       
       
      [2019-12-05T15:28:52-08:00, sequoiatools/couchbase-cli:6.5:5bfb5a] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
      warning using 'json' filter:  unexpected end of JSON input []
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
       
       
      docker logs 5bfb5a
      docker start 5bfb5a
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T15:35:16-08:00, sequoiatools/cmd:a23d6f] 60
       
       
       
       
      [2019-12-05T16:07:42-08:00, sequoiatools/couchbase-cli:6.5:c0ad80] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.242 -u Administrator -p password]
       
       
      docker logs c0ad80
      docker start c0ad80
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-12-05T16:12:18-08:00, sequoiatools/cmd:12bade] 60
       

      Test not recovered after this all rebalance failing 

      eventing : 3 ===== > [172.23.104.87:8091 172.23.96.148:8091 172.23.98.135:8091]

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.0.0-1119 contains eventing commit c4caf9e with commit message:
          MB-37179 Move stats loading into a separate go routine

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1119 contains eventing commit c4caf9e with commit message: MB-37179 Move stats loading into a separate go routine

          Build couchbase-server-6.5.0-4936 contains eventing commit 9253758 with commit message:
          MB-37179 Move stats loading into a separate go routine

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4936 contains eventing commit 9253758 with commit message: MB-37179 Move stats loading into a separate go routine

          Build couchbase-server-6.5.1-6005 contains eventing commit 9253758 with commit message:
          MB-37179 Move stats loading into a separate go routine

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains eventing commit 9253758 with commit message: MB-37179 Move stats loading into a separate go routine

          Build couchbase-server-6.5.1-6005 contains eventing commit ad17df9 with commit message:
          MB-37179: Takes sleep out of critical section

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains eventing commit ad17df9 with commit message: MB-37179 : Takes sleep out of critical section

          Not seen on 6.5.0-4947

          vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 6.5.0-4947

          People

            Gautham.Banasandra Gautham Banasandra (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty