Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45053

[System Test] Eventing Rebalance failed

    XMLWordPrintable

Details

    Description

      System Test:

      Eventing handlers deployment{

      [2021-03-17T16:06:53-07:00, sequoiatools/eventing:7.0:a87aaf] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -s default.event_0.coll0 -m ITEM.event_0.coll0 -d dst_bucket.NEW_ORDER.event_0.coll0.rw -t timers -o create --name timers
      [2021-03-17T16:07:01-07:00, sequoiatools/eventing:7.0:dfee66] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -s default.event_0.coll0 -m ITEM.event_0.coll1 -d dst_bucket.NEW_ORDER.event_0.coll1.rw -t n1ql -o create --name n1ql
      [2021-03-17T16:07:08-07:00, sequoiatools/eventing:7.0:dd9a54] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -s WAREHOUSE.event_0.coll0 -m ITEM.event_0.coll2 -d dst_bucket.WAREHOUSE.event_0.coll0.rw -t sbm -o create --name sbm
      [2021-03-17T16:07:17-07:00, sequoiatools/eventing:7.0:ef5686] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -s WAREHOUSE.event_0.coll0 -m ITEM.event_0.coll3 -d dst_bucket.NEW_ORDER.event_0.coll2.rw -t curl -o create --name curl
      [2021-03-17T16:07:25-07:00, sequoiatools/eventing:7.0:b3959c] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -o deploy
      [2021-03-17T16:07:30-07:00, sequoiatools/eventing:7.0:d2d34d] eventing_helper.py -i 172.23.105.183 -u Administrator -p password -o wait_for_state --state deployed
      

      – At this point of time there should be not data in collections.

      Current step

      [2021-03-17T21:13:59-07:00, sequoiatools/couchbase-cli:7.0:e63846] server-add -c 172.23.104.232:8091 --server-add https://172.23.104.244 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2021-03-17T21:14:11-07:00, sequoiatools/couchbase-cli:7.0:d5525c] rebalance -c 172.23.104.232:8091 --server-remove 172.23.105.25 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.104.232:8091 --server-remove 172.23.105.25 -u Administrator -p password]
       
      docker logs d5525c
      docker start d5525c
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      Rebalance Failed -

      Rebalance exited with reason {service_rebalance_failed,eventing,
      {worker_died,
      {'EXIT',<0.1247.496>,
      {{badmatch,
      {error,
      {bad_nodes,eventing,prepare_rebalance,
      [{'ns_1@172.23.104.214',
      {error,
      {unknown_error,
      <<"Some apps are deploying or resuming on nodeId: d0c98164b79fbb3e57b4808d7c71ef3b Apps: map[timers_0:2021-03-17 16:07:27.988668711 -0700 PDT m=+2440.940195591]">>}}}]}}},
      [{service_rebalancer,rebalance_worker,1,
      [{file,"src/service_rebalancer.erl"},
      {line,164}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,234}]}]}}}}.
      Rebalance Operation Id = 6c309e9434208e5789108f241ad43a4b
      

      – All 4 handlers above are undeployed.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            arunkumar Arunkumar Senthilnathan added a comment - - edited

            As per discussion with Vikas Chaudhary, this can be closed out if longevity runs with eventing for 2 iterations and does not hit this issue - currently longevity is blocked on MB-45459

            arunkumar Arunkumar Senthilnathan added a comment - - edited As per discussion with Vikas Chaudhary , this can be closed out if longevity runs with eventing for 2 iterations and does not hit this issue - currently longevity is blocked on MB-45459

            Build couchbase-server-7.0.0-4881 contains eventing commit 1ea70cd with commit message:
            MB-45053: Revert gocb to 2.1.5

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4881 contains eventing commit 1ea70cd with commit message: MB-45053 : Revert gocb to 2.1.5
            arunkumar Arunkumar Senthilnathan added a comment - - edited http://172.23.109.231/job/centos-systest-launcher/2362/console running with 7.0.0-4910

            Issue not seen in run with 7.0.0-4910

            arunkumar Arunkumar Senthilnathan added a comment - Issue not seen in run with 7.0.0-4910

            Build couchbase-server-7.0.0-4964 contains eventing commit efcb129 with commit message:
            Revert "MB-45053: Upgrade gocb@2.2.2 + fixes"

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4964 contains eventing commit efcb129 with commit message: Revert " MB-45053 : Upgrade gocb@2.2.2 + fixes"

            People

              arunkumar Arunkumar Senthilnathan
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty