Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28775

zombie eventing-consumer processes are running even after couchbase-sever is uninstalled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.5.0
    • 5.5.0
    • eventing
    • 5.5.0-2211
    • Untriaged
    • Windows 64-bit
    • No

    Description

      How it was seen
      This was noticed while running eventing tests on windows server pool.
      1) Run eventing jobs on a set of windows servers(Some of the scripts might pass/fail)
      2) Each tests resets the cluster before starting.
      3) Once the tests are run, the nodes are returned back to server pool and other scripts pickup these server.
      4) When the new tests run they fail in installation as these processes still are running and are accessing the couchbase installed directories.

      Findings on the server
      Couchbase installations fail on the server and we noticed lot of processes with .rbf extension.
      See 

      When we clicked on "Go to details" on these individual processes they pointed to eventing-consumer.exe as the owner.
      See 

      At the end of rebalance tests there were 92 eventing-consumers running(172.23.211.46)

      C:\Users\Administrator>tasklist | FIND "eventing-consumer.exe" /c
      92
       
      C:\Users\Administrator>
      

      Impact
      This causes all the subsequent tests to fail after eventing tests as the installation fails

      Workaround
      Currently we are killing eventing-consumer's manually or rebooting the servers manually before installation. I will add a fix in testrunner to kill eventing-consumers before installation.

       
      I had a discussion with Abhishek Singh regarding. We have identified the scripts/jobs that caused these failures.
      Following are the node/job combinations for the same( Jobs might not have any data in couple of days)
      The hosts are available for live debugging. Please let us know once you are done with these hosts.

      172.23.211.46
      http://qa.sc.couchbase.com/job/test_suite_executor/55672/consoleText
      

      172.23.108.66
      http://qa.sc.couchbase.com/job/test_suite_executor/56140/consoleText
      

      Attaching cbcollect logs for 172.23.211.46.
       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            satya.nand Satya Nand (Inactive)
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty