Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28486

kv/eventing rebalance hangs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 5.5.0
    • 5.5.0
    • eventing
    • 5.5.0-2036

    Description

      Scripts to Repro

      ./testrunner -i /tmp/testexec.26219.ini get-cbcollect-info=True,GROUP=n1ql_op_with_timers,use_memory_manager=False -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_in_when_existing_eventing_node_is_processing_mutations,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=10,sock_batch_size=8,worker_count=8,cpp_worker_thread_count=8,handler_code=n1ql_op_with_timers,GROUP=n1ql_op_with_timers
      

      ./testrunner -i /tmp/testexec.22036.ini get-cbcollect-info=True,GROUP=bucket_op_with_cron_timers,use_memory_manager=False -t eventing.eventing_rebalance.EventingRebalance.test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations,nodes_init=6,services_init=kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=10,handler_code=bucket_op_with_cron_timers,GROUP=bucket_op_with_cron_timers
      

      Attached logs (1 in attachment and other has aws link)

      It should be noted there at-least 10 such hangs in windows/linux runs. I haven't attached logs from all. It can be found in below links.

      Automation Run for windows runs
      http://qa.sc.couchbase.com/job/test_suite_executor/52303/
      http://qa.sc.couchbase.com/job/test_suite_executor/52300/

      Attachments

        1. cpu utilization on all nodes.png
          339 kB
          Dave Finlay
        2. eventing_hosts_during_test_runs.png
          280 kB
          Abhishek Singh
        3. eventing_producer_file_owners.png
          247 kB
          Abhishek Singh
        4. memcached cpu.png
          236 kB
          Dave Finlay
        5. Resource_monitor.png
          267 kB
          Abhishek Singh
        6. Screen Shot 2018-03-23 at 11.43.40 AM.png
          125 kB
          Balakumaran Gopal
        For Gerrit Dashboard: MB-28486
        # Subject Branch Project Status CR V

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty