Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29271

Eventing Rebalance in hangs when memcached is killed on kv and eventing nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.0.0
    • 5.5.0
    • eventing
    • 5.5.0-2497

    • Untriaged
    • Centos 64-bit
    • No

    Description

      Script to Repro

      ./testrunner -i /tmp/testexec.8098.ini -p get-cbcollect-info=True,GROUP=bucket_op -t eventing.eventing_rebalance.EventingRebalance.test_memcache_crash_on_kv_and_eventing_node_during_eventing_rebalance,doc-per-day=10,dataset=default,nodes_init=5,services_init=kv-kv-eventing-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op

      Steps
      1) Create a 5 node cluster of kv-kv-eventing-eventing-index:n1ql
      2) Deployed a eventing function.
      3) Start loading docs on source bucket.
      4) When 3 is in progress rebalance in an eventing node.
      5) After rebalance reaches 30% or so kill memcached on 1 kv(172.23.108.91) and 1 eventing(172.23.109.137) node.

      Rebalance hangs. Logs attached.

      Cluster details
      172.23.107.67 - kv
      172.23.108.91 - kv
      172.23.109.137 - eventing
      172.23.109.152 - eventing
      172.23.109.153 - index:n1ql
      172.23.98.165 - eventing (Rebalancing in)

      cbcollect_info : https://s3.amazonaws.com/bugdb/jira/memcache_crash_hang/test_24.zip

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-29271
          # Subject Branch Project Status CR V

          Activity

            People

              asingh Abhishek Singh (Inactive)
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty