Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30830

K8S: Service rebalance failed when rebalancing in the analytics node in the cluster

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 115

    Description

      CB server build used: Enterprise Edition 5.5.0 build 2958

      This test was executed using docker images running on kubernetes1.10.0 platform

      Test scenario:

      1. Created 7 node cluster with 1 (data+analytics), 3 (data+index+query) and 3 (analytics) node
      2. Populated "default" bucket with data and created analytics function and wait for analytics function to deploy fully
      3. Separate thread was started to insert data into "default" bucket in parallel
      4. Killed 5th node (0004) which is running only analytics
      5. New node (0007) was created to replace the killed node and trying to rebalance the cluster

      During this, rebalance is failing with the reason,

      Rebalance exited with reason {service_rebalance_failed,cbas, {rebalance_failed, {service_error, <<"Rebalance 7700522e1718b5691f8ee8365ee7baf2 failed, see analytics log for details">>}}}

      Attaching cb cluster logs along with the ticket.

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. K8S-459 Stabilize all regression for new features

          • Major - Major loss of function.
          • Closed
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-30873 Analytics Service exiting during rebalance with error "Unexpectedly error during replication"

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. K8S-566 Rebalance failing with errors service_rebalance_failed when analytics service is enabled

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. K8S-615 Ephemeral pod with log PV: Seeing multiple rebalance failure when server pod is killed

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Replication fix commit

          Activity

            People

              tanzeem.ahmed Tanzeem Ahmed (Inactive)
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty