Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37216

Eventing rebalance failure during kv node out rebalance

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build 6.5.0-4926

      Observed eventing worker died message while doing kv node out rebalance in high bucket density test, due to which rebalance failed.

      Following eventing function deployed in cluster-

      {
          "actionsVisible": true,
          "appcode": "function OnUpdate(doc, meta) {\n    bucket1[meta.id]=doc[\"alt_email\"];\n}\n\nfunction OnDelete(doc) {\n}\n",
          "appname": "bucket-1-test1",
          "depcfg": {
              "buckets": [
                  {
                      "alias": "bucket1",
                      "bucket_name": "eventing-bucket-1"
                  }
              ],
              "metadata_bucket": "eventing",
              "source_bucket": "bucket-1"
          },
          "settings": {
              "breakpad_on": false,
              "cpp_worker_thread_count": 2,
              "dcp_stream_boundary": "from_now",
              "deployment_status": true,
              "description": "Perf test",
              "enable_recursive_mutation": true,
              "log_level": "INFO",
              "processing_status": true,
              "timer_worker_pool_size": 1,
              "worker_count": 3,
              "worker_queue_cap": 100000
          }
      }
      

      Rebalance failure message-

      "Rebalance exited with reason {service_rebalance_failed,eventing
      ,\n {worker_died,\n {'EXIT',<0.20276.305>,\n {rebalance_failed,\n 
       \{service_error,\n <<\"eventing rebalance hasn't made progress for past 600 secs\">>}}}}}."}
      

      Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/339

      Logs-

      Eventing node- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.177.zip

      Other nodes-

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.20.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.12.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.13.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.14.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.15.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.19.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.20.zip

      We did not observed this issue in 6.5.0-4908.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankit.prabhu Ankit Prabhu
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty