Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36400

[high-bucket] FTS rebalance out in high bucket density test takes really long hours (28hrs+)

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      Build 6.5.0-4380

      Observed that rebalancing out FTS node in high bucket density test with 30 buckets taking 764.42 min and fails with following reason.

      Each bucket have 1 FTS index.

      Rebalance failure reason-

      Rebalance exited with reason {service_rebalance_failed,fts,
      {worker_died,
      {'EXIT',<0.13606.525>,
      {{linked_process_died,<27016.24380.381>,
      {timeout,
      {gen_server,call,
      [<27016.22501.381>,
      {call,"ServiceAPI.GetCurrentTopology",
      #Fun<json_rpc_connection.0.102434519>},
      60000]}}},
      {gen_server,call,
      [{'service_agent-fts','ns_1@172.23.97.15'},
      {if_rebalance,<0.19462.525>,
      {start_rebalance,
      <<"f81721335b42662e837612d6b366cca3">>,
      rebalance,
      [{[{node_id,
      <<"827b0f5298e3447fa469c0ebdcad3da0">>},
      {priority,0},
      {opaque,null}],
      full}],
      [[{node_id,
      <<"592321897f0f721188d778a77f4c9d9b">>},
      {priority,0},
      {opaque,null}]],
      <0.13606.525>}},
      90000]}}}}}.
      Rebalance Operation Id = f5116259df628e7619516c5b6cba1997
      

       On build 6.5.0-2082, this rebalance out took 79 min.

      Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/

      Logs-
      FTS node- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip
      FTS node going out- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip

      Other nodes-
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.12.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.13.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.14.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.177.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.19.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.20.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            No work has yet been logged on this issue.

            People

              jyotsna.nayak Jyotsna Nayak
              mahesh.mandhare Mahesh Mandhare (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty