Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25719

Plasma: Indexer swap rebalance failed with error "Timeout exceeded while awaiting headers"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.0.0
    • 5.0.0
    • secondary-index
    • None
    • Untriaged
    • Unknown

    Description

      Observed indexer rebalance failing with following error in logs:
      Rebalance exited with reason {service_rebalance_failed,index, {rebalance_failed, {service_error, <<"Post http://127.0.0.1:9102/dropIndex: net/http: request canceled (Client.Timeout exceeded while awaiting headers)">>}}}

      Note that this is observed in perf test to get swap rebalance time for 100M docs, 15k kv ops were running in background.
      Without background ops rebalance completes successfully.

      Job: http://perf.jenkins.couchbase.com/job/secondary-plasma-dev/160/
      Logs:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.98.118.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.98.119.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.118.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.119.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.122.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.123.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.241.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.245.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.246.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.247.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dev-160/172.23.99.251.zip
      Indexer nodes: 172.23.98.118 and 172.23.98.119

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sarath Sarath Lakshman
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty