Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-9885

rebalance failure may happen during etimedout errors

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Critical
    • 3.0
    • 2.5.1, 2.5.0
    • ns_server
    • Security Level: Public
    • centos 6.2 64-bit, windows 2008 R2 64-bit

    Description

      Enviroment:
      8 nodes centos 6.2 64-bit (each has 32 GB RAM and SSD )
      1:10.6.2.42
      2:10.6.2.43
      3:10.6.2.44
      4:10.6.2.45
      5:10.6.2.66
      6:10.6.2.68
      7:10.6.2.69

      8:10.6.2.89

      Setup:
      Create a 7 nodes cluster with couchbase server 2.5.0-1024
      Create 2 buckets:
      sasl-1 with 1 replica, 1 doc with 1 view
      sasl-2 with 2 replica, 1 doc with 1 view

      Load items with size from 128 bytes to 512 bytes into cluster.
      Load dgm until active resident got 90% at each bucket.
      Then test
      Warmup test without load: passed
      Warmup test with load: passed
      Add a node (89) and rebalance: failed with error

      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.7063.546>,
      {badmatch,
      [{'EXIT',
      badmatch,{error,etimedout,

      {gen_server,call, [<17007.23364.496>,had_backfill, 30000]}

      }}]}}}
      ns_orchestrator002 ns_1@10.6.2.42 16:10:48 - Thu Jan 9, 2014
      <0.6159.546> exited with {unexpected_exit,
      {'EXIT',<0.7063.546>,
      {badmatch,
      [{'EXIT',
      badmatch,{error,etimedout,

      {gen_server,call, [<17007.23364.496>,had_backfill,30000]}

      }}]}}} ns_vbucket_mover000 ns_1@10.6.2.42 16:10:37 - Thu Jan 9, 2014
      Bucket "sasl-2" rebalance does not seem to be swap rebalance

      After a while, tried to rebalance again. It's still failed with same error

      I will try rebalance one more time and update result.

      Attachments

        1. 10.6.2.42-1142014-1141-diag.zip
          35.82 MB
        2. 10.6.2.43-1142014-1146-diag.zip
          36.44 MB
        3. 10.6.2.44-1142014-1152-diag.zip
          35.94 MB
        4. 10.6.2.45-1142014-1157-diag.zip
          36.23 MB
        5. 10.6.2.66-1142014-122-diag.zip
          43.44 MB
        6. 10.6.2.68-1142014-128-diag.zip
          34.46 MB
        7. 10.6.2.69-1142014-1212-diag.zip
          37.41 MB
        8. 10.6.2.89-1142014-1218-diag.zip
          28.45 MB
        9. cpu_node_66.cpu0
          1.20 MB
        10. cpu_node_66.cpu1
          1.18 MB
        11. cpu_node_66.cpu2
          1.18 MB
        12. cpu_node_66.cpu3
          1.18 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pvarley Patrick Varley (Inactive)
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty