Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37146

[Magma]: Rebalance failed due to memcached crash on orchestrator node

    XMLWordPrintable

Details

    Description

      1. Create a cluster with 1 node and a default bucket with replica=1.
      2. Add 3 nodes into the cluster and start 1M doc loading in parallel.

      +----------------+----------+--------------+
      | Nodes          | Services | Status       |
      +----------------+----------+--------------+
      | 172.23.105.220 | kv       | Cluster node |
      | 172.23.105.221 | None     | <--- IN ---  |
      | 172.23.105.223 | None     | <--- IN ---  |
      | 172.23.105.225 | None     | <--- IN ---  |
      +----------------+----------+--------------+
      

      3. Rebalance is stuck at 1%, 12 vBuckets have moved so far. Rebalance failed after some time.

      Service 'memcached' exited with status 139 on node .221(incoming node)
      Service 'memcached' exited with status 134 in node .220(Orchestrator node)

      memcahced is crashing in infinite loop.

      Note: Test passed with couchstore.

      QE Note:

      -p num_items=1000000,GROUP=P0;default,magma_storage=True -t rebalance_new.rebalance_in.RebalanceInTests.test_rebalance_in_with_ops,nodes_in=3,GROUP=P0;default,skip_cleanup=True, -p rerun=False,infra_log_level=debug,log_level=debug -m rest
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            scott.lashley Scott Lashley
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty