Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8193

[system test] Memcached segfault during rebalance with workload running causes rebalance failure

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.1.0
    • 2.1.0
    • couchbase-bucket
    • Security Level: Public
    • 2.0.2-780-rel
      NAME="Ubuntu"
      VERSION="12.04.2 LTS, Precise Pangolin"
    • Ubuntu 64-bit

    Description

      15G RAM 500G regular disc machine
      cluster_ip: 172.23.105.23

      Create a 8 node cluster, create default bucket with 6G RAM, 1 replica, sasl bucket with 5G RAM, 1 replica

      Load items to both buckets into dgm state, around 70%.

      Access the data for 2 hours, with workload:
      default bucket: create:5,update:5,get:80,delete:5,expire:5,cache_miss:5, 10K ops/sec
      saslbucket: create:5,update:5,get:20,delete:10,expire:60,cache_miss:5., 10K ops/sec

      Then rebalance in one node with workload continue
      Find core.memcached under /data path.
      Saw a lot of errors like this:

      Control connection to memcached on 'ns_1@172.23.105.23' disconnected: {{badmatch,
      {error,
      timeout}},
      [

      {mc_client_binary, cmd_binary_vocal_recv, 5}

      ,

      {mc_client_binary, select_bucket, 2}

      ,

      {ns_memcached, ensure_bucket, 2}

      ,

      {ns_memcached, handle_info, 2}

      ,

      {gen_server, handle_msg, 5}

      ,

      {ns_memcached, init, 1}

      ,

      {gen_server, init_it, 6}

      ,

      {proc_lib, init_p_do_apply, 3}

      ]} (repeated 1 times)

      Rebalance failed in a short time

      This also happened in Windows the bug number is MB-8192
      The stack trace of the core.memcached.xxxx on some node will be added later

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Chisheng Chisheng Hong (Inactive)
            Chisheng Chisheng Hong (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty