Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4542

timeout and connection failures under moderate load

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0-developer-preview-3
    • Fix Version/s: 2.0-beta
    • Component/s: None
    • Security Level: Public
    • Labels:
      None
    • Environment:
      2.0 DP3
      originally reported on Debian GNU/Linux 6.0
      reproduced with slight differences on Ubuntu 11.10 64 bit

      Description

      Running initially with a PHP client, but then simplifying the test and running both clustered and unclustered basic workload tests shows that on both micro instances and large instances basic workload tests timeout and/or fail. On micro instances, this is shows up particularly quickly even in a single node localhost situation.

      Steps:

      • Set up EC2 instance
      • Install Couchbase Server 2.0 DP3
      • Run /opt/couchbase/bin/memcachetest -l -t 32 -h localhost:11211

      Observed behavior:
      Many errors such as:
      ASCII set failure: SERVER_ERROR proxy downstream timeout
      ASCII set failure: SERVER_ERROR proxy downstream timeout 10.48.43.67
      ASCII set failure: SERVER_ERROR proxy out of downstreams
      ASCII set failure: SERVER_ERROR proxy downstream timeout
      <3944> isn't there anymore
      ...

      Also note that the observed throughput would start high, slow down, speed up, etc.

      Expected behavior:
      Workload should degrade or possibly timeout in an extreme workload case, but not be issuing out of downstreams. The server should definitely handle this workload without errors.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        buzzy Daniele Testa added a comment -

        Why was Moxi added if it cripples memcached that much? Normal memcached has like 1024 connection limit by default. Moxi limits this to 16? Would make sense to follow memcached default limit to 1024 then.

        Show
        buzzy Daniele Testa added a comment - Why was Moxi added if it cripples memcached that much? Normal memcached has like 1024 connection limit by default. Moxi limits this to 16? Would make sense to follow memcached default limit to 1024 then.
        Hide
        buzzy Daniele Testa added a comment -

        Perry, but I am running all this on one machine. What difference would a "client-side" moxi do then? Do you have a link how to just turned the moxi off all together and rely on the clients hashing instead?

        Show
        buzzy Daniele Testa added a comment - Perry, but I am running all this on one machine. What difference would a "client-side" moxi do then? Do you have a link how to just turned the moxi off all together and rely on the clients hashing instead?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Daniele,

        memslap is not a supported tool or a binary against recent membase or couchbase servers. It would b best to try out with a smart client or use the client side moxi where you can change its configurations and increase the number of connections and etc as Perry mentioned.

        you should be able to use memcachetest to run smaple load against memcached directly if you have one node.

        Show
        farshid Farshid Ghods (Inactive) added a comment - Daniele, memslap is not a supported tool or a binary against recent membase or couchbase servers. It would b best to try out with a smart client or use the client side moxi where you can change its configurations and increase the number of connections and etc as Perry mentioned. you should be able to use memcachetest to run smaple load against memcached directly if you have one node.
        Hide
        perry Perry Krug added a comment -

        The "client-side" Moxi gives you the ability to change these parameters...the server-side has hardcoded defaults which are obviously too low for certain scenarios.

        There is no way to turn off Moxi altogether. You don't need to turn off the server-side one if you're using a client-side Moxi and/or smart client, the server-side one will just be sitting idle. Unless you are using a smart client (currently available for Java and .NET, soon to be released for PHP and Ruby), you must use Moxi.

        Show
        perry Perry Krug added a comment - The "client-side" Moxi gives you the ability to change these parameters...the server-side has hardcoded defaults which are obviously too low for certain scenarios. There is no way to turn off Moxi altogether. You don't need to turn off the server-side one if you're using a client-side Moxi and/or smart client, the server-side one will just be sitting idle. Unless you are using a smart client (currently available for Java and .NET, soon to be released for PHP and Ruby), you must use Moxi.
        Hide
        buzzy Daniele Testa added a comment -

        Ok, understood. I was looking for a way to change the config for the "server-side" moxi without success.

        I guess in my case, I will install another moxi beside the existing one and put it on another port and run my tests again.

        Btw, the "memslap" binary is included in your package so I assumed it was ok to use it
        I have also used memcachetest with the same result, but at least now I know why.

        I will let you know if the client-side moxi solves my issues. Thank you.

        Show
        buzzy Daniele Testa added a comment - Ok, understood. I was looking for a way to change the config for the "server-side" moxi without success. I guess in my case, I will install another moxi beside the existing one and put it on another port and run my tests again. Btw, the "memslap" binary is included in your package so I assumed it was ok to use it I have also used memcachetest with the same result, but at least now I know why. I will let you know if the client-side moxi solves my issues. Thank you.

          People

          • Assignee:
            Unassigned
            Reporter:
            ingenthr Matt Ingenthron
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes