Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.1.6
    • Fix Version/s: 1.2.7
    • Component/s: library
    • Labels:
      None

      Description

      In our production system thread occasionally get aborted to prevent long-running tasks from using resources when the user is no longer interested in the results. We are seeing these kinds of errors in the log:

      ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Could not init pool.
      and
      ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Failed to reset an acquired socket.

      Also, we are seeing hangs in MessageStreamListener.cs, line 400:
      while ((line = reader.ReadLine()) != null)

      This will occasionally hang indefinitely waiting for data to come in to the stream when there is no more data coming in.
      I have created a reproduction application that creates a bunch of threads in an "abortable" thread pool (which just allows you to abort the threads, instead of being a black box). Starting the application will print a "." for every thread started, a "!" for every thread stopped and an "x" for every thread aborted. The thread itself just does a simple set/get. It typically hangs after about 15-20 seconds, on this ReadLine statement.

      You can find the repro attached to this ticket.

      This happens with Couchbase server 1.8.1 and the latest client code from Github.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        ingenthr Matt Ingenthron added a comment -

        Can you see if this is reproducible with current server/client?

        Show
        ingenthr Matt Ingenthron added a comment - Can you see if this is reproducible with current server/client?
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        Cluster version - 2.0.1-170
        Client - latest from Github

        Please find the attached program HangRepro-NCBC-111.zip that I used to reproduce the scenario.
        The output is "output-with-using-statement.txt" (attached herewith)
        The program hangs after 15-20 seconds.

        Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this:

        while (true)
        {
        Console.Write(".");
        var client = new CouchbaseClient(section);
        // Start a thread that just does a simple add/get and then stops
        var wi = AbortableThreadPool.QueueUserWorkItem(_ =>

        { client.Store(StoreMode.Add, "somekey", "somevalue"); var someValue = client.Get<string>("somekey"); if (someValue != "somevalue") throw new InvalidOperationException(); Console.Write("!"); }

        );

        // Maybe kill the thread
        if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);

        // Wait a bit
        Thread.Sleep((int)Math.Floor(rnd.NextDouble() * 10.0));

        // Maybe kill the thread
        if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);
        }

        With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine.
        The output is "output-without-using-statement.txt" (attached herewith)

        Show
        saakshi.manocha Saakshi Manocha added a comment - Cluster version - 2.0.1-170 Client - latest from Github Please find the attached program HangRepro- NCBC-111 .zip that I used to reproduce the scenario. The output is "output-with-using-statement.txt" (attached herewith) The program hangs after 15-20 seconds. Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this: while (true) { Console.Write("."); var client = new CouchbaseClient(section); // Start a thread that just does a simple add/get and then stops var wi = AbortableThreadPool.QueueUserWorkItem(_ => { client.Store(StoreMode.Add, "somekey", "somevalue"); var someValue = client.Get<string>("somekey"); if (someValue != "somevalue") throw new InvalidOperationException(); Console.Write("!"); } ); // Maybe kill the thread if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true); // Wait a bit Thread.Sleep((int)Math.Floor(rnd.NextDouble() * 10.0)); // Maybe kill the thread if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true); } With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine. The output is "output-without-using-statement.txt" (attached herewith)
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        @Roy: Could you please try again without the "using" statement, and let me know if it helps!

        Show
        saakshi.manocha Saakshi Manocha added a comment - @Roy: Could you please try again without the "using" statement, and let me know if it helps!
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        @Roy: Could you please confirm whether the issue is resolved with the above-mentioned solution. Thanks!

        Show
        saakshi.manocha Saakshi Manocha added a comment - @Roy: Could you please confirm whether the issue is resolved with the above-mentioned solution. Thanks!
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        Closing this issue for now, as no response from customer.
        Customer may reopen or create a new issue if problem reoccurs.

        Show
        saakshi.manocha Saakshi Manocha added a comment - Closing this issue for now, as no response from customer. Customer may reopen or create a new issue if problem reoccurs.

          People

          • Assignee:
            saakshi.manocha Saakshi Manocha
            Reporter:
            roy.jacobs roy.jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes