Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-426

Memory leak and application pool shutdown after Impersonation error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Critical
    • None
    • 1.3.4
    • None
    • None
    • Windows 2008 Server R2

    Description

      I'm using client version 1.3.4 and server version 2.1.1

      After several investigation and some help from a community member, here is the conclusions I have.

      My application pool running my web app inside the windows process (w3wp) has a memory leak of some kind only when I use couchbase. Without couchbase, the process is really stable (private memory around 400Mb, CPU rarely over 75, most of the times around 10). But when I do use couchbase, private memory doesn't stop growing up to 900Mb and the CPU is almost always over 75 (some from sigar and erl process, and some from w3wp).

      I have 3 web server in a farm and I installed couchbase server on all of them so the 3 web servers can make calls to all 3 couchbase server which in my case are the same servers. Maybe this is the problem but I tried using only 1 server for couchbase and 2 web servers with the exact same result so I guess this was not the problem.

      After a while, even with a really small load on the server (couple users), the application pool process throws an error.

      An unhandled exception occurred and the process was terminated. Exception: System.TypeInitializationException Message: The type initializer for 'Enyim.Caching.MemcachedClient' threw an exception. StackTrace: at Enyim.Caching.MemcachedClient.Finalize() InnerException: System.Web.HttpException Message: An error occurred while attempting to impersonate. Execution of this request cannot continue.

      I also receive a system warning in my event viewer saying this :

      A process serving application pool 'my application pool name' suffered a fatal communication error with the Windows Process Activation Service. The process id was '6944'. The data field contains the error number.

      The application pool process restarts at the same time.

      This happens every 20-30 minutes with a small load. But with a big load, this happens every 2-3 minutes and cause my application pool to stop. I can disable the rapid fail protection to be sure my application pool never stops but this doesn't solve my problem.

      My web app only does Get, Store, Remove calls. My CouchBaseClient is a static member of a singleton object and is used by the data access layer before and after calls to the database (if needed). When i go check the couchbase console, I see that the couchbase server does not have a lot of data. Around 2000 objects and only 3-4Mb under a big load. I also have around 500-900 Page faults (no minor no major).

      I do not use impersonation anywhere. I even tested it to see which user is running my app before calling a couchbaseclient method and the user is "IIS APPPool\My Application Name" which is normal with IIS7.

      Does Couchbase/memcached use impersonation somewhere and this user should have specific privileges ? The community member who helped me think that for some reason a problem could occur inside memcached during impersonation (why ? I dont know) and this would cause all TCP connection to close...which could lead to a severe exception because my application pool user doesn't have the rights to do this.

      Which privileges should I give to my user or why there's an impersonation problem inside MemcachedClient.Finalize.

      The community member also told me that the Finalize method should not be called if .Dipose() is called before. So I added the call to .Dipose() inside the Application_End method but that didn't change anything. Why and when would .Finalize be called in the first place ? With a better anwser to that, I could add .Dispose somewhere else.

      Finally, I added Log4Net and was able to log the error.

      Also, I did increase the number of ephemeral ports allowed on the server to 60 000. In fact, I answered Yes when ask during installation process and restart the server. I don't know where to check if it was done properly or not.

      In the attached log file, the error happens at 14h 13mins 31secs

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jmorris Jeff Morris
            AlainQ Alain Quirion
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty