Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-284

after some time with lots of connections, client gets stuck requiring a full IIS reset

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: 1.2.6, 1.2.7
    • Fix Version/s: None
    • Component/s: library
    • Labels:
      None
    • Environment:
      Couchbase Server 2.1

      Description

      Bug reporter can reproduce this issue in production, but has not been able to reproduce it in test environment. From the bug reporter:

      "once we get lots of connections to the live server with a single instance it eventually totally gets stuck, and it is not clear where or why. A full IIS reset is needed.

      "I am going to try to load test our site on our staging server to see if we can get it to fail outside of the production server. I suspect the issue can be replicated if you add a large load to a web app that is making extensive use of couchbase for caching. In our case we have to Couchbase clients that we use; one is to the memcached buckets we use purely for caching, and the second is to a couchbase bucket that currently only stores session data. Not sure that makes any difference, but we do use two clients per request. And since we use it for session data that one gets a good load on it but the cache one is hit many times per request.

      And after some further investigation:

      "However the good news is that it behaved differently to how it has in the past. Usually as soon as we turn it on and recycle the application pool, the site falls over and becomes non-responsive immediately, and the CPU gets pegged at 100% in the IIS process. In this case however we actually got a page load to come up pretty quickly after we turned it on, which for a second made me think it was working. But alas more page loads after that started to load up and the site ground to a halt. Oddly the CPU was not pegged at 100% like it normally does, but it was definitely way higher than normal, in the 80-95% range I think (normal CPU sits around 10-15% max once the site has come up and settled in).

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        ingenthr Matt Ingenthron added a comment -

        Saakshi: I know you're trying to reproduce this. Can you drop your current status in here, including any references to tools we've built or modified? Let's track status on this investigation on this NCBC.

        Show
        ingenthr Matt Ingenthron added a comment - Saakshi: I know you're trying to reproduce this. Can you drop your current status in here, including any references to tools we've built or modified? Let's track status on this investigation on this NCBC.
        Hide
        saakshi.manocha Saakshi Manocha added a comment - - edited

        Hi Kendall,

        1. Please note that constructing the client should not be done per operation, but rather per app domain. Could you please try to use one Couchbase client per request and try if that helps.

        2. Since the problems are seen with increase in number of threads, could you try once by gradually increasing the time out in the sessionState config section as below:

        <sessionState customProvider="Couchbase" mode="Custom" timeout="500">
        <providers>
        <add name="Couchbase" type="Couchbase.AspNet.SessionState.CouchbaseSessionStateProvider, Couchbase.AspNet" section="couchbaseSessionState" />
        </providers>
        </sessionState>

        Show
        saakshi.manocha Saakshi Manocha added a comment - - edited Hi Kendall, 1. Please note that constructing the client should not be done per operation, but rather per app domain. Could you please try to use one Couchbase client per request and try if that helps. 2. Since the problems are seen with increase in number of threads, could you try once by gradually increasing the time out in the sessionState config section as below: <sessionState customProvider="Couchbase" mode="Custom" timeout="500"> <providers> <add name="Couchbase" type="Couchbase.AspNet.SessionState.CouchbaseSessionStateProvider, Couchbase.AspNet" section="couchbaseSessionState" /> </providers> </sessionState>
        Hide
        kendallb Kendall Bennett added a comment -

        1. The demo code shows creating a single, static instance of the Couchbase client, so that all threads share the same instance and internally Couchbase uses the Enyim library that caches HTTP connections in a pool and reuses them. That is basically one client per app domain, and that is what does NOT work. Our site hangs up if we try to do that.

        To work around the issue we create one client per HTTP application instance, not per request. So we are reusing instances over and over for subsequent requests, but there is only one per HTTP instance that IIS starts. We average about 40 or so instances under load. This means no thread safety is needed because IIS only handles one request per instance at a time, and is the reason we made this change. Which indicates that the current library has some kind of threading issue that causes our site to lock up and get stuck if we try to use the single instance mode where all threads (40 at a time) are coming into the library at once.

        2. I am not sure why the session state config will make any difference? I don't believe it is a timeout problem in the sessions or we should see a session timeout error getting logged?

        Show
        kendallb Kendall Bennett added a comment - 1. The demo code shows creating a single, static instance of the Couchbase client, so that all threads share the same instance and internally Couchbase uses the Enyim library that caches HTTP connections in a pool and reuses them. That is basically one client per app domain, and that is what does NOT work. Our site hangs up if we try to do that. To work around the issue we create one client per HTTP application instance, not per request. So we are reusing instances over and over for subsequent requests, but there is only one per HTTP instance that IIS starts. We average about 40 or so instances under load. This means no thread safety is needed because IIS only handles one request per instance at a time, and is the reason we made this change. Which indicates that the current library has some kind of threading issue that causes our site to lock up and get stuck if we try to use the single instance mode where all threads (40 at a time) are coming into the library at once. 2. I am not sure why the session state config will make any difference? I don't believe it is a timeout problem in the sessions or we should see a session timeout error getting logged?
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        Is it possible for you to share your sample code or code snippet so that I can have a look?

        Show
        saakshi.manocha Saakshi Manocha added a comment - Is it possible for you to share your sample code or code snippet so that I can have a look?
        Hide
        kendallb Kendall Bennett added a comment -

        Sure, I can do that. I am busy the rest of this week, but I can try next week. What specifically are you looking for? Something that shows how we set up per-application instance? Or how we are using it in single instance mode? We have it set up so we can change from per-app instance (which works best for us) and single instance mode with a configuration file change.

        Show
        kendallb Kendall Bennett added a comment - Sure, I can do that. I am busy the rest of this week, but I can try next week. What specifically are you looking for? Something that shows how we set up per-application instance? Or how we are using it in single instance mode? We have it set up so we can change from per-app instance (which works best for us) and single instance mode with a configuration file change.
        Hide
        saakshi.manocha Saakshi Manocha added a comment -

        Could you please share some code snippet or a simple test case that demonstrates the error you are getting. I won't be able to debug your entire program as it might be dependent on any of the customer-specific binaries, but lets target a small code which replicates the behaviour at your end and i will try to debug and analyse the error.

        Show
        saakshi.manocha Saakshi Manocha added a comment - Could you please share some code snippet or a simple test case that demonstrates the error you are getting. I won't be able to debug your entire program as it might be dependent on any of the customer-specific binaries, but lets target a small code which replicates the behaviour at your end and i will try to debug and analyse the error.
        Hide
        jmorris Jeff Morris added a comment -

        Closing these issues because they are for versions of the SDK that are no longer supported.

        Show
        jmorris Jeff Morris added a comment - Closing these issues because they are for versions of the SDK that are no longer supported.

          People

          • Assignee:
            saakshi.manocha Saakshi Manocha
            Reporter:
            ingenthr Matt Ingenthron
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes