Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-340

System.InvalidOperationException an asynchronous module or handler completed while an asynchronous operation was still pending

    Details

    • Type: Task
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: backlog-1.0
    • Component/s: library
    • Labels:
      None
    • Environment:
      ASP.NET Session Provider
    • Sprint:
      Sprint 2 - Feb 5-26 2014, Sprint 4

      Description

      I get the following error randomly on my production Couchbase server:

      "System.InvalidOperationException
      an asynchronous module or handler completed while an asynchronous operation was still pending"

      I am using the latest Couchbase server and the latest version of the .Net SDK. When I run locally I do not see this error at all.

      On production I have a two node Couchbase client. The config section is the following:

      <couchbase>
      <servers bucket="Staging" bucketPassword="">
      <add uri="http://COUCH01.mobitorhost00.local:8091/pools" />
      </servers>
      </couchbase>
      <enyim.com>
      <memcached protocol="Binary">
      <servers>
      <add address="COUCH01.mobitorhost00.local" port="11211" />
      </servers>
      <socketPool minPoolSize="10" maxPoolSize="100" connectionTimeout="00:01:00" deadTimeout="00:01:00"/>
      </memcached>
      </enyim.com>

      When I switch to the Memcache provider instead the error does not appear.

      Thanks for your help. This is really a show-stopper for us.

      Here is a stack trace:

      Storex.Site.dll!Storex.Site.MvcApplication.Application_Error() Line 53 C#
      [Native to Managed Transition]
      System.Web.dll!System.Web.Util.ArglessEventHandlerProxy.Callback(object sender, System.EventArgs e) Unknown
      System.Web.dll!System.Web.HttpApplication.RaiseOnError() Unknown
      System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error) Unknown
      System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb) Unknown
      System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context) Unknown
      System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
      System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
      [Native to Managed Transition]
      [Managed to Native Transition]
      System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
      System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
      [AppDomain Transition]

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        tporadowski Tomasz Poradowski added a comment -

        I could not reproduce exactly the same problem in this test scenario as after finding the workaround I didn't spend any more time on it. I've just tested your fix to NCBC-337 (http://review.couchbase.org/#/c/31007/ applied by hand to current master branch of couchbase-net-client) and was hoping to finally be able to reproduce exception reported originally in this ticket - but unfortunately the same error persists ("view mapped to a dead node").

        Show
        tporadowski Tomasz Poradowski added a comment - I could not reproduce exactly the same problem in this test scenario as after finding the workaround I didn't spend any more time on it. I've just tested your fix to NCBC-337 ( http://review.couchbase.org/#/c/31007/ applied by hand to current master branch of couchbase-net-client) and was hoping to finally be able to reproduce exception reported originally in this ticket - but unfortunately the same error persists ("view mapped to a dead node").
        Hide
        tporadowski Tomasz Poradowski added a comment -

        Ok, now I finally know how to reproduce this very issue - in my sample project attached to this ticket please modify NCBC340/Controllers/PostController.cs as follows:

        public class PostController : Controller
        {
        //
        // GET: /Post/
        private PostRepository repository = new PostRepository();
        private CommentRepository comments = new CommentRepository();

        public ActionResult Show(string id)

        { var post = repository.Get(id); post.Comments = comments.GetAllByPost(id); return View(post); }

        }

        and like that it fails for the very first time when you go to details of a test "post" shown on main page. Therefore it seems that this asynchronous call in DefaultHttpClient's constructor does not finish on time. Subsequent calls to view details succeeded, so it's likely per design document or per view.

        Show
        tporadowski Tomasz Poradowski added a comment - Ok, now I finally know how to reproduce this very issue - in my sample project attached to this ticket please modify NCBC340/Controllers/PostController.cs as follows: public class PostController : Controller { // // GET: /Post/ private PostRepository repository = new PostRepository(); private CommentRepository comments = new CommentRepository(); public ActionResult Show(string id) { var post = repository.Get(id); post.Comments = comments.GetAllByPost(id); return View(post); } } and like that it fails for the very first time when you go to details of a test "post" shown on main page. Therefore it seems that this asynchronous call in DefaultHttpClient's constructor does not finish on time. Subsequent calls to view details succeeded, so it's likely per design document or per view.
        Hide
        tporadowski Tomasz Poradowski added a comment - - edited

        Though I'm not sure what impact this may have on performance - another workaround for this issue is to add following to web.config:

        <couchbase>
        ...
        <httpClient initializeConnection="false" />
        </couchbase>

        This setting causes the problematic asynchronous call in DefaultHttpClient's constructor to be skipped.

        Show
        tporadowski Tomasz Poradowski added a comment - - edited Though I'm not sure what impact this may have on performance - another workaround for this issue is to add following to web.config: <couchbase> ... <httpClient initializeConnection="false" /> </couchbase> This setting causes the problematic asynchronous call in DefaultHttpClient's constructor to be skipped.
        Hide
        jmorris Jeff Morris added a comment - - edited

        Tomasz -

        This is great, we can test the performance between setting false/true.

        >>I've just tested your fix to NCBC-337 (http://review.couchbase.org/#/c/31007/ applied by hand to current master branch of couchbase-net-client) and was hoping to finally be able to reproduce exception reported originally in this ticket - but unfortunately the same error persists ("view mapped to a dead node").

        This should be resolved with that commit, so if your still seeing it then it's likely a bigger problem. Could you describe your environment (cluster size, etc)?

        Thanks,

        -Jeff

        Show
        jmorris Jeff Morris added a comment - - edited Tomasz - This is great, we can test the performance between setting false/true. >>I've just tested your fix to NCBC-337 ( http://review.couchbase.org/#/c/31007/ applied by hand to current master branch of couchbase-net-client) and was hoping to finally be able to reproduce exception reported originally in this ticket - but unfortunately the same error persists ("view mapped to a dead node"). This should be resolved with that commit, so if your still seeing it then it's likely a bigger problem. Could you describe your environment (cluster size, etc)? Thanks, -Jeff
        Hide
        tporadowski Tomasz Poradowski added a comment -

        @Jeff - I assume your fix to NCBC-337 was fine, but that issue was also hit (though indirectly) by the same root cause as NCBC-340. My test environment is currently a single node running Couchbase Enterprise 2.2.0 on Win7 Pro, using 2GB RAM.

        Show
        tporadowski Tomasz Poradowski added a comment - @Jeff - I assume your fix to NCBC-337 was fine, but that issue was also hit (though indirectly) by the same root cause as NCBC-340 . My test environment is currently a single node running Couchbase Enterprise 2.2.0 on Win7 Pro, using 2GB RAM.

          People

          • Assignee:
            jmorris Jeff Morris
            Reporter:
            jmorris Jeff Morris
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes