Uploaded image for project: 'Couchbase Java Client'
  1. Couchbase Java Client
  2. JCBC-189

Views having odd timeout issues on some clusters

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: .backlog1.x
    • Component/s: Core
    • Security Level: Public
    • Labels:
      None

      Description

      We're seeing really strange timeout issues which seem to affect only this specific cluster and only in terms of views. We've re-installed this cluster time and time again, and we've had similar configurations run successfully as well.

      More details in comment..

      1. failover-debug.args
        0.4 kB
        Deepti Dawar
      2. fo-dbg.log.txt
        20 kB
        Mark Nunberg
      3. vm-4nodes-2.ini
        0.2 kB
        Deepti Dawar

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          deeptida Deepti Dawar added a comment -

          JCBC-189 was primarily related to the timeouts and environmental issues. Now, after some progress in this, its figured out that these env related issues have been ironed out with the latest set up of the client and server VMs.
          Also, the issue that remains now is very specific to the cluster and the client which can be tracked as part of a new JCBC issue - JCBC-333

          Show
          deeptida Deepti Dawar added a comment - JCBC-189 was primarily related to the timeouts and environmental issues. Now, after some progress in this, its figured out that these env related issues have been ironed out with the latest set up of the client and server VMs. Also, the issue that remains now is very specific to the cluster and the client which can be tracked as part of a new JCBC issue - JCBC-333
          Hide
          daschl Michael Nitschinger added a comment -

          This is "kinda" blocker.

          Show
          daschl Michael Nitschinger added a comment - This is "kinda" blocker.
          Hide
          mnunberg Mark Nunberg added a comment -

          So this bug isn't such an "unknown" anymore, and has exposed itself in 1.1.0 as well as 1.1.1 and isn't limited to particular clusters - (it just seems that some clusters are more likely than others to trigger this bug).

          However this still needs a lot of care and analysis

          Show
          mnunberg Mark Nunberg added a comment - So this bug isn't such an "unknown" anymore, and has exposed itself in 1.1.0 as well as 1.1.1 and isn't limited to particular clusters - (it just seems that some clusters are more likely than others to trigger this bug). However this still needs a lot of care and analysis
          Hide
          daschl Michael Nitschinger added a comment -

          According to the posted logs, it looks like debugging was not turned on. Can you please run this again with debugging turned on? To get the full logs to STDOUT, use this before initializing the CouchbaseClient in The App:

                // Tell spy to use the SunLogger
                 Properties systemProperties = System.getProperties();
                 systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");
                 System.setProperties(systemProperties);

                 Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);

                 //get the top Logger
                 Logger topLogger = java.util.logging.Logger.getLogger("");

                 // Handler for console (reuse it if it already exists)
                 Handler consoleHandler = null;
                 //see if there is already a console handler
                 for (Handler handler : topLogger.getHandlers()) {
                     if (handler instanceof ConsoleHandler)

          {                //found the console handler                consoleHandler = handler;                break;            }

                 }

                 if (consoleHandler == null)

          {            //there was no console handler found, create a new one            consoleHandler = new ConsoleHandler();            topLogger.addHandler(consoleHandler);        }

                 //set the console handler to fine:
                 consoleHandler.setLevel(java.util.logging.Level.FINEST);

          Would be great if we can get all the output so we can investigate where the timeouts come from. I'm sure with a debug log it will be much easier. Thanks!

          Show
          daschl Michael Nitschinger added a comment - According to the posted logs, it looks like debugging was not turned on. Can you please run this again with debugging turned on? To get the full logs to STDOUT, use this before initializing the CouchbaseClient in The App:       // Tell spy to use the SunLogger        Properties systemProperties = System.getProperties();        systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");        System.setProperties(systemProperties);        Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);        //get the top Logger        Logger topLogger = java.util.logging.Logger.getLogger("");        // Handler for console (reuse it if it already exists)        Handler consoleHandler = null;        //see if there is already a console handler        for (Handler handler : topLogger.getHandlers()) {            if (handler instanceof ConsoleHandler) {                //found the console handler                consoleHandler = handler;                break;            }        }        if (consoleHandler == null) {            //there was no console handler found, create a new one            consoleHandler = new ConsoleHandler();            topLogger.addHandler(consoleHandler);        }        //set the console handler to fine:        consoleHandler.setLevel(java.util.logging.Level.FINEST); Would be great if we can get all the output so we can investigate where the timeouts come from. I'm sure with a debug log it will be much easier. Thanks!
          Hide
          daschl Michael Nitschinger added a comment -

          Please pass me the script as commented and then (or if you can't) please assign it back to me! Thanks

          Show
          daschl Michael Nitschinger added a comment - Please pass me the script as commented and then (or if you can't) please assign it back to me! Thanks
          Hide
          ingenthr Matt Ingenthron added a comment -

          Michael: found out today that this is a large issue for SDKQE. Can you have a quick look at this in the next day? You may find the underlying issue.

          Show
          ingenthr Matt Ingenthron added a comment - Michael: found out today that this is a large issue for SDKQE. Can you have a quick look at this in the next day? You may find the underlying issue.
          Hide
          deeptida Deepti Dawar added a comment - - edited

          Its working fine for server deployed on localhost.

          Show
          deeptida Deepti Dawar added a comment - - edited Its working fine for server deployed on localhost.
          Hide
          deeptida Deepti Dawar added a comment -

          This issue persists when the view retrieval is ran from standalone java program as well.
          I tried this on both the VMs - 10.3.3.203, 10.3.3.209.

          Show
          deeptida Deepti Dawar added a comment - This issue persists when the view retrieval is ran from standalone java program as well. I tried this on both the VMs - 10.3.3.203, 10.3.3.209.
          Hide
          mnunberg Mark Nunberg added a comment -

          Just try to do this with a simple java script (i.e. connect and query the view)

          Show
          mnunberg Mark Nunberg added a comment - Just try to do this with a simple java script (i.e. connect and query the view)

            People

            • Assignee:
              daschl Michael Nitschinger
              Reporter:
              mnunberg Mark Nunberg
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes