Uploaded image for project: 'Couchbase Java Client'
  1. Couchbase Java Client
  2. JCBC-189

Views having odd timeout issues on some clusters

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: .backlog1.x
    • Component/s: Core
    • Security Level: Public
    • Labels:
      None

      Description

      We're seeing really strange timeout issues which seem to affect only this specific cluster and only in terms of views. We've re-installed this cluster time and time again, and we've had similar configurations run successfully as well.

      More details in comment..

      1. failover-debug.args
        0.4 kB
        Deepti Dawar
      2. fo-dbg.log.txt
        20 kB
        Mark Nunberg
      3. vm-4nodes-2.ini
        0.2 kB
        Deepti Dawar

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          mnunberg Mark Nunberg created issue -
          Hide
          mnunberg Mark Nunberg added a comment -

          Just try to do this with a simple java script (i.e. connect and query the view)

          Show
          mnunberg Mark Nunberg added a comment - Just try to do this with a simple java script (i.e. connect and query the view)
          mnunberg Mark Nunberg made changes -
          Field Original Value New Value
          Assignee Michael Nitschinger [ daschl ] Deepti Dawar [ deeptida ]
          mnunberg Mark Nunberg made changes -
          Attachment fo-dbg.log.txt [ 16032 ]
          Hide
          deeptida Deepti Dawar added a comment -

          This issue persists when the view retrieval is ran from standalone java program as well.
          I tried this on both the VMs - 10.3.3.203, 10.3.3.209.

          Show
          deeptida Deepti Dawar added a comment - This issue persists when the view retrieval is ran from standalone java program as well. I tried this on both the VMs - 10.3.3.203, 10.3.3.209.
          Hide
          deeptida Deepti Dawar added a comment - - edited

          Its working fine for server deployed on localhost.

          Show
          deeptida Deepti Dawar added a comment - - edited Its working fine for server deployed on localhost.
          deeptida Deepti Dawar made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          mnunberg Mark Nunberg made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          mnunberg Mark Nunberg made changes -
          Assignee Deepti Dawar [ deeptida ] Michael Nitschinger [ daschl ]
          deeptida Deepti Dawar made changes -
          Link This issue blocks JCBC-176 [ JCBC-176 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.1 [ 10430 ]
          Hide
          ingenthr Matt Ingenthron added a comment -

          Michael: found out today that this is a large issue for SDKQE. Can you have a quick look at this in the next day? You may find the underlying issue.

          Show
          ingenthr Matt Ingenthron added a comment - Michael: found out today that this is a large issue for SDKQE. Can you have a quick look at this in the next day? You may find the underlying issue.
          Hide
          daschl Michael Nitschinger added a comment -

          Please pass me the script as commented and then (or if you can't) please assign it back to me! Thanks

          Show
          daschl Michael Nitschinger added a comment - Please pass me the script as commented and then (or if you can't) please assign it back to me! Thanks
          daschl Michael Nitschinger made changes -
          Assignee Michael Nitschinger [ daschl ] Deepti Dawar [ deeptida ]
          deeptida Deepti Dawar made changes -
          Attachment vm-4nodes-2.ini [ 16111 ]
          Attachment failover-debug.args [ 16112 ]
          deeptida Deepti Dawar made changes -
          Assignee Deepti Dawar [ deeptida ] Michael Nitschinger [ daschl ]
          mnunberg Mark Nunberg made changes -
          Summary Views having odd timeout issues on a specific cluster Views having odd timeout issues on some clusters
          Hide
          daschl Michael Nitschinger added a comment -

          According to the posted logs, it looks like debugging was not turned on. Can you please run this again with debugging turned on? To get the full logs to STDOUT, use this before initializing the CouchbaseClient in The App:

                // Tell spy to use the SunLogger
                 Properties systemProperties = System.getProperties();
                 systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");
                 System.setProperties(systemProperties);

                 Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);

                 //get the top Logger
                 Logger topLogger = java.util.logging.Logger.getLogger("");

                 // Handler for console (reuse it if it already exists)
                 Handler consoleHandler = null;
                 //see if there is already a console handler
                 for (Handler handler : topLogger.getHandlers()) {
                     if (handler instanceof ConsoleHandler)

          {                //found the console handler                consoleHandler = handler;                break;            }

                 }

                 if (consoleHandler == null)

          {            //there was no console handler found, create a new one            consoleHandler = new ConsoleHandler();            topLogger.addHandler(consoleHandler);        }

                 //set the console handler to fine:
                 consoleHandler.setLevel(java.util.logging.Level.FINEST);

          Would be great if we can get all the output so we can investigate where the timeouts come from. I'm sure with a debug log it will be much easier. Thanks!

          Show
          daschl Michael Nitschinger added a comment - According to the posted logs, it looks like debugging was not turned on. Can you please run this again with debugging turned on? To get the full logs to STDOUT, use this before initializing the CouchbaseClient in The App:       // Tell spy to use the SunLogger        Properties systemProperties = System.getProperties();        systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");        System.setProperties(systemProperties);        Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);        //get the top Logger        Logger topLogger = java.util.logging.Logger.getLogger("");        // Handler for console (reuse it if it already exists)        Handler consoleHandler = null;        //see if there is already a console handler        for (Handler handler : topLogger.getHandlers()) {            if (handler instanceof ConsoleHandler) {                //found the console handler                consoleHandler = handler;                break;            }        }        if (consoleHandler == null) {            //there was no console handler found, create a new one            consoleHandler = new ConsoleHandler();            topLogger.addHandler(consoleHandler);        }        //set the console handler to fine:        consoleHandler.setLevel(java.util.logging.Level.FINEST); Would be great if we can get all the output so we can investigate where the timeouts come from. I'm sure with a debug log it will be much easier. Thanks!
          daschl Michael Nitschinger made changes -
          Assignee Michael Nitschinger [ daschl ] Mark Nunberg [ mnunberg ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.2 [ 10480 ]
          Fix Version/s 1.1.1 [ 10430 ]
          Hide
          mnunberg Mark Nunberg added a comment -

          So this bug isn't such an "unknown" anymore, and has exposed itself in 1.1.0 as well as 1.1.1 and isn't limited to particular clusters - (it just seems that some clusters are more likely than others to trigger this bug).

          However this still needs a lot of care and analysis

          Show
          mnunberg Mark Nunberg added a comment - So this bug isn't such an "unknown" anymore, and has exposed itself in 1.1.0 as well as 1.1.1 and isn't limited to particular clusters - (it just seems that some clusters are more likely than others to trigger this bug). However this still needs a lot of care and analysis
          daschl Michael Nitschinger made changes -
          Link This issue depends on MB-7661 [ MB-7661 ]
          Hide
          daschl Michael Nitschinger added a comment -

          This is "kinda" blocker.

          Show
          daschl Michael Nitschinger added a comment - This is "kinda" blocker.
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.3 [ 10496 ]
          Fix Version/s 1.1.2 [ 10480 ]
          ingenthr Matt Ingenthron made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.4 [ 10514 ]
          Fix Version/s 1.1.3 [ 10496 ]
          daschl Michael Nitschinger made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.5 [ 10515 ]
          Fix Version/s 1.1.4 [ 10514 ]
          daschl Michael Nitschinger made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.6 [ 10531 ]
          Fix Version/s 1.1.5 [ 10515 ]
          mnunberg Mark Nunberg made changes -
          Assignee Mark Nunberg [ mnunberg ] Michael Nitschinger [ daschl ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.7 [ 10532 ]
          Fix Version/s 1.1.6 [ 10531 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.1.8 [ 10628 ]
          Fix Version/s 1.1.7 [ 10532 ]
          deeptida Deepti Dawar made changes -
          Link This issue blocks SDKQE-322 [ SDKQE-322 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.2 [ 10383 ]
          Fix Version/s 1.1.8 [ 10628 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s 1.2 [ 10383 ]
          daschl Michael Nitschinger made changes -
          Fix Version/s .next [ 10495 ]
          Hide
          deeptida Deepti Dawar added a comment -

          JCBC-189 was primarily related to the timeouts and environmental issues. Now, after some progress in this, its figured out that these env related issues have been ironed out with the latest set up of the client and server VMs.
          Also, the issue that remains now is very specific to the cluster and the client which can be tracked as part of a new JCBC issue - JCBC-333

          Show
          deeptida Deepti Dawar added a comment - JCBC-189 was primarily related to the timeouts and environmental issues. Now, after some progress in this, its figured out that these env related issues have been ironed out with the latest set up of the client and server VMs. Also, the issue that remains now is very specific to the cluster and the client which can be tracked as part of a new JCBC issue - JCBC-333
          deeptida Deepti Dawar made changes -
          Status In Progress [ 3 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          ingenthr Matt Ingenthron made changes -
          Workflow jira [ 23267 ] Couchbase SDK Workflow [ 38477 ]

            People

            • Assignee:
              daschl Michael Nitschinger
              Reporter:
              mnunberg Mark Nunberg
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes