Uploaded image for project: 'Couchbase Java Client'
  1. Couchbase Java Client
  2. JCBC-1181

Slow logs generation even network delay is less then 500ms

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • None
    • None
    • 1

    Description

      Testing End to end traceability with current sdk 2.6.0dp1 and vulcan build 5.5.0-2211. 

      I'am slowing down the network on sdk with less then 500ms still few slow logs are getting generated. 

       

      Here is the code snippet:

      NodeHost masterNode = cluster.getMasterNode();
      try {
      System.out.println("Ramping up for 30 seconds");
      Thread.sleep(30000);

      System.out.println("Introducing delay and running workload for 240 seconds");
      RemoteCommands.runCommand(masterNode.getSSH(), "tc qdisc add dev eth0 root netem delay 400ms 200ms loss 10% 25%").get();
      Thread.sleep(240000);

      System.out.println("Cancelling workload now");
      isCancelled = true;

      kvQueryWorkload.get();
      ftsWorkload.get();
      n1qlWorkload.get();

      executor.shutdown();
      } finally {
      System.out.println("Removing simulated delay");
      RemoteCommands.runCommand(masterNode.getSSH(), "tc qdisc del dev eth0 root netem").get();
      System.out.println("Removed simulated delay");
      }

       

      Actual Result: 

      [559.30 WARN] (ThresholdLogReporter warn:150) Operations over threshold: [{"top":[\{"local_id":"79A583CE4D121060/FFFFFFFF91B78D85","local_address":"192.168.3.193:55321","operation_id":"n1ql:f3ffbd97-4ae7-4ac0-943e-1954a8de5abb","dispatch_us":1281080,"remote_address":"172.23.123.176:8093","total_us":1281177}],"service":"n1ql","count":1}]
      number of Get operations so far 4017
      number of Get operations so far 4018
      number of Get operations so far 4019
      number of Get operations so far 4020
      number of Get operations so far 4021
      number of Get operations so far 4022
      number of Get operations so far 4023
      number of Get operations so far 4024
      number of Get operations so far 4025
      number of Get operations so far 4026
      number of Get operations so far 4027
      [560.74 WARN] (ThresholdLogReporter warn:150) Operations over threshold: [{"top":[\{"server_us":32,"local_id":"79A583CE4D121060/FFFFFFFFD9183AA6","local_address":"192.168.3.193:55268","operation_id":"get:0x4b56","decode_us":11,"dispatch_us":536021,"remote_address":"172.23.123.176:11210","total_us":536169}],"service":"kv","count":1}]
      number of Get operations so far 4028
      number of Get operations so far 4029
      number of Get operations so far 4030
      number of Get operations so far 4031
      Cancelling workload now
      number of Get operations so far 4032
      Total number of Get operations 4032
      Removing simulated delay
      [562.09 WARN] (ThresholdLogReporter warn:150) Operations over threshold: [{"top":[\{"server_us":23,"local_id":"79A583CE4D121060/FFFFFFFFD9183AA6","local_address":"192.168.3.193:55268","operation_id":"get:0x4b60","decode_us":15,"dispatch_us":634274,"remote_address":"172.23.123.176:11210","total_us":634427}],"service":"kv","count":1},{"top":[\{"local_id":"79A583CE4D121060/FFFFFFFF91B78D85","local_address":"192.168.3.193:55321","operation_id":"n1ql:ba343141-5b86-4d49-9f92-e56de08698dd","dispatch_us":1147966,"remote_address":"172.23.123.176:8093","total_us":1148036}],"service":"n1ql","count":1}]
      Removed simulated delay
      Downloading server logs
      [581.33 INFO] (Endpoint info:120) [/172.23.121.38:8094][SearchEndpoint]: Got notified from Channel as inactive, attempting reconnect.
      [581.38 INFO] (Endpoint info:120) [/172.23.121.38:8094][SearchEndpoint]: Got notified from Channel as inactive, attempting reconnect.
      Finished downloading server logs

      Is this expected result or slow logs should not be generated at all ? 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            akshata.trivedi Akshata Trivedi [X] (Inactive)
            akshata.trivedi Akshata Trivedi [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty