Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-644

ops/sec is not restored to the expected value even when warmup completed (restart 3/4 nodes)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0 Beta 2
    • 2.0 Beta
    • library
    • None

    Description

      http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/97/consoleFull

      Running SvcRestart-HYBRID: 2014-09-17 01:47:02
      service/restore_delay=5
      workload=HybridWorkloadGroup
      service/name=SYSV_SERVICE
      service/action=kill
      service/count=3
      testcase=ServiceFailureScenario
      Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-17-14\031622\c8a06f8889f4ee9408205f024dabd0fe

      To re-run the test, copy/paste the following into the shell.
      You may also copy/paste (except the first line) into an argfile
      8<----------------------------------------
      ./stester \
      --service-restore_delay 5 --workload HybridWorkloadGroup
      \
      --service-name SYSV_SERVICE --service-action kill --service-count
      \
      3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
      \
      C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
      \
      --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
      \
      --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
      \
      --cluster_ssh-username=root --cluster_ssh-password=couchbase
      ---------------------------------------->8
      [1587.82 WARN] (Drivers getDriver:76) 'rexec' is now mapped to local execution only. Use RemoteExecutingDriver for remote execution
      [1587.91 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: Execute '/etc/init.d/couchbase-server stop' on 3 nodes waiting 1 seconds between each invocation sleep for 5 seconds and restore the nodes. Restoring is peformed by: re-active the services by invoking /etc/init.d/couchbase-server starton each of the nodes. Rebound for 90 seconds.

      ..............

      [1687.17 INFO] (SDKD log:137)  "ViewName": "test_view"
      [1687.17 INFO] (SDKD log:137) } 
      [1717.17 INFO] (Scenario run:86) Starting CHANGE phase
      [1717.17 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.135
      [1721.05 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409 
      [1722.70 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@6dc8f3cd
      [1723.71 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.136
      [1726.96 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@4271c5bc
      [1727.98 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.3.206
      [1731.62 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1bb1deea
      [1732.63 INFO] (Scenario sleepAndDo:109) Sleeping 5 seconds
      [1737.63 INFO] (Scenario executeChange:134) Turning on services for nodes
      [1737.63 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.135
      [1739.48 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@ab245dc
      [1740.53 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.136
      [1742.20 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@5c538b31
      [1743.25 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.3.206
      [1744.92 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@66de04cd
      [1745.93 INFO] (Scenario run:97) CHANGE phase done
      [1745.93 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
      [1751.36 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
      [1835.93 INFO] (Scenario run:105) REBOUND done. Will collect results
      [1835.93 DEBUG] (Handle sendMessageAsync:183) > CANCEL@278.597

      please see report c8a06f8889f4ee9408205f024dabd0fe-MC.html

      warmup should not take more than a minute(it should be even less 20 seconds, only 1600 items are in the bucket) but within 90 seconds ops/sec statistic is extremely slow.

      in client logs I see a large number of exceptions:
      2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 | 10.3.121.135:11210 | [0, 2] - 8f050465-bd1e-4798-9e75-b7dced55a859 - Disposed: False
      2014-09-17 01:49:19,230 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3692REP3692REP3692REP3692REP3692 using 313 on 10.3.121.135:11210
      2014-09-17 01:49:19,230 [27] WARN Couchbase.IO.ConnectionBase - Opcode=Set | Key=SimpleKeyREP3692REP3692REP3692REP3692REP3692 | Host=10.3.121.135:11210
      System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
      at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
      at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
      — End of inner exception stack trace —
      at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
      at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
      2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 on 10.3.121.135:11210 - 8f050465-bd1e-4798-9e75-b7dced55a859
      2014-09-17 01:49:19,231 [27] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.134:11210
      2014-09-17 01:49:19,231 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 4be40fbd-8d82-49de-931f-598e6276e7cd | 10.3.121.134:11210 | [0, 2] - b7812f0a-380b-41a4-92f4-ac1db0e0f3a0 - Disposed: False
      2014-09-17 01:49:19,231 [27] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.134:11210
      2014-09-17 01:49:19,228 [31] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 457 for key SimpleKeyREP3796REP3796REP3796REP3796REP3796 - rev4
      2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP3796REP3796REP3796REP3796REP3796 using server 10.3.121.135:11210
      2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: e63556d1-caf1-44fc-8f81-11a670a45a9b | 10.3.121.135:11210 | [0, 2] - 3cd6215b-e17c-493c-bf94-885b7cafaf1a - Disposed: False
      2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3796REP3796REP3796REP3796REP3796 using 457 on 10.3.121.135:11210
      2014-09-17 01:49:19,231 [31] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP3796REP3796REP3796REP3796REP3796 | Host=10.3.121.135:11210
      System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
      at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
      at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
      — End of inner exception stack trace —
      at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
      at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
      2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: e63556d1-caf1-44fc-8f81-11a670a45a9b on 10.3.121.135:11210 - 3cd6215b-e17c-493c-bf94-885b7cafaf1a
      2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.136:11210
      2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 092717c9-ef47-41df-8a81-839302b58bde | 10.3.121.136:11210 | [0, 2] - df9f7db5-1e94-4d4e-ac0d-441ff1b9c698 - Disposed: False
      2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.136:11210

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jmorris Jeff Morris
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty