Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0 Beta
-
None
Description
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/97/consoleFull
Running SvcRestart-HYBRID: 2014-09-17 01:47:02
service/restore_delay=5
workload=HybridWorkloadGroup
service/name=SYSV_SERVICE
service/action=kill
service/count=3
testcase=ServiceFailureScenario
Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-17-14\031622\c8a06f8889f4ee9408205f024dabd0fe
To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
--service-restore_delay 5 --workload HybridWorkloadGroup
\
--service-name SYSV_SERVICE --service-action kill --service-count
\
3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
\
C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
\
--cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\
--cluster_node=10.3.121.136 --cluster_node=10.3.3.206
\
--cluster_ssh-username=root --cluster_ssh-password=couchbase
---------------------------------------->8
[1587.82 WARN] (Drivers getDriver:76) [33m'rexec' is now mapped to local execution only. Use RemoteExecutingDriver for remote execution[0;39m
[1587.91 INFO] (RunContext run:102) [39mRamp for 30 seconds. Cluster modification: Execute '/etc/init.d/couchbase-server stop' on 3 nodes waiting 1 seconds between each invocation sleep for 5 seconds and restore the nodes. Restoring is peformed by: re-active the services by invoking /etc/init.d/couchbase-server starton each of the nodes. Rebound for 90 seconds.[0;39m
..............
[1687.17 INFO] (SDKD log:137) [39m "ViewName": "test_view"[0;39m
[1687.17 INFO] (SDKD log:137) [39m} [0;39m
[1717.17 INFO] (Scenario run:86) [39mStarting CHANGE phase[0;39m
[1717.17 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.121.135[0;39m
[1721.05 INFO] (SDKD log:137) [39m[Sdkd.Result|Warn] Unrecognized return code 409 [0;39m
[1722.70 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@6dc8f3cd[0;39m
[1723.71 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.121.136[0;39m
[1726.96 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@4271c5bc[0;39m
[1727.98 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.3.206[0;39m
[1731.62 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@1bb1deea[0;39m
[1732.63 INFO] (Scenario sleepAndDo:109) [39mSleeping 5 seconds[0;39m
[1737.63 INFO] (Scenario executeChange:134) [39mTurning on services for nodes[0;39m
[1737.63 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.121.135[0;39m
[1739.48 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@ab245dc[0;39m
[1740.53 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.121.136[0;39m
[1742.20 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@5c538b31[0;39m
[1743.25 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.3.206[0;39m
[1744.92 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@66de04cd[0;39m
[1745.93 INFO] (Scenario run:97) [39mCHANGE phase done[0;39m
[1745.93 INFO] (Scenario run:103) [39mStarting REBOUND for 90 seconds[0;39m
[1751.36 INFO] (SDKD log:137) [39m[Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268[0;39m
[1835.93 INFO] (Scenario run:105) [39mREBOUND done. Will collect results[0;39m
[1835.93 DEBUG] (Handle sendMessageAsync:183) [34m> CANCEL@278.597[0;39m
please see report c8a06f8889f4ee9408205f024dabd0fe-MC.html
warmup should not take more than a minute(it should be even less 20 seconds, only 1600 items are in the bucket) but within 90 seconds ops/sec statistic is extremely slow.
in client logs I see a large number of exceptions:
2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 | 10.3.121.135:11210 | [0, 2] - 8f050465-bd1e-4798-9e75-b7dced55a859 - Disposed: False
2014-09-17 01:49:19,230 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3692REP3692REP3692REP3692REP3692 using 313 on 10.3.121.135:11210
2014-09-17 01:49:19,230 [27] WARN Couchbase.IO.ConnectionBase - Opcode=Set | Key=SimpleKeyREP3692REP3692REP3692REP3692REP3692 | Host=10.3.121.135:11210
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 on 10.3.121.135:11210 - 8f050465-bd1e-4798-9e75-b7dced55a859
2014-09-17 01:49:19,231 [27] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.134:11210
2014-09-17 01:49:19,231 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 4be40fbd-8d82-49de-931f-598e6276e7cd | 10.3.121.134:11210 | [0, 2] - b7812f0a-380b-41a4-92f4-ac1db0e0f3a0 - Disposed: False
2014-09-17 01:49:19,231 [27] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.134:11210
2014-09-17 01:49:19,228 [31] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 457 for key SimpleKeyREP3796REP3796REP3796REP3796REP3796 - rev4
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP3796REP3796REP3796REP3796REP3796 using server 10.3.121.135:11210
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: e63556d1-caf1-44fc-8f81-11a670a45a9b | 10.3.121.135:11210 | [0, 2] - 3cd6215b-e17c-493c-bf94-885b7cafaf1a - Disposed: False
2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3796REP3796REP3796REP3796REP3796 using 457 on 10.3.121.135:11210
2014-09-17 01:49:19,231 [31] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP3796REP3796REP3796REP3796REP3796 | Host=10.3.121.135:11210
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: e63556d1-caf1-44fc-8f81-11a670a45a9b on 10.3.121.135:11210 - 3cd6215b-e17c-493c-bf94-885b7cafaf1a
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.136:11210
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 092717c9-ef47-41df-8a81-839302b58bde | 10.3.121.136:11210 | [0, 2] - df9f7db5-1e94-4d4e-ac0d-441ff1b9c698 - Disposed: False
2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.136:11210
Attachments
Issue Links
- duplicates
-
NCBC-671 During rebalance client blocks while updating configuration causing high latency
- Resolved
- is duplicated by
-
NCBC-673 MEMD_ENOENT errors after failover nodes even they returned back to cluster (ReAdd2-HYBRID test) + GENERIC errors for SvcRestart
- Closed
- relates to
-
NCBC-683 When a node goes on/offline ensure that the cluster map is up to date
- Resolved