Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-4226
Test : -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_basic.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Scale : 2
Iteration : 1st
In this test, when a KV node 172.23.104.17 is being rebalanced out of the cluster, there are errors seen in fts logs on different nodes.
172.23.121.165
2021-01-17T23:56:42.048-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.577-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.969-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
172.23.96.30
2021-01-17T23:56:42.059-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:43.968-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc035395270` failed to shut down client socket (close tcp 172.23.96.30:53280->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.579-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.739-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.739-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.743-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.743-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.249-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:52.302-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc013aeb540` failed to shut down errored client socket (close tcp 172.23.96.30:52466->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:52.898-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0059567d0` failed to shut down client socket (close tcp 172.23.96.30:52360->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
172.23.96.31
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.054-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:43.259-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0026ce2d0` failed to shut down client socket (close tcp 172.23.96.31:38916->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.579-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.740-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.740-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.741-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.249-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.970-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.970-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.980-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc009684730` failed to shut down client socket (close tcp 172.23.96.31:39218->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:52.624-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc01cd04ff0` failed to shut down client socket (close tcp 172.23.96.31:39506->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
172.23.97.77
2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.575-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.576-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:52.860-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc008aa0f50` failed to shut down client socket (close tcp 172.23.97.77:40000->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
172.23.97.82
2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:13.481-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:16.790-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:19.881-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:19.886-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:27.396-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:30.541-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:32.398-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:32.399-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:32.400-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:35.329-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:36.083-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:36.088-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:37.080-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:37.084-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.577-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.734-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.735-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:48.032-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc001e24f50` failed to shut down errored client socket (close tcp 172.23.97.82:50670->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:48.047-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0292500f0` failed to shut down errored client socket (close tcp 172.23.97.82:51404->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: read tcp 172.23.97.82:50744->172.23.104.17:11210: read: connection reset by peer -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:53.180-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc01e6096d0` failed to shut down client socket (close tcp 172.23.97.82:51542->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:53.188-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc012161360` failed to shut down client socket (close tcp 172.23.97.82:51160->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
172.23.97.83
2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.576-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:57:47.738-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.966-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:51.966-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
2021-01-17T23:58:53.141-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc008973f90` failed to shut down client socket (close tcp 172.23.97.83:50282->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
|
Attachments
Issue Links
- depends on
-
GOCBC-1041 Drop log level for memdclient read failures
- Resolved