Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43736

[System Test] : Various errors in FTS logs seen during rebalance out of KV node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.0.0
    • Cheshire-Cat
    • fts

    Description

      Build : 7.0.0-4226
      Test : -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_basic.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Scale : 2
      Iteration : 1st

      In this test, when a KV node 172.23.104.17 is being rebalanced out of the cluster, there are errors seen in fts logs on different nodes.

      172.23.121.165

      2021-01-17T23:56:42.048-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.577-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.969-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      172.23.96.30

      2021-01-17T23:56:42.059-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:43.968-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc035395270` failed to shut down client socket (close tcp 172.23.96.30:53280->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.579-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.739-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.739-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.743-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.743-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.249-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:52.302-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc013aeb540` failed to shut down errored client socket (close tcp 172.23.96.30:52466->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:52.898-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0059567d0` failed to shut down client socket (close tcp 172.23.96.30:52360->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      172.23.96.31

      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.054-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:42.399-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:56:43.259-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0026ce2d0` failed to shut down client socket (close tcp 172.23.96.31:38916->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.579-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.740-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.740-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:57:47.741-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.249-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.970-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.970-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.972-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:51.980-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc009684730` failed to shut down client socket (close tcp 172.23.96.31:39218->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      /opt/couchbase/var/lib/couchbase/logs/fts.log:2021-01-17T23:58:52.624-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc01cd04ff0` failed to shut down client socket (close tcp 172.23.96.31:39506->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      172.23.97.77

      2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.398-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.575-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.576-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:52.860-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc008aa0f50` failed to shut down client socket (close tcp 172.23.97.77:40000->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      172.23.97.82

      2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.396-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:13.481-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:16.790-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:19.881-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:19.886-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:27.396-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:30.541-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:32.398-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:32.399-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:32.400-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:35.329-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:36.083-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:36.088-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:37.080-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:37.084-08:00 [ERRO] (GOCBCORE) Reschedule failed, failing request (connection shut down) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.577-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.734-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.735-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:48.032-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc001e24f50` failed to shut down errored client socket (close tcp 172.23.97.82:50670->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:48.047-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc0292500f0` failed to shut down errored client socket (close tcp 172.23.97.82:51404->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.967-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.968-08:00 [ERRO] (GOCBCORE) memdClient read failure: read tcp 172.23.97.82:50744->172.23.104.17:11210: read: connection reset by peer -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:53.180-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc01e6096d0` failed to shut down client socket (close tcp 172.23.97.82:51542->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:53.188-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc012161360` failed to shut down client socket (close tcp 172.23.97.82:51160->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      172.23.97.83

      2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:56:42.395-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.576-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.736-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.737-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:57:47.738-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.246-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.966-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:51.966-08:00 [ERRO] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      2021-01-17T23:58:53.141-08:00 [ERRO] (GOCBCORE) Pipeline client `172.23.104.17:11210/0xc008973f90` failed to shut down client socket (close tcp 172.23.97.83:50282->172.23.104.17:11210: use of closed network connection) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:613
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhinav Abhi Dangeti
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty