Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58729

Too many i/o timeout are coming up while connecting to 9101 port in logs.

    XMLWordPrintable

Details

    Description

      1. Create a 3 KV node, 2 GSI/N1QL node cluster.
      2. Create bucket, load data, start queries asynchronously.
      3. Scale up the cluster by 1KV, 1 GSI/N1QL node.
      4. Rebalance is successful. Repeat the above step.
      5. Rebalances are looking fine but a lot of i/o timeout have been observed in the logs: 

        2023-09-18T16:05:20.365+00:00 [Info] [Queryport-connpool:svc-qi-node-016.vg-9bboethcxizu4.sandbox.nonprod-project-avengers.com:9101] closing unhealthy connection "127.0.0.1:50234" authenticated true
        		 
        2023-09-18T16:05:20.368+00:00 [Error] receiving packet: read tcp 127.0.0.1:50210->127.0.0.1:9101: i/o timeout
        		 
        2023-09-18T16:05:20.368+00:00 [Error] [GsiScanClient:"svc-qi-node-016.vg-9bboethcxizu4.sandbox.nonprod-project-avengers.com:9101"] req(731cf000-81ae-4581-abb1-9bf66c7a19f6) connection "127.0.0.1:50210" response transport failed `read tcp 127.0.0.1:50210->127.0.0.1:9101: i/o timeout`
        		 
        2023-09-18T16:05:20.368+00:00 [Info] [Queryport-connpool:svc-qi-node-016.vg-9bboethcxizu4.sandbox.nonprod-project-avengers.com:9101] closing unhealthy connection "127.0.0.1:50210" authenticated true
        		 
        2023-09-18T16:05:20.370+00:00 [Error] receiving packet: read tcp 10.2.0.18:57500->10.2.0.19:9101: i/o timeout
        		 
        2023-09-18T16:05:20.370+00:00 [Error] [GsiScanClient:"svc-qi-node-017.vg-9bboethcxizu4.sandbox.nonprod-project-avengers.com:9101"] req(731cf000-81ae-4581-abb1-9bf66c7a19f6) connection "10.2.0.18:57500" response transport failed `read tcp 10.2.0.18:57500->10.2.0.19:9101: i/o timeout`
        		 
        2023-09-18T16:05:20.370+00:00 [Info] [Queryport-connpool:svc-qi-node-017.vg-9bboethcxizu4.sandbox.nonprod-project-avengers.com:9101] closing unhealthy connection "10.2.0.18:57500" authenticated true
        		 
        2023-09-18T16:05:20.426+00:00 [WARN] Memory growth rate threshold exceeded: 25% > 20%
        		 
        2023-09-18T16:05:20.426+00:00 [WARN] FFDC: Memory growth rate threshold exceeded
        		 
        2023-09-18T16:05:20.462+00:00 [INFO] FFDC: Captured: query_ffdc_MRE_heap_00004135_2023-09-18-160520.426.gz
        		 
        2023-09-18T16:05:20.464+00:00 [INFO] FFDC: Captured: query_ffdc_MRE_mems_00004135_2023-09-18-160520.426.gz
        		 
        2023-09-18T16:05:20.466+00:00 [INFO] FFDC: Captured: query_ffdc_MRE_areq_00004135_2023-09-18-160520.426.gz
        		 
        2023-09-18T16:05:20.527+00:00 [Error] receiving packet: read tcp 10.2.0.18:57508->10.2.0.19:9101: i/o timeout
        		 
        2023-09-18T16:05:20.527+00:00 [Error] receiving packet: read tcp 127.0.0.1:50244->127.0.0.1:9101: i/o timeout
        		 
        2023-09-18T16:05:20.528+00:00 [Error] receiving packet: read tcp 127.0.0.1:50248->127.0.0.1:9101: i/o timeout

      Attachments

        For Gerrit Dashboard: MB-58729
        # Subject Branch Project Status CR V
        203061,2 MB-58729: Read StreamEndResponse for EndStreamRequest in closeStream unstable indexing Status: MERGED +2 +1
        203085,3 MB-58729: Use new scan protocol from 7.6 not 7.2 unstable indexing Status: MERGED +2 +1
        203089,2 MB-58729: Make logging uniform in GsiScanClient unstable indexing Status: MERGED +2 +1

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty