Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-3372

Removed/rebalanced out node continues to be hit with http requests

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.4.5
    • 3.4.5
    • None
    • 0

    Description

      Running a query/view/analytics-query workload whilst removing and rebalancing out a node results in consistent errors for that service. It looks like it's continuing to try and do the requests against the node that has been removed. KV works as expected.

      Here is a test with a query workload:
      http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-7.0.4-7279-enterprise/Rb1OutQuery-N1QL/04-12-23/058570/614b3d2526ee67e922f32b72b1b2752c-N1QL.html

      Starting with 4 nodes:
      172.23.111.132:::kv
      172.23.111.133:::kv,index,fts,n1ql
      172.23.111.134:::kv,index,fts,n1ql
      172.23.111.135:::kv,index,fts,n1ql

      Then the test removes/rebalances out 172.23.111.133.

      And in the logs it is clear that queries are still being sent to that node after it is removed:

      2023-04-12 17:21:14.6637||Couchbase.Query.QueryClient|DEBUG|Request canceled Connection refused (172.23.111.133:8093)
      

      Full set of logs: SdkdConsole.log.zip

      There's also similar results for the equivalent analytics test:
      http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-7.0.4-7279/Rb1OutAnalytics-CBAS/04-12-23/043666/d9542e0ac04d0217309081a0a010659c-CBAS.html

      And views has some errors:
      http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-7.0.4-7279/Rb2Out-HYBRID/04-12-23/039723/d084516afbbe54ac9986bb4e5245246c-HT.html

      Attachments

        For Gerrit Dashboard: NCBC-3372
        # Subject Branch Project Status CR V

        Activity

          People

            jmorris Jeff Morris
            will.broadbelt Will Broadbelt
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty