Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50584

ns_server stop responding after continuous CRUD on bucket and collections

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0, 7.0.3
    • 7.1.0
    • secondary-index
    • None
    • 7.1.0-2123

    Description

      1. Create 2 buckets first and second
      2. Add scope and collections to these buckets
      3. Create users for scope and collections, and access then using pythonSDK3
      The test cases does above in the loop for different roles related to scope and collections.

      At one point of time ns_server stops responding.

      [2022-01-25 04:42:26,900] - [rest_client:2989] INFO - http://172.23.123.41:8091/pools/default/buckets with param: name=first&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
      [2022-01-25 04:42:27,032] - [rest_client:3014] INFO - 0.13 seconds to create bucket first
      [2022-01-25 04:42:27,032] - [basetestcase:829] INFO - sleep for 10 secs.  ...
      [2022-01-25 04:42:37,158] - [rest_client:666] INFO - http://172.23.123.41:8091/pools/default/buckets/first/scopes with params: name=scope1
      [2022-01-25 04:42:37,159] - [rest_client:669] INFO - Scope created first->scope1 {'uid': '1'}
      [2022-01-25 04:42:37,447] - [rest_client:689] INFO - http://172.23.123.41:8091/pools/default/buckets/first/scopes/scope1/collections with params: name=collection1
      [2022-01-25 04:42:37,448] - [rest_client:700] INFO - Collection created first->scope1->collection1 manifest:{'uid': '2'}
      [2022-01-25 04:42:37,448] - [basetestcase:829] INFO - sleep for 10 secs.  ...
      [2022-01-25 04:42:48,137] - [rest_client:2989] INFO - http://172.23.123.41:8091/pools/default/buckets with param: name=second&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
      [2022-01-25 04:42:48,251] - [rest_client:3014] INFO - 0.11 seconds to create bucket second
      [2022-01-25 04:42:48,251] - [basetestcase:829] INFO - sleep for 10 secs.  ...
      [2022-01-25 04:44:58,334] - [rest_client:1078] ERROR - socket error while connecting to http://172.23.123.41:8091/pools/default/buckets/second/scopes error timed out 
      [2022-01-25 04:44:58,334] - [rest_client:1081] ERROR - Giving up due to timed out! Tried http://172.23.123.41:8091/pools/default/buckets/second/scopes connect 1 times.
      [2022-01-25 04:46:58,380] - [rest_client:1078] ERROR - socket error while connecting to http://172.23.123.41:8091/nodes/self error timed out 
      [2022-01-25 04:46:58,380] - [rest_client:1081] ERROR - Giving up due to timed out! Tried http://172.23.123.41:8091/nodes/self connect 1 times.
      

      This is a regression from build 7.1.0-2021

      Attachments

        1. consoleText (8).txt
          1.47 MB
        2. image-2022-01-25-17-28-54-677.png
          image-2022-01-25-17-28-54-677.png
          43 kB
        3. rest_enter_leave_diffs.png
          rest_enter_leave_diffs.png
          51 kB
        4. streamingCallsMB1.jpg
          streamingCallsMB1.jpg
          411 kB
        5. streaming requests.png
          streaming requests.png
          29 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Update: Checking if fix for MB-46610 might cause this issue. Addition of `defer io.Copy(ioutil.Discard, res.Body)` to read the data from streaming endpoints might block.

            sai.teja Sai Krishna Teja added a comment - Update: Checking if fix for MB-46610 might cause this issue. Addition of `defer io.Copy(ioutil.Discard, res.Body)` to read the data from streaming endpoints might block.

            Update: When service notifier instance is being cleaned up all the existing streaming connections does not terminate immediately, they will get terminated on next notification. This instance cleanup happens on bucket delete. NodesServicesStreaming connection is not getting closed as there is no update on this endpoint when there are only CRUD operations going on. Adding a close channel to service notifier for closing the streaming endpoints explicitly.

            sai.teja Sai Krishna Teja added a comment - Update: When service notifier instance is being cleaned up all the existing streaming connections does not terminate immediately, they will get terminated on next notification. This instance cleanup happens on bucket delete. NodesServicesStreaming connection is not getting closed as there is no update on this endpoint when there are only CRUD operations going on. Adding a close channel to service notifier for closing the streaming endpoints explicitly.
            sai.teja Sai Krishna Teja added a comment - - edited

            sai.teja Sai Krishna Teja added a comment - - edited

            Build couchbase-server-7.1.0-2326 contains indexing commit 1e3e01a with commit message:
            MB-50584: Add close channel to streaming endpoints in service notifier

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2326 contains indexing commit 1e3e01a with commit message: MB-50584 : Add close channel to streaming endpoints in service notifier

            Build couchbase-server-7.1.0-2326 contains indexing commit 6bc9911 with commit message:
            MB-50584: Revert "MB-46610 Fix more TCP connection leaks"

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2326 contains indexing commit 6bc9911 with commit message: MB-50584 : Revert " MB-46610 Fix more TCP connection leaks"

            People

              ritam.sharma Ritam Sharma
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty