Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21432

[CBAS] Inconsistent state when Couchbase node with which CBAS is connected to gets restarted

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • bug-backlog
    • CBAS DP
    • analytics
    • None
    • Untriaged
    • Unknown
    • CX Sprint 49, CX Sprint 50, CX Sprint 51

    Description

      CBAS Build : 283

      Issue : If the CB node with which CBAS bucket is connected to, gets restarted, things go haywire with CBAS.
      1. The connection with the CB bucket on the CBAS side does not get disconnected, but the feed is not ingested from the source. One has to disconnect and reconnect.
      2. After a couple of mins, if any queries are issued for that bucket via UI or REST API, there is no response even after a few mins. No errors in the logs as well. Looks like CBAS hangs.
      3. If restart of CBAS is attempted in this state using the stop-sample-cluster.sh script, sometimes, not all processes gets stopped.

      Steps to Reproduce :
      1. Create a CB bucket. Dont load docs yet.
      2. On CBAS, create a bucket, shadow dataset and connect to the CB bucket.
      3. Load docs on the CB bucket
      4. Ensure all docs are replicated in CBAS.
      5. Now, stop the CB service and start after a minute.
      6. Observe the CBAS logs. There would be exceptions like "java.io.IOException: Connection reset by peer". (At this point, CBAS should disconnect its connections with the host too, but it does not.)
      7. Now, check the count of the docs in the CBAS bucket. It should be same as in Step 4.
      8. Delete some docs from CB bucket.
      9. Check the count of the docs in the CBAS bucket – Issue -> The count is still as in #4.
      10. Disconnect CBAS bucket and connect again.
      11. Check the count of the docs in the CBAS bucket – Issue -> The count is still as in #4.
      12. Wait for 5 mins, and then again query the count. – Issue -> CBAS wouldnt respond and return the count. Tried via UI and REST API.
      13. Wait for a few mins, and then stop CBAS service. – Issue -> Sometimes, all processes do not shutdown. There are tons of exceptions in the logs during this time.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Abdullah.Alamoudi Abdullah Alamoudi [X] (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty