Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31136

[System Test] "Storage side failure" fatal errors logged while disconnecting link

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 6.0.0
    • analytics
    • centos2 cluster

    Description

      Build : 6.0.0-1592
      Test : -test tests/analytics/test_analytics_rebalance.yml -scope tests/analytics/scope_analytics_rebalance.yml
      Scale : 4
      Iteration : 2nd

      In the component level system test for analytics, the test was disconnecting the link.

      [2018-09-01T07:11:13-07:00, sequoiatools/cbq:b586af] -e=http://172.23.96.212:8095 -u=Administrator -p=password -script=disconnect link Local; -t 2m
      		 
      [pull] sequoiatools/cmd
      		 
      [2018-09-01T07:55:43-07:00, sequoiatools/cmd:d95537] 30

      While this was in progress, "Storage side failure" fatal errors were seen in the analytics logs on 172.23.96.215.

      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.831-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:19:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.848-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:16:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.845-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:12:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.854-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:18:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.840-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:17:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.835-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:14:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.928-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:13:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:48.969-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14740:TAID:TID:ANID:ODID:2:0:15:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:51.190-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14732:TAID:TID:ANID:ODID:2:0:18:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:51.250-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14732:TAID:TID:ANID:ODID:2:0:12:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:51.339-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14732:TAID:TID:ANID:ODID:2:0:15:0:0] Storage side failure
      		 
      /opt/couchbase/var/lib/couchbase/logs/analytics_error.log:2018-09-01T07:23:51.362-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:3.14732:TAID:TID:ANID:ODID:2:0:13:0:0] Storage side failure

      These errors didn't have any visible impact to the disconnect link operation, i.e. the disconnect operation did not fail because of this.

      Attachments

        For Gerrit Dashboard: MB-31136
        # Subject Branch Project Status CR V
        99202,2 MB-31136 Improve log message when enforcing data integrity through halt alice asterix-opt Status: MERGED +2 +1

        Activity

          People

            Abdullah.Alamoudi Abdullah Alamoudi [X] (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty