Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27280

bigfun_query_withbg_rb_cbas_10m test failed, data sent to CB is not synced to CBAS for 1 hour during rebalance.

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      bigfun_query_withbg_rb_cbas_10m test failed, rebalance failed. 

      http://perf.jenkins.couchbase.com/job/oceanus-cbas/233/console

      This test does the following test steps:

      • Populates bigfun dataset (at 10M gbook user scale level, ie. 160M rows) in Couchbase.
      • Connect Couchbase bucket with data sets in Analtyics, wait for them to be populated in Analytics.
      • Start 3 threads, each thread run all bigfun queries (3 times each) and report the server side execution latency.
      • Meanwhile, start a background mutation process which does 6k mutations per second in the background. (80% update, 10% insert, 10% delete)
      • Meanwhile, rebalance in 1 analytics node.

      Meanwhile there is a background process keep inserting new document into Couchbase and query Analytics to make sure it is appearing in Analytics side in 1 hour.

      The rebalance failed at 16:26 

      2017-12-14T16:26:27.018-08:00 WARN CBAS.netty.DcpMessageHandler [nioEventLoopGroup-4-48] BufferAckWatermark absolute is 26214
      		 
       2017-12-14T16:26:27.381-08:00 WARN CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-0] Rebalance ff6407bdd10b92fe8d74c1d58b0c33e2 failed
      		 
       com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0001: Shadows in different partitions have different DCP states
      		 
       at com.couchbase.analytics.adapter.ShadowStatesRequest.exact(ShadowStatesRequest.java:296) ~[cbas-connector-5.1.0.jar:5.1.0]
      		 
       at com.couchbase.analytics.adapter.ShadowStatesRequest.handle(ShadowStatesRequest.java:131) ~[cbas-connector-5.1.0.jar:5.1.0]
      		 
       at org.apache.asterix.messaging.NCMessageBroker.receivedMessage(NCMessageBroker.java:92) ~[asterix-app-5.1.0.jar:5.1.0]
      		 
       at org.apache.hyracks.control.nc.work.ApplicationMessageWork.run(ApplicationMessageWork.java:51) ~[hyracks-control-nc-5.1.0.jar:5.1.0]
      		 
       at org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127) ~[hyracks-control-common-5.1.0.jar:5.1.0]

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-28620 [CX] Rebalance when DCP states are different

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Abdullah.Alamoudi Abdullah Alamoudi [X] (Inactive)
              hui.wang Hui Wang (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty