Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23393

[CBAS] Connect to bucket after CBAS restart fails

    XMLWordPrintable

Details

    Description

      CBAS Build : RC0 - 613

      Issue : Trying to connect to Couchbase bucket after CBAS restart fails. Following is the error in cc.log : 

      2017-03-18T12:02:23.699-07:00 INFO CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-11] \{"host":"172.23.108.232:8095","path":"/_p/query/query/service","statement":"disconnect bucket travel if connected;","pretty":true,"mode":null,"clientContextID":"935e442c-e02a-4d70-9a99-07ad5f7009c9"}
       
      2017-03-18T12:02:31.530-07:00 INFO CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-12] \{"host":"172.23.108.232:8095","path":"/_p/query/query/service","statement":"connect bucket travel with \{\\\"password\\\":\\\"\\\"};","pretty":true,"mode":null,"clientContextID":"ba75e49a-6749-4663-915b-799ab6eddc46"}
       
      2017-03-18T12:02:31.591-07:00 INFO CBAS.work.WorkQueue [Worker:ClusterController] Executing: GetNodeControllersInfo
       
      2017-03-18T12:02:31.692-07:00 INFO CBAS.util.EventLoopProvider [HttpExecutor(port:8095)-12] Initializing Couchbase Environment
       
      2017-03-18T12:02:32.783-07:00 WARN CBAS.config.ConfigManager [HttpExecutor(port:8095)-12] NC option [nc] storage.lsm.bloomfilter.falsepositiverate being accessed outside of NC-scoped configuration.
       
      2017-03-18T12:02:34.408-07:00 WARN CBAS.config.ConfigManager [HttpExecutor(port:8095)-12] NC option [nc] storage.lsm.bloomfilter.falsepositiverate being accessed outside of NC-scoped configuration.
       
      2017-03-18T12:04:34.437-07:00 WARN CBAS.lang.ConnectBucketStatement [HttpExecutor(port:8095)-12] Failed connecting a bucket
       
      com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0004: 60.0s passed before getting back the DCP state from NCs
       
      at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:317) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.calculateStartingPoint(CouchbaseConnectorFactory.java:262) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.runtime.BucketOperatorDescriptor.calculateStartingPoint(BucketOperatorDescriptor.java:64) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.lang.ConnectBucketStatement.combine(ConnectBucketStatement.java:173) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.lang.ConnectBucketStatement.handle(ConnectBucketStatement.java:433) [cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:378) [asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.asterix.api.http.server.QueryServiceServlet.handleRequest(QueryServiceServlet.java:435) [asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.asterix.api.http.server.QueryServiceServlet.post(QueryServiceServlet.java:87) [asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:76) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:70) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:55) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:36) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
       
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
       
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
       
      at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
       
      2017-03-18T12:04:34.446-07:00 ERRO CBAS.apache.asterix [HttpExecutor(port:8095)-12] CBAS0004: 60.0s passed before getting back the DCP state from NCs
       
      org.apache.hyracks.api.exceptions.HyracksDataException: CBAS0004: 60.0s passed before getting back the DCP state from NCs
       
      at com.couchbase.analytics.lang.ConnectBucketStatement.handle(ConnectBucketStatement.java:447) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:378) ~[asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.asterix.api.http.server.QueryServiceServlet.handleRequest(QueryServiceServlet.java:435) [asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.asterix.api.http.server.QueryServiceServlet.post(QueryServiceServlet.java:87) [asterix-app-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:76) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:70) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:55) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:36) [hyracks-http-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT]
       
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
       
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
       
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
       
      at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
       
      Caused by: com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0004: 60.0s passed before getting back the DCP state from NCs
       
      at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:317) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.calculateStartingPoint(CouchbaseConnectorFactory.java:262) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.runtime.BucketOperatorDescriptor.calculateStartingPoint(BucketOperatorDescriptor.java:64) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.lang.ConnectBucketStatement.combine(ConnectBucketStatement.java:173) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      at com.couchbase.analytics.lang.ConnectBucketStatement.handle(ConnectBucketStatement.java:433) ~[cbas-connector-1.0.0-DP2.jar:1.0.0-DP2]
       
      ... 11 more
      

      Steps :
      1. Create a Couchbase bucket and load some docs
      2. On CBAS create a bucket and a shadow dataset for this CB bucket
      3. Connect to the couchbase bucket
      4. Restart CBAS
      5. Disconnect to couchbase bucket if connected
      6. Connect to the couchbase bucket. This is when the above failure is shown.

      Logs attached.

      Attachments

        1. blue-service.log
          313 kB
        2. cc.log
          119 kB
        3. red-service.log
          292 kB
        For Gerrit Dashboard: MB-23393
        # Subject Branch Project Status CR V

        Activity

          People

            Abdullah.Alamoudi Abdullah Alamoudi [X] (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty