Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25862

[CX] ArrayIndexOutOfBoundsException on rebalance-without-disconnect

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.5.0
    • CBAS DP3
    • analytics
    • None
    • Untriaged
    • Unknown
    • CX Sprint 68

    Description

      With ingestion active, if a node is restarted followed by a rebalance-in, intermittently an array out of bounds exception is observed on the new node, due to the recovery of the primary node erroneously being run on it:

      Caused by: java.lang.ArrayIndexOutOfBoundsException: 1

      at org.apache.asterix.external.util.FeedUtils.getFeedLogManager(FeedUtils.java:110) ~[asterix-external-data-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT]

      at org.apache.asterix.external.adapter.factory.GenericAdapterFactory.createAdapter(GenericAdapterFactory.java:105) ~[asterix-external-data-0.9.3-SNAPSHOT.jar:0.9.3-SNAPSHOT]

      at com.couchbase.analytics.runtime.BucketOperatorNodePushable.<init>(BucketOperatorNodePushable.java:31) ~[cbas-connector-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]

       

      the recovery task seems to not find out about the failed connect, and hangs:

      "name": "RecoveryTask (Default.beersample(CouchbaseMetadataExtension))","stack": ["java.lang.Object.wait(Native Method)","java.lang.Object.wait(Object.java:502)","org.apache.asterix.external.feed.watch.AbstractSubscriber.sync(AbstractSubscriber.java:57)","com.couchbase.analytics.lang.ConnectBucketStatement.doConnect(ConnectBucketStatement.java:491)","com.couchbase.analytics.metadata.BucketEventsListener.doConnect(BucketEventsListener.java:242)","com.couchbase.analytics.metadata.BucketEventsListener.doStart(BucketEventsListener.java:220)","org.apache.asterix.app.active.RecoveryTask.doRecover(RecoveryTask.java:146)"...

       

      this blocks any rebalances due to the lock being held by the recovery task:

       

      "org.apache.asterix.metadata.lock.MetadataLockManager.acquireActiveEntityWriteLock(MetadataLockManager.java:139)","org.apache.asterix.app.active.ActiveNotificationHandler.suspend(ActiveNotificationHandler.java:245)","com.couchbase.analytics.control.rebalance.Rebalance.call(Rebalance.java:113)",

      Attachments

        1. diagnostics-n_0.json
          73 kB
        2. diagnostics-n_1.json
          35 kB
        3. logs.zip
          2.20 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            michael.blow Michael Blow
            michael.blow Michael Blow
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty