Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7786

[RN 2.0.2] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.1.0
    • Component/s: ns_server
    • Security Level: Public
    • Environment:
      2.0.1.-160-rel
    • Flagged:
      Release Note

      Description

      Seeing 2 issues when trying to setup replication for the first time between two clusters -

      1.Frequently seeing error messages on creating the 1st time replication from cluster1 to cluster2.
      "Failed to grab remote bucket info, vbucket.."

      Both the buckets on source/destination are available for a long period of time, so it does not look like it is an issue w/ bucket not being ready.

      I dont have logs on this currently, will add soon.

      Seeing this across platforms - linux / windows and on most 2.0.1 runs.

      2.Replication replicates data as expected, and these error messages persist for over an hour on the xdcr-last 10 errors. This gives user a wrong idea about the state of replication.

      The intial Replication-call should either wait long enough to avoid these errors/ figure out if something else can be done here.

      And also, how frequently do we clean up the xdcr-error messages on the console? Can we clear them sooner than current time?

        Issue Links

        # Subject Project Status CR V
        For Gerrit Dashboard: &For+MB-7786=message:MB-7786

          Activity

          ketaki Ketaki Gangal created issue -
          perry Perry Krug made changes -
          Field Original Value New Value
          Link This issue is duplicated by MB-7457 [ MB-7457 ]
          perry Perry Krug made changes -
          Fix Version/s 2.1 [ 10414 ]
          perry Perry Krug made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          Hide
          ketaki Ketaki Gangal added a comment -

          Please release-note this.

          Show
          ketaki Ketaki Gangal added a comment - Please release-note this.
          ketaki Ketaki Gangal made changes -
          Assignee Aliaksey Artamonau [ aliaksey artamonau ] Jin Lim [ jin ]
          ketaki Ketaki Gangal made changes -
          Planned End (re-schedule end date based on new assignee)
          Show
          ketaki Ketaki Gangal added a comment - Logs from the node it is trying to reach : clusters http://ec2-54-235-229-199.compute-1.amazonaws.com:8091/index.html#sec=replications to http://ec2-107-22-40-124.compute-1.amazonaws.com:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fsasl%3Fbucket_uuid%3De1f9d1e199f28b83c35f26c61ee90ec9
          ketaki Ketaki Gangal made changes -
          Hide
          ketaki Ketaki Gangal added a comment -

          Hi Aliaksey,

          I ve added logs from one of the nodes. Could you take a look?

          Please re-assign this to me/ Jin after you do so.

          thanks,
          Ketaki

          Show
          ketaki Ketaki Gangal added a comment - Hi Aliaksey, I ve added logs from one of the nodes. Could you take a look? Please re-assign this to me/ Jin after you do so. thanks, Ketaki
          ketaki Ketaki Gangal made changes -
          Assignee Jin Lim [ jin ] Aliaksey Artamonau [ aliaksey artamonau ]
          ketaki Ketaki Gangal made changes -
          Planned End (re-schedule end date based on new assignee)
          Hide
          ketaki Ketaki Gangal added a comment -

          Change added here http://review.couchbase.org/#/c/24986/, will be part of next branch.

          Show
          ketaki Ketaki Gangal added a comment - Change added here http://review.couchbase.org/#/c/24986/ , will be part of next branch.
          Show
          ketaki Ketaki Gangal added a comment - http://review.couchbase.org/#/c/24986/
          Hide
          kzeller kzeller added a comment -

          Added as known issue to RN 2.0.1:

          When you create a replication between two clusters, you
          may experience the incorrect error message
          "Failed to grab remote bucket info, vbucket". Replication will start as
          and function expected, but the incorrect error message may persist for some time.
          Please ignore this incorrect error.

          Show
          kzeller kzeller added a comment - Added as known issue to RN 2.0.1: When you create a replication between two clusters, you may experience the incorrect error message "Failed to grab remote bucket info, vbucket". Replication will start as and function expected, but the incorrect error message may persist for some time. Please ignore this incorrect error.
          Hide
          Aliaksey Artamonau Aliaksey Artamonau added a comment -

          I would not call the error incorrect. It's just that replication is able to recover from it.

          Show
          Aliaksey Artamonau Aliaksey Artamonau added a comment - I would not call the error incorrect. It's just that replication is able to recover from it.
          Aliaksey Artamonau Aliaksey Artamonau made changes -
          Fix Version/s 2.0.2 [ 10418 ]
          Fix Version/s 2.1 [ 10414 ]
          Aliaksey Artamonau Aliaksey Artamonau made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          Hide
          kzeller kzeller added a comment -

          Redo as:

          When you create a replication between two clusters, you
          may experience two error messages:
          "Failed to grab remote bucket info, vbucket" and "Error replicating vbucket X". Nonetheless,
          replication
          will still start and then function as expected, but the error messages may appear
          for some time in the Web Console. Please ignore this behavior.

          Show
          kzeller kzeller added a comment - Redo as: When you create a replication between two clusters, you may experience two error messages: "Failed to grab remote bucket info, vbucket" and "Error replicating vbucket X". Nonetheless, replication will still start and then function as expected, but the error messages may appear for some time in the Web Console. Please ignore this behavior.
          Hide
          Aliaksey Artamonau Aliaksey Artamonau added a comment -

          Looks good to me.

          Show
          Aliaksey Artamonau Aliaksey Artamonau added a comment - Looks good to me.
          Hide
          kzeller kzeller added a comment -

          Yes indeed.... : )

          Show
          kzeller kzeller added a comment - Yes indeed.... : )
          Show
          kzeller kzeller added a comment - http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-server-rn_2-0-0l.html
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          Not sure who to assign on.

          Code-wise we've fixed it. It was caused by thundering herd of those remote bucket info requests and we don't allow that anymore.

          I believe folks wanted to add this to release note.

          Anyways Aliaksey is done with that.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - Not sure who to assign on. Code-wise we've fixed it. It was caused by thundering herd of those remote bucket info requests and we don't allow that anymore. I believe folks wanted to add this to release note. Anyways Aliaksey is done with that.
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Assignee Aliaksey Artamonau [ aliaksey artamonau ] Maria McDuff [ maria ]
          maria Maria McDuff (Inactive) made changes -
          Summary Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication. [RN] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication.
          maria Maria McDuff (Inactive) made changes -
          Assignee Maria McDuff [ maria ] Karen Zeller [ kzeller ]
          maria Maria McDuff (Inactive) made changes -
          Assignee Karen Zeller [ kzeller ] Maria McDuff [ maria ]
          Hide
          maria Maria McDuff (Inactive) added a comment -

          karen, are you finished documenting this?
          just flagging this for you for release note.

          Will assign to Ketaki for verification/closing.

          Thanks.

          Show
          maria Maria McDuff (Inactive) added a comment - karen, are you finished documenting this? just flagging this for you for release note. Will assign to Ketaki for verification/closing. Thanks.
          maria Maria McDuff (Inactive) made changes -
          Assignee Maria McDuff [ maria ] Ketaki Gangal [ ketaki ]
          Hide
          kzeller kzeller added a comment -

          I add this to the 2.0.1 release notes as minor known issue to ignore. Is the message now fixed for 2.0.2?

          Show
          kzeller kzeller added a comment - I add this to the 2.0.1 release notes as minor known issue to ignore. Is the message now fixed for 2.0.2?
          kzeller kzeller made changes -
          Summary [RN] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication. [RN 2.0.2?] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication.
          Labels 2.0.2-release-notes
          Flagged Release Note [ 10010 ]
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          As can be seen above it is fixed.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - As can be seen above it is fixed.
          kzeller kzeller made changes -
          Summary [RN 2.0.2?] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication. [RN 2.0.2] Frequent replication start error messages "Failed to grab remote bucket info, vbucket.." at start of replication.
          Hide
          maria Maria McDuff (Inactive) added a comment -

          pls verify / close.
          if issue is fixed, karen does not need to RN for 2.0.2

          Show
          maria Maria McDuff (Inactive) added a comment - pls verify / close. if issue is fixed, karen does not need to RN for 2.0.2
          maria Maria McDuff (Inactive) made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kzeller kzeller added a comment - - edited

          Relabeled in RN 2.0.2 as Fix. For earlier versions was in release notes as known issue.

          Show
          kzeller kzeller added a comment - - edited Relabeled in RN 2.0.2 as Fix. For earlier versions was in release notes as known issue.
          ketaki Ketaki Gangal made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          VERIFICATION STEPS verified w/ 2.1.0-718

            People

            • Assignee:
              ketaki Ketaki Gangal
              Reporter:
              ketaki Ketaki Gangal
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes