Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7899

XDCR: enhance logging to provide information about conflicts encountered and steps taken

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0
    • Fix Version/s: None
    • Component/s: XDCR
    • Security Level: Public
    • Labels:
      None

      Description

      At the moment it appears that the xdcr log file shows that conflicts are happening:
      [xdcr:debug,2013-03-10T4:13:30.487,ns_1@172.31.0.65:<0.20529.50>:capi_replication:get_missing_revs:47][Bucket:"cdb", Vb:833]: after conflict resolution for 1 docs, num of remote winners is 1 and number of local winners is 0. (time spent in ms: 0, avg latency in ms per doc: 0)
      and
      [xdcr:debug,2013-03-10T4:13:35.843,ns_1@172.31.0.65:<0.22973.50>:xdc_vbucket_rep_worker:find_missing:133]after conflict resolution at target ("http://admin:LeG0la%242012@192.168.3.45:8092/mdb%2f851%3b1251e880f2f22df8ef32e43207f173ff/"), out of all 1 docs the number of docs we need to replicate is: 0

      However there is no way to know:
      -Which key(s) were affected
      -Which revision ID/cas value/flags/ttl were used to resolve

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          chiyoung Chiyoung Seo added a comment -

          This is the XDCR erlang side world.

          Show
          chiyoung Chiyoung Seo added a comment - This is the XDCR erlang side world.
          Hide
          junyi Junyi Xie (Inactive) added a comment -

          Perry,

          Thanks for looking into the log. The log is for stat purpose, I intentionally avoid dump specific keys and revision info used to solve the conflict, to avoid the overhead on performance and bloated log file. This piece of code in on data path, and we have limitation on size of logs to dump. We cannot afford to dumping too much information on disk.

          Also, please notice the "conflict resolution" here in the log just means the process we pick up winner and loser, strictly speaking, it does not necessarily mean there is an conflict between two candidates.

          Show
          junyi Junyi Xie (Inactive) added a comment - Perry, Thanks for looking into the log. The log is for stat purpose, I intentionally avoid dump specific keys and revision info used to solve the conflict, to avoid the overhead on performance and bloated log file. This piece of code in on data path, and we have limitation on size of logs to dump. We cannot afford to dumping too much information on disk. Also, please notice the "conflict resolution" here in the log just means the process we pick up winner and loser, strictly speaking, it does not necessarily mean there is an conflict between two candidates.
          Hide
          perry Perry Krug added a comment -

          Thanks for the quick response Junyi. I guess that does make sense...but then I need to ask how you suggest we diagnose any conflict resolution issues. How can we tell which keys and which one is winning?

          Show
          perry Perry Krug added a comment - Thanks for the quick response Junyi. I guess that does make sense...but then I need to ask how you suggest we diagnose any conflict resolution issues. How can we tell which keys and which one is winning?
          Hide
          junyi Junyi Xie (Inactive) added a comment -

          Hi Perry,

          That is a good question but we probably cannot diag from the log.

          One possible way to verify conflict resolution is to check the meta on both sides before XDCR, and verify the metadata after XDCR, to make sure XDCR pick up the correct winner.

          This is how we do functional test in QE. Hope it helps.

          Show
          junyi Junyi Xie (Inactive) added a comment - Hi Perry, That is a good question but we probably cannot diag from the log. One possible way to verify conflict resolution is to check the meta on both sides before XDCR, and verify the metadata after XDCR, to make sure XDCR pick up the correct winner. This is how we do functional test in QE. Hope it helps.
          Hide
          perry Perry Krug added a comment -

          But after the replication and it is eventually consistent...how would we know what the metadata was beforehand?

          Show
          perry Perry Krug added a comment - But after the replication and it is eventually consistent...how would we know what the metadata was beforehand?
          Hide
          junyi Junyi Xie (Inactive) added a comment -

          We do not know. We override the losers with winner's metadata (setWithMeta)

          That why I said, in QE test, we intentionally created some metadata at src and dest beforehand, e.g., to make src the winner, and after XDCR, we verify the src is really the winner.

          Show
          junyi Junyi Xie (Inactive) added a comment - We do not know. We override the losers with winner's metadata (setWithMeta) That why I said, in QE test, we intentionally created some metadata at src and dest beforehand, e.g., to make src the winner, and after XDCR, we verify the src is really the winner.
          Hide
          maria Maria McDuff (Inactive) added a comment -

          perry, pls re-open if you need more action to be taken on this one.

          Show
          maria Maria McDuff (Inactive) added a comment - perry, pls re-open if you need more action to be taken on this one.
          Hide
          tgreen Tom Green (Inactive) added a comment -

          I opened MB-15561 yesterday, which I now realised is probably really the same feature request in this MB.

          Show
          tgreen Tom Green (Inactive) added a comment - I opened MB-15561 yesterday, which I now realised is probably really the same feature request in this MB.

            People

            • Assignee:
              perry Perry Krug
              Reporter:
              perry Perry Krug
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes