Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8377

CBBackup - Need to ignore deleted items which are unecessary in backup

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.0, 2.0.1, 2.1.0
    • Fix Version/s: 2.2.0
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      PCI Team - Sprint 10, PCI Team - Sprint 11

      Description

      http://www.couchbase.com/issues/browse/MB-7149

      In case of Deletes on items – tool currently only captures the snapshot of 'active items' and doesn't consider any items getting deleted. Hence when it transfers it not only transfers current active items but also any deleted items which is unnecessary. To fix this we require some changes in EP-Engine side to provide stats on deleted items so that tool can smartly ignore those. Considering the timeframe for release this won't make it for 2.1.0 but we will have documentation explaining this to users.

      [Bala]:
      There are 2 issues here. One has been addressed in 2.1 (considering DGM in backup % calculation). But the other part (ignoring tombstones in backup) is not yet addressed. I see Anil's comment that it will not make to 2.1. But should the ticket still be kept open to avoid loosing visibility ?
      Also while ignoring tombstones in backup may be involved; how involved is just considering the # of tombstones in the % calculation ? Atleast we will not confuse the customers if we consider that in the denominator and completion displayed is never more than 100% (or max marginially above 100%).

        Issue Links

        # Subject Project Status CR V
        For Gerrit Dashboard: &For+MB-8377=message:MB-8377

          Activity

          anil Anil Kumar created issue -
          Hide
          perry Perry Krug added a comment -

          Agree definitely with the last sentence here...it's better to show too many items need to be transferred than to go over 100% where the user would have no idea or expectation of when it will stop.

          Show
          perry Perry Krug added a comment - Agree definitely with the last sentence here...it's better to show too many items need to be transferred than to go over 100% where the user would have no idea or expectation of when it will stop.
          anil Anil Kumar made changes -
          Field Original Value New Value
          Description http://www.couchbase.com/issues/browse/MB-7149

          In case of Deletes on items – tool currently only captures the snapshot of 'active items' and doesn't consider any items getting deleted. Hence when it transfers it not only transfers current active items but also any deleted items which is unnecessary. To fix this we require some changes in EP-Engine side to provide stats on deleted items so that tool can smartly ignore those. Considering the timeframe for release this won't make it for 2.0.2 but we will have documentation explaining this to users.

          [Bala]:
          There are 2 issues here. One has been addressed in 2.1 (considering DGM in backup % calculation). But the other part (ignoring tombstones in backup) is not yet addressed. I see Anil's comment that it will not make to 2.1. But should the ticket still be kept open to avoid loosing visibility ?
          Also while ignoring tombstones in backup may be involved; how involved is just considering the # of tombstones in the % calculation ? Atleast we will not confuse the customers if we consider that in the denominator and completion displayed is never more than 100% (or max marginially above 100%).
          http://www.couchbase.com/issues/browse/MB-7149

          In case of Deletes on items – tool currently only captures the snapshot of 'active items' and doesn't consider any items getting deleted. Hence when it transfers it not only transfers current active items but also any deleted items which is unnecessary. To fix this we require some changes in EP-Engine side to provide stats on deleted items so that tool can smartly ignore those. Considering the timeframe for release this won't make it for 2.1.0 but we will have documentation explaining this to users.

          [Bala]:
          There are 2 issues here. One has been addressed in 2.1 (considering DGM in backup % calculation). But the other part (ignoring tombstones in backup) is not yet addressed. I see Anil's comment that it will not make to 2.1. But should the ticket still be kept open to avoid loosing visibility ?
          Also while ignoring tombstones in backup may be involved; how involved is just considering the # of tombstones in the % calculation ? Atleast we will not confuse the customers if we consider that in the denominator and completion displayed is never more than 100% (or max marginially above 100%).
          kzeller kzeller made changes -
          Summary CBBackup - Need to ignore deleted items which are unecessary in backup [RN 2.1.0]CBBackup - Need to ignore deleted items which are unecessary in backup
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s 2.2.0 [ 10620 ]
          Fix Version/s .major-release [ 10414 ]
          Affects Version/s 2.1.0 [ 10418 ]
          Priority Major [ 3 ] Critical [ 2 ]
          anil Anil Kumar made changes -
          Rank Ranked higher
          anil Anil Kumar made changes -
          Rank Ranked higher
          wayne Wayne Siu made changes -
          Sprint PCI Team - Sprint 10 [ 31 ]
          Show
          bcui Bin Cui (Inactive) added a comment - http://review.couchbase.org/#/c/27920/
          pavelpaulau Pavel Paulau made changes -
          Rank Ranked higher
          pavelpaulau Pavel Paulau made changes -
          Sprint PCI Team - Sprint 10 [ 31 ] PCI Team - Sprint 10, PCI Team - Sprint 11 [ 31, 34 ]
          bcui Bin Cui (Inactive) made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          anil Anil Kumar made changes -
          Rank Ranked higher
          maria Maria McDuff (Inactive) made changes -
          Assignee Bin Cui [ bcui ] Shashank Gupta [ shashank ]
          anil Anil Kumar made changes -
          Summary [RN 2.1.0]CBBackup - Need to ignore deleted items which are unecessary in backup CBBackup - Need to ignore deleted items which are unecessary in backup
          Hide
          shashank Shashank Gupta added a comment -

          I tried the following scenario:

          1. Loaded 199982 items in a bucket.
          2. Deleted some items. Items remaining : 163757

          3. Took backup :

          a) With build 2.1.1 : output :

          1. ./cbbackup http://10.3.3.66:8091 /tmp/backup/ -b default -u Administrator -p password

          ####################### 122.1% (199982/163757 msgs)
          bucket: default, msgs transferred...
          : total | last | per sec
          batch : 2712 | 2712 | 83.7
          byte : 204864653 | 204864653 | 6320891.2
          msg : 199982 | 199982 | 6170.2

          b) With build 2.2.0-821 : output :

          1. ./cbbackup http://10.3.3.66:8091 /tmp/backup/ -b default -u Administrator -p password

          ############################ 144.2% (236207/163757 msgs)
          bucket: default, msgs transferred...
          : total | last | per sec
          batch : 2729 | 2729 | 88.4
          byte : 204864653 | 204864653 | 6634171.2
          msg : 236207 | 236207 | 7649.1

          So the only difference found I found is that with 2.1.1, cbbackup ran upto 122.1% and then terminated successfully, but with 2.2.0, cbbackup ran upto 144.2% and then got terminated successfully. So, now also the user will not be able to predict that upto what extend the process will run.

          Show
          shashank Shashank Gupta added a comment - I tried the following scenario: 1. Loaded 199982 items in a bucket. 2. Deleted some items. Items remaining : 163757 3. Took backup : a) With build 2.1.1 : output : ./cbbackup http://10.3.3.66:8091 /tmp/backup/ -b default -u Administrator -p password ####################### 122.1% (199982/163757 msgs) bucket: default, msgs transferred... : total | last | per sec batch : 2712 | 2712 | 83.7 byte : 204864653 | 204864653 | 6320891.2 msg : 199982 | 199982 | 6170.2 b) With build 2.2.0-821 : output : ./cbbackup http://10.3.3.66:8091 /tmp/backup/ -b default -u Administrator -p password ############################ 144.2% (236207/163757 msgs) bucket: default, msgs transferred... : total | last | per sec batch : 2729 | 2729 | 88.4 byte : 204864653 | 204864653 | 6634171.2 msg : 236207 | 236207 | 7649.1 So the only difference found I found is that with 2.1.1, cbbackup ran upto 122.1% and then terminated successfully, but with 2.2.0, cbbackup ran upto 144.2% and then got terminated successfully. So, now also the user will not be able to predict that upto what extend the process will run.
          shashank Shashank Gupta made changes -
          Assignee Shashank Gupta [ shashank ] Bin Cui [ bcui ]
          Hide
          bcui Bin Cui (Inactive) added a comment -

          That's the bug you filed as MB-9075. A fix is pushed for review. It should be part of 2.2.1 hot fix release.

          Show
          bcui Bin Cui (Inactive) added a comment - That's the bug you filed as MB-9075 . A fix is pushed for review. It should be part of 2.2.1 hot fix release.
          Hide
          maria Maria McDuff (Inactive) added a comment -

          closing as dupe.

          Show
          maria Maria McDuff (Inactive) added a comment - closing as dupe.
          maria Maria McDuff (Inactive) made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Bin Cui [ bcui ] Shashank Gupta [ shashank ]
          Hide
          maria Maria McDuff (Inactive) added a comment -
          Show
          maria Maria McDuff (Inactive) added a comment - MB-9075 .
          maria Maria McDuff (Inactive) made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          maria Maria McDuff (Inactive) made changes -
          Link This issue relates to MB-9075 [ MB-9075 ]
          Hide
          maria Maria McDuff (Inactive) added a comment -
          Show
          maria Maria McDuff (Inactive) added a comment - MB-9075 .
          maria Maria McDuff (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              shashank Shashank Gupta
              Reporter:
              anil Anil Kumar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile

                  Gerrit Reviews

                  There are no open Gerrit changes