Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6519

cbtransfer can not recover data from couchstore files in build 2.0.0-1677-rel

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0
    • 2.0
    • tools
    • Security Level: Public
    • None

    Description

      I create 5 items in default bucket, each item is a str with 256 bytes long. Then I update 3 of them, delete 2 of them. So there are 3 items at the end. Then I use cbtransfer to use couchstore files to recover these items to another cluster. It doesn't work in build 1677. You can see the "byte" transfered is incorrect:

      [2012-09-04 16:17:46,221] - [remote_util:1105] INFO - running command.raw sudo /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data/ http://Administrator:password@10.5.2.22:8091 -b default -B default -v -v -v
      [2012-09-04 16:17:46,263] - [transport:1368] INFO - Secsh channel 7 opened.
      [2012-09-04 16:17:46,556] - [remote_util:1134] INFO - command executed successfully
      [2012-09-04 16:17:46,556] - [remote_util:1086] INFO - 2012-09-04 16:15:34,019: mt cbtransfer...
      [2012-09-04 16:17:46,557] - [remote_util:1086] INFO - 2012-09-04 16:15:34,019: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/
      [2012-09-04 16:17:46,557] - [remote_util:1086] INFO - 2012-09-04 16:15:34,019: mt sink : http://Administrator:password@10.5.2.22:8091
      [2012-09-04 16:17:46,557] - [remote_util:1086] INFO - 2012-09-04 16:15:34,020: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra':

      {'max_retry': 10.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}

      , 'single_node': False, 'bucket_destination': 'default', 'destination_operation': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': 'default'}
      [2012-09-04 16:17:46,558] - [remote_util:1086] INFO - 2012-09-04 16:15:34,020: mt source_class: <class 'pump_sfd.SFDSource'>
      [2012-09-04 16:17:46,558] - [remote_util:1086] INFO - 2012-09-04 16:15:34,057: mt sink_class: <class 'pump_cb.CBSink'>
      [2012-09-04 16:17:46,558] - [remote_util:1086] INFO - 2012-09-04 16:15:34,058: mt rest_request: Administrator@10.5.2.22:8091/pools/default/buckets
      [2012-09-04 16:17:46,559] - [remote_util:1086] INFO - 2012-09-04 16:15:34,071: mt source_bucket: default
      [2012-09-04 16:17:46,559] - [remote_util:1086] INFO - 2012-09-04 16:15:34,071: mt sink_bucket: default
      [2012-09-04 16:17:46,559] - [remote_util:1086] INFO - 2012-09-04 16:15:34,072: mt source_buckets: default
      [2012-09-04 16:17:46,559] - [remote_util:1086] INFO - 2012-09-04 16:15:34,072: mt bucket_source: default
      [2012-09-04 16:17:46,560] - [remote_util:1086] INFO - 2012-09-04 16:15:34,072: mt source_buckets filtered: default
      [2012-09-04 16:17:46,560] - [remote_util:1086] INFO - 2012-09-04 16:15:34,073: mt bucket: default
      [2012-09-04 16:17:46,560] - [remote_util:1086] INFO - 2012-09-04 16:15:34,073: mt source_nodes: N/A
      [2012-09-04 16:17:46,561] - [remote_util:1086] INFO - 2012-09-04 16:15:34,074: mt enqueueing node: N/A
      [2012-09-04 16:17:46,561] - [remote_util:1086] INFO - 2012-09-04 16:15:34,075: w0 node: N/A
      [2012-09-04 16:17:46,561] - [remote_util:1086] INFO - 2012-09-04 16:15:34,113: w0 pump (couchstore-files:///opt/couchbase/var/lib/couchbase/data/(default@N/A)->http://Administrator:password@10.5.2.22:8091(default@N/A)) done.
      [2012-09-04 16:17:46,561] - [remote_util:1086] INFO - 2012-09-04 16:15:34,114: w0 source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/(default@N/A)
      [2012-09-04 16:17:46,562] - [remote_util:1086] INFO - 2012-09-04 16:15:34,114: w0 sink : http://Administrator:password@10.5.2.22:8091(default@N/A)
      [2012-09-04 16:17:46,562] - [remote_util:1086] INFO - 2012-09-04 16:15:34,114: w0 : total | last | per sec
      [2012-09-04 16:17:46,562] - [remote_util:1086] INFO - 2012-09-04 16:15:34,115: w0 batch : 1 | 1 | 26.2
      [2012-09-04 16:17:46,562] - [remote_util:1086] INFO - 2012-09-04 16:15:34,115: w0 byte : 85 | 85 | 2229.2
      [2012-09-04 16:17:46,563] - [remote_util:1086] INFO - 2012-09-04 16:15:34,115: w0 msg : 5 | 5 | 131.1
      [2012-09-04 16:17:46,563] - [remote_util:1086] INFO - 2012-09-04 16:15:34,116: w0 node: N/A, done; rv: 0
      [2012-09-04 16:17:46,563] - [remote_util:1086] INFO - .
      [2012-09-04 16:17:46,563] - [remote_util:1086] INFO - bucket: default, msgs transferred...
      [2012-09-04 16:17:46,564] - [remote_util:1086] INFO - : total | last | per sec
      [2012-09-04 16:17:46,564] - [remote_util:1086] INFO - batch : 1 | 1 | 16.5
      [2012-09-04 16:17:46,564] - [remote_util:1086] INFO - byte : 85 | 85 | 1400.7
      [2012-09-04 16:17:46,564] - [remote_util:1086] INFO - msg : 5 | 5 | 82.4
      [2012-09-04 16:17:46,564] - [remote_util:1086] INFO - done

      Then I use "print" to see the item value before and after recovery
      The original item value is nosqlV1aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.

      But when recovered, it becomes:
      ?nosqlV1a????

      But this feature works in 1666 and previous build:
      [2012-09-04 16:14:25,988] - [remote_util:1105] INFO - running command.raw sudo /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data/ http://Administrator:password@10.1.3.72:8091 -b default -B default -v -v -v
      [2012-09-04 16:14:26,026] - [transport:1368] INFO - Secsh channel 7 opened.
      [2012-09-04 16:14:26,286] - [remote_util:1134] INFO - command executed successfully
      [2012-09-04 16:14:26,287] - [remote_util:1086] INFO - 2012-09-04 16:13:12,264: mt cbtransfer...
      [2012-09-04 16:14:26,287] - [remote_util:1086] INFO - 2012-09-04 16:13:12,264: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/
      [2012-09-04 16:14:26,288] - [remote_util:1086] INFO - 2012-09-04 16:13:12,264: mt sink : http://Administrator:password@10.1.3.72:8091
      [2012-09-04 16:14:26,288] - [remote_util:1086] INFO - 2012-09-04 16:13:12,264: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra':

      {'max_retry': 10.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}

      , 'single_node': False, 'bucket_destination': 'default', 'destination_operation': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': 'default'}
      [2012-09-04 16:14:26,288] - [remote_util:1086] INFO - 2012-09-04 16:13:12,265: mt source_class: <class 'pump_sfd.SFDSource'>
      [2012-09-04 16:14:26,289] - [remote_util:1086] INFO - 2012-09-04 16:13:12,293: mt sink_class: <class 'pump_cb.CBSink'>
      [2012-09-04 16:14:26,289] - [remote_util:1086] INFO - 2012-09-04 16:13:12,294: mt rest_request: Administrator@10.1.3.72:8091/pools/default/buckets
      [2012-09-04 16:14:26,289] - [remote_util:1086] INFO - 2012-09-04 16:13:12,302: mt source_bucket: default
      [2012-09-04 16:14:26,290] - [remote_util:1086] INFO - 2012-09-04 16:13:12,302: mt sink_bucket: default
      [2012-09-04 16:14:26,290] - [remote_util:1086] INFO - 2012-09-04 16:13:12,302: mt source_buckets: default
      [2012-09-04 16:14:26,290] - [remote_util:1086] INFO - 2012-09-04 16:13:12,303: mt bucket_source: default
      [2012-09-04 16:14:26,290] - [remote_util:1086] INFO - 2012-09-04 16:13:12,303: mt source_buckets filtered: default
      [2012-09-04 16:14:26,291] - [remote_util:1086] INFO - 2012-09-04 16:13:12,303: mt bucket: default
      [2012-09-04 16:14:26,291] - [remote_util:1086] INFO - 2012-09-04 16:13:12,303: mt source_nodes: N/A
      [2012-09-04 16:14:26,291] - [remote_util:1086] INFO - 2012-09-04 16:13:12,304: mt enqueueing node: N/A
      [2012-09-04 16:14:26,291] - [remote_util:1086] INFO - 2012-09-04 16:13:12,305: w0 node: N/A
      [2012-09-04 16:14:26,292] - [remote_util:1086] INFO - 2012-09-04 16:13:12,337: w0 pump (couchstore-files:///opt/couchbase/var/lib/couchbase/data/(default@N/A)->http://Administrator:password@10.1.3.72:8091(default@N/A)) done.
      [2012-09-04 16:14:26,292] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/(default@N/A)
      [2012-09-04 16:14:26,292] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 sink : http://Administrator:password@10.1.3.72:8091(default@N/A)
      [2012-09-04 16:14:26,293] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 : total | last | per sec
      [2012-09-04 16:14:26,293] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 batch : 1 | 1 | 30.4
      [2012-09-04 16:14:26,293] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 byte : 768 | 768 | 23361.2
      [2012-09-04 16:14:26,293] - [remote_util:1086] INFO - 2012-09-04 16:13:12,338: w0 msg : 5 | 5 | 152.1
      [2012-09-04 16:14:26,294] - [remote_util:1086] INFO - 2012-09-04 16:13:12,339: w0 node: N/A, done; rv: 0
      [2012-09-04 16:14:26,294] - [remote_util:1086] INFO - .
      [2012-09-04 16:14:26,294] - [remote_util:1086] INFO - bucket: default, msgs transferred...
      [2012-09-04 16:14:26,294] - [remote_util:1086] INFO - : total | last | per sec
      [2012-09-04 16:14:26,294] - [remote_util:1086] INFO - batch : 1 | 1 | 16.5
      [2012-09-04 16:14:26,295] - [remote_util:1086] INFO - byte : 768 | 768 | 12689.0
      [2012-09-04 16:14:26,295] - [remote_util:1086] INFO - msg : 5 | 5 | 82.6
      [2012-09-04 16:14:26,295] - [remote_util:1086] INFO - done

      Original Value:
      nosql-1aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

      Recovered Value:
      nosql-1aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

      Attachments

        For Gerrit Dashboard: MB-6519
        # Subject Branch Project Status CR V

        Activity

          People

            steve Steve Yen
            Chisheng Chisheng Hong (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty