Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32419

cbrestore doesn't restore view definitions taken via cbbackup

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Steps to reproduce:

      • Load any sample bucket, say beer-sample.
      • Take backup using cbbackup locally via

      ./cbbackup http://localhost:8091 ~/backup1 -u Administrator -p couchbase -vvv
      

      • Delete all design documents for the bucket.
      • Run cbrestore with or without design_doc_only option:

      ./cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p couchbase -vvv
      

      • Cannot see any design docs in the UI. Neither are the corresponding files created on the server.

      Following are the logs generated for the above cbrestore command for 6.0.0:

      ./cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p couchbase -vvv
      2018-12-20 05:15:37,883: mt cbrestore...
      2018-12-20 05:15:37,884: mt  source : /home/vagrant/backup1
      2018-12-20 05:15:37,884: mt  sink   : http://127.0.0.1:8091
      2018-12-20 05:15:37,884: mt  opts   : {'username': '<xxx>', 'verbose': 3, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
      2018-12-20 05:15:37,884: mt source_class: <class 'pump_bfd.BFDSource'>
      2018-12-20 05:15:37,885: mt sink_class: <class 'pump_cb.CBSink'>
      2018-12-20 05:15:37,896: mt Starting new HTTP connection (1): 127.0.0.1
      2018-12-20 05:15:37,900: mt "GET /pools/default/buckets HTTP/1.1" 200 10043
      2018-12-20 05:15:37,902: mt source_bucket: beer-sample
      2018-12-20 05:15:37,902: mt sink_bucket: beer-sample
      2018-12-20 05:15:37,902: mt source_buckets: beer-sample
      2018-12-20 05:15:37,902: mt bucket_source: beer-sample
      2018-12-20 05:15:37,902: mt source_buckets filtered: beer-sample
      2018-12-20 05:15:37,902: mt bucket: beer-sample
      transfer design doc only. bucket msgs will be skipped.
      done
      

      And following are the logs generated for a successful run where data was backed up and restored from an earlier version, 5.1.1:

      vagrant@node1-cb511-ubuntu14:/tmp/backup/bucket-travel-sample$ /opt/couchbase/bin/cbrestore /tmp/backup http://127.0.0.1:8091 -x design_doc_only=1 -b travel-sample -u Administrator -p password -vvv
      2018-12-20 04:22:58,732: mt cbrestore...
      2018-12-20 04:22:58,732: mt  source : /tmp/backup
      2018-12-20 04:22:58,733: mt  sink   : http://127.0.0.1:8091
      2018-12-20 04:22:58,734: mt  opts   : {'username': '<xxx>', 'verbose': 3, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'travel-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
      2018-12-20 04:22:58,735: mt source_class: <class 'pump_bfd.BFDSource'>
      2018-12-20 04:22:58,735: mt sink_class: <class 'pump_cb.CBSink'>
      2018-12-20 04:22:58,741: mt Starting new HTTP connection (1): 127.0.0.1
      2018-12-20 04:22:58,745: mt "GET /pools/default/buckets HTTP/1.1" 200 10023
      2018-12-20 04:22:58,747: mt source_bucket: travel-sample
      2018-12-20 04:22:58,747: mt sink_bucket: travel-sample
      2018-12-20 04:22:58,747: mt source_buckets: travel-sample
      2018-12-20 04:22:58,748: mt bucket_source: travel-sample
      2018-12-20 04:22:58,748: mt source_buckets filtered: travel-sample
      2018-12-20 04:22:58,748: mt bucket: travel-sample
      transfer design doc only. bucket msgs will be skipped.
      2018-12-20 04:22:58,749: mt design_doc row: {u'controllers': {u'compact': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/setUpdateMinChanges'}, u'doc': {u'json': {u'views': {u'a': {u'map': u'function (doc, meta) {\n  emit(meta.id, null);\n}'}}}, u'meta': {u'rev': u'2-c362e223', u'id': u'_design/dev_a'}}}
      2018-12-20 04:22:58,749: mt design_doc: {"views": {"a": {"map": "function (doc, meta) {\n  emit(meta.id, null);\n}"}}}
      2018-12-20 04:22:58,750: mt design_doc id: _design/dev_a at: /travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a
      2018-12-20 04:22:58,750: mt rest_request: Administrator@127.0.0.1:8092/travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a; reason: consume_design
      2018-12-20 04:22:58,853: mt design_doc created at: /travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a
      2018-12-20 04:22:58,857: mt Starting new HTTP connection (1): 127.0.0.1
      2018-12-20 04:22:58,859: mt "GET /pools/default/nodeServices HTTP/1.1" 200 337
      2018-12-20 04:22:58,861: mt Starting new HTTP connection (1): 127.0.0.1
      2018-12-20 04:22:58,960: mt "POST /restoreIndexMetadata?bucket=travel-sample HTTP/1.1" 200 18
      done
      

      Note that the log statement in the successful run, printing the contents of the ddoc.json file created by cbbackup, which is not present in the unsuccessful run.

      2018-12-20 04:22:58,749: mt design_doc row: {u'controllers': {u'compact': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/setUpdateMinChanges'}, u'doc': {u'json': {u'views': {u'a': {u'map': u'function (doc, meta) {\n  emit(meta.id, null);\n}'}}}, u'meta': {u'rev': u'2-c362e223', u'id': u'_design/dev_a'}}}
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-32419
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 0db2826 with commit message:
            MB-32672 Merge branch 'vulcan' into 'alice'

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 0db2826 with commit message: MB-32672 Merge branch 'vulcan' into 'alice'

            Build couchbase-server-6.5.0-2078 contains couchbase-cli commit bd24e5e with commit message:
            MB-32419 Simplify transfering of design doc

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2078 contains couchbase-cli commit bd24e5e with commit message: MB-32419 Simplify transfering of design doc

            Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 25816fd with commit message:
            MB-32419 Simplify transfering of design doc

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 25816fd with commit message: MB-32419 Simplify transfering of design doc

            Verified in 5.5.4-4321:

            [root@node1-mad-hatter-testing-centos7 ~]# /opt/couchbase/bin/cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p password -vvv
            2019-02-28 03:25:58,710: mt cbrestore...
            2019-02-28 03:25:58,711: mt source : /root/backup1
            2019-02-28 03:25:58,711: mt sink : http://127.0.0.1:8091
            2019-02-28 03:25:58,711: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra':

            {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}

            , 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
            2019-02-28 03:25:58,711: mt source_class: <class 'pump_bfd.BFDSource'>
            2019-02-28 03:25:58,711: mt sink_class: <class 'pump_cb.CBSink'>
            2019-02-28 03:25:58,723: mt Starting new HTTP connection (1): 127.0.0.1
            2019-02-28 03:25:58,727: mt "GET /pools/default/buckets HTTP/1.1" 200 10065
            2019-02-28 03:25:58,734: mt source_bucket: beer-sample
            2019-02-28 03:25:58,734: mt sink_bucket: beer-sample
            2019-02-28 03:25:58,734: mt source_buckets: beer-sample
            2019-02-28 03:25:58,734: mt bucket_source: beer-sample
            2019-02-28 03:25:58,734: mt source_buckets filtered: beer-sample
            2019-02-28 03:25:58,735: mt bucket: beer-sample
            transfer design doc only. bucket msgs will be skipped.
            2019-02-28 03:25:58,735: mt design_doc row: {u'controllers':

            {u'compact': u'/pools/default/buckets/beer-sample/ddocs/_design%2Fbeer/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/beer-sample/ddocs/_design%2Fbeer/controller/setUpdateMinChanges'}

            , u'doc': {u'json': {u'_id': u'_design/beer', u'language': u'javascript', u'views': {u'brewery_beers': {u'map': u'function(doc, meta) {\n switch(doc.type) {\n case "brewery":\n emit([meta.id]);\n break;\n case "beer":\n if (doc.brewery_id)

            {\n emit([doc.brewery_id, meta.id]);\n }\n break;\n }\n}\n'}, u'by_location': {u'map': u'function (doc, meta) {\n if (doc.country, doc.state, doc.city) {\n emit([doc.country, doc.state, doc.city], 1);\n } else if (doc.country, doc.state) {\n emit([doc.country, doc.state], 1);\n } else if (doc.country) {\n emit([doc.country], 1);\n }\n}', u'reduce': u'_count'}}}, u'meta': {u'rev': u'1-bd8521cf', u'id': u'_design/beer'}}}
            2019-02-28 03:25:58,735: mt design_doc: {"_id": "_design/beer", "language": "javascript", "views": {"brewery_beers": {"map": "function(doc, meta) {\n switch(doc.type) {\n case \"brewery\":\n emit([meta.id]);\n break;\n case \"beer\":\n if (doc.brewery_id) {n emit([doc.brewery_id, meta.id]);n }

            \n break;\n }\n}\n"}, "by_location": {"map": "function (doc, meta) {\n if (doc.country, doc.state, doc.city)

            {\n emit([doc.country, doc.state, doc.city], 1);\n }

            else if (doc.country, doc.state)

            {\n emit([doc.country, doc.state], 1);\n }

            else if (doc.country)

            {\n emit([doc.country], 1);\n }

            \n}", "reduce": "_count"}}}
            2019-02-28 03:25:58,735: mt design_doc id: _design/beer at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer
            2019-02-28 03:25:58,735: mt rest_request: <ud>Administrator</ud>@127.0.0.1:8092/beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer; reason: consume_design
            2019-02-28 03:25:58,782: mt design_doc created at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer
            2019-02-28 03:25:58,783: mt Starting new HTTP connection (1): 127.0.0.1
            2019-02-28 03:25:58,785: mt "GET /pools/default/nodeServices HTTP/1.1" 200 363
            2019-02-28 03:25:58,786: mt Starting new HTTP connection (1): 127.0.0.1
            2019-02-28 03:25:58,879: mt "POST /restoreIndexMetadata?bucket=beer-sample HTTP/1.1" 200 18
            done

            arunkumar Arunkumar Senthilnathan added a comment - Verified in 5.5.4-4321: [root@node1-mad-hatter-testing-centos7 ~] # /opt/couchbase/bin/cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p password -vvv 2019-02-28 03:25:58,710: mt cbrestore... 2019-02-28 03:25:58,711: mt source : /root/backup1 2019-02-28 03:25:58,711: mt sink : http://127.0.0.1:8091 2019-02-28 03:25:58,711: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0} , 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'} 2019-02-28 03:25:58,711: mt source_class: <class 'pump_bfd.BFDSource'> 2019-02-28 03:25:58,711: mt sink_class: <class 'pump_cb.CBSink'> 2019-02-28 03:25:58,723: mt Starting new HTTP connection (1): 127.0.0.1 2019-02-28 03:25:58,727: mt "GET /pools/default/buckets HTTP/1.1" 200 10065 2019-02-28 03:25:58,734: mt source_bucket: beer-sample 2019-02-28 03:25:58,734: mt sink_bucket: beer-sample 2019-02-28 03:25:58,734: mt source_buckets: beer-sample 2019-02-28 03:25:58,734: mt bucket_source: beer-sample 2019-02-28 03:25:58,734: mt source_buckets filtered: beer-sample 2019-02-28 03:25:58,735: mt bucket: beer-sample transfer design doc only. bucket msgs will be skipped. 2019-02-28 03:25:58,735: mt design_doc row: {u'controllers': {u'compact': u'/pools/default/buckets/beer-sample/ddocs/_design%2Fbeer/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/beer-sample/ddocs/_design%2Fbeer/controller/setUpdateMinChanges'} , u'doc': {u'json': {u'_id': u'_design/beer', u'language': u'javascript', u'views': {u'brewery_beers': {u'map': u'function(doc, meta) {\n switch(doc.type) {\n case "brewery":\n emit( [meta.id] );\n break;\n case "beer":\n if (doc.brewery_id) {\n emit([doc.brewery_id, meta.id]);\n }\n break;\n }\n}\n'}, u'by_location': {u'map': u'function (doc, meta) {\n if (doc.country, doc.state, doc.city) {\n emit([doc.country, doc.state, doc.city], 1);\n } else if (doc.country, doc.state) {\n emit([doc.country, doc.state], 1);\n } else if (doc.country) {\n emit([doc.country], 1);\n }\n}', u'reduce': u'_count'}}}, u'meta': {u'rev': u'1-bd8521cf', u'id': u'_design/beer'}}} 2019-02-28 03:25:58,735: mt design_doc: {"_id": "_design/beer", "language": "javascript", "views": {"brewery_beers": {"map": "function(doc, meta) {\n switch(doc.type) {\n case \"brewery\":\n emit( [meta.id] );\n break;\n case \"beer\":\n if (doc.brewery_id) {n emit([doc.brewery_id, meta.id]);n } \n break;\n }\n}\n"}, "by_location": {"map": "function (doc, meta) {\n if (doc.country, doc.state, doc.city) {\n emit([doc.country, doc.state, doc.city], 1);\n } else if (doc.country, doc.state) {\n emit([doc.country, doc.state], 1);\n } else if (doc.country) {\n emit([doc.country], 1);\n } \n}", "reduce": "_count"}}} 2019-02-28 03:25:58,735: mt design_doc id: _design/beer at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer 2019-02-28 03:25:58,735: mt rest_request: <ud>Administrator</ud>@127.0.0.1:8092/beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer; reason: consume_design 2019-02-28 03:25:58,782: mt design_doc created at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer 2019-02-28 03:25:58,783: mt Starting new HTTP connection (1): 127.0.0.1 2019-02-28 03:25:58,785: mt "GET /pools/default/nodeServices HTTP/1.1" 200 363 2019-02-28 03:25:58,786: mt Starting new HTTP connection (1): 127.0.0.1 2019-02-28 03:25:58,879: mt "POST /restoreIndexMetadata?bucket=beer-sample HTTP/1.1" 200 18 done
            thuan Thuan Nguyen added a comment -

            Verified on build 6.5.0-3143, index defs are restored back to restore cluster.

            thuan Thuan Nguyen added a comment - Verified on build 6.5.0-3143, index defs are restored back to restore cluster.

            People

              thuan Thuan Nguyen
              abhishek.jindal Abhishek Jindal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty