Details
Description
Steps to reproduce:
- Load any sample bucket, say beer-sample.
- Take backup using cbbackup locally via
./cbbackup http://localhost:8091 ~/backup1 -u Administrator -p couchbase -vvv
|
- Delete all design documents for the bucket.
- Run cbrestore with or without design_doc_only option:
./cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p couchbase -vvv
|
- Cannot see any design docs in the UI. Neither are the corresponding files created on the server.
—
Following are the logs generated for the above cbrestore command for 6.0.0:
./cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p couchbase -vvv
|
2018-12-20 05:15:37,883: mt cbrestore...
|
2018-12-20 05:15:37,884: mt source : /home/vagrant/backup1
|
2018-12-20 05:15:37,884: mt sink : http://127.0.0.1:8091
|
2018-12-20 05:15:37,884: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
|
2018-12-20 05:15:37,884: mt source_class: <class 'pump_bfd.BFDSource'>
|
2018-12-20 05:15:37,885: mt sink_class: <class 'pump_cb.CBSink'>
|
2018-12-20 05:15:37,896: mt Starting new HTTP connection (1): 127.0.0.1
|
2018-12-20 05:15:37,900: mt "GET /pools/default/buckets HTTP/1.1" 200 10043
|
2018-12-20 05:15:37,902: mt source_bucket: beer-sample
|
2018-12-20 05:15:37,902: mt sink_bucket: beer-sample
|
2018-12-20 05:15:37,902: mt source_buckets: beer-sample
|
2018-12-20 05:15:37,902: mt bucket_source: beer-sample
|
2018-12-20 05:15:37,902: mt source_buckets filtered: beer-sample
|
2018-12-20 05:15:37,902: mt bucket: beer-sample
|
transfer design doc only. bucket msgs will be skipped.
|
done
|
And following are the logs generated for a successful run where data was backed up and restored from an earlier version, 5.1.1:
vagrant@node1-cb511-ubuntu14:/tmp/backup/bucket-travel-sample$ /opt/couchbase/bin/cbrestore /tmp/backup http://127.0.0.1:8091 -x design_doc_only=1 -b travel-sample -u Administrator -p password -vvv
|
2018-12-20 04:22:58,732: mt cbrestore...
|
2018-12-20 04:22:58,732: mt source : /tmp/backup
|
2018-12-20 04:22:58,733: mt sink : http://127.0.0.1:8091
|
2018-12-20 04:22:58,734: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 1.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'travel-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
|
2018-12-20 04:22:58,735: mt source_class: <class 'pump_bfd.BFDSource'>
|
2018-12-20 04:22:58,735: mt sink_class: <class 'pump_cb.CBSink'>
|
2018-12-20 04:22:58,741: mt Starting new HTTP connection (1): 127.0.0.1
|
2018-12-20 04:22:58,745: mt "GET /pools/default/buckets HTTP/1.1" 200 10023
|
2018-12-20 04:22:58,747: mt source_bucket: travel-sample
|
2018-12-20 04:22:58,747: mt sink_bucket: travel-sample
|
2018-12-20 04:22:58,747: mt source_buckets: travel-sample
|
2018-12-20 04:22:58,748: mt bucket_source: travel-sample
|
2018-12-20 04:22:58,748: mt source_buckets filtered: travel-sample
|
2018-12-20 04:22:58,748: mt bucket: travel-sample
|
transfer design doc only. bucket msgs will be skipped.
|
2018-12-20 04:22:58,749: mt design_doc row: {u'controllers': {u'compact': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/setUpdateMinChanges'}, u'doc': {u'json': {u'views': {u'a': {u'map': u'function (doc, meta) {\n emit(meta.id, null);\n}'}}}, u'meta': {u'rev': u'2-c362e223', u'id': u'_design/dev_a'}}}
|
2018-12-20 04:22:58,749: mt design_doc: {"views": {"a": {"map": "function (doc, meta) {\n emit(meta.id, null);\n}"}}}
|
2018-12-20 04:22:58,750: mt design_doc id: _design/dev_a at: /travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a
|
2018-12-20 04:22:58,750: mt rest_request: Administrator@127.0.0.1:8092/travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a; reason: consume_design
|
2018-12-20 04:22:58,853: mt design_doc created at: /travel-sample%2B84a31a2dae6f5d1b9c8cbdb93a09cd37/_design/dev_a
|
2018-12-20 04:22:58,857: mt Starting new HTTP connection (1): 127.0.0.1
|
2018-12-20 04:22:58,859: mt "GET /pools/default/nodeServices HTTP/1.1" 200 337
|
2018-12-20 04:22:58,861: mt Starting new HTTP connection (1): 127.0.0.1
|
2018-12-20 04:22:58,960: mt "POST /restoreIndexMetadata?bucket=travel-sample HTTP/1.1" 200 18
|
done
|
Note that the log statement in the successful run, printing the contents of the ddoc.json file created by cbbackup, which is not present in the unsuccessful run.
2018-12-20 04:22:58,749: mt design_doc row: {u'controllers': {u'compact': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/compactView', u'setUpdateMinChanges': u'/pools/default/buckets/travel-sample/ddocs/_design%2Fdev_a/controller/setUpdateMinChanges'}, u'doc': {u'json': {u'views': {u'a': {u'map': u'function (doc, meta) {\n emit(meta.id, null);\n}'}}}, u'meta': {u'rev': u'2-c362e223', u'id': u'_design/dev_a'}}}
|
Attachments
Issue Links
- backports to
-
MB-32626
[BP 5.5.x] cbrestore doesn't restore view definitions taken via cbbackup
-
- Closed
-
Gerrit Reviews
Activity
Build couchbase-server-6.0.1-2012 contains couchbase-cli commit 25816fd with commit message:
MB-32419 Simplify transfering of design doc
Verified in 6.0.1-2013:
[root@node3-cb500-testing-centos6 bin]# ./cbrestore ~/backup1 http://10.111.170.101:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p password -vvv
2018-12-26 12:06:08,611: mt cbrestore...
2018-12-26 12:06:08,611: mt source : /root/backup1
2018-12-26 12:06:08,611: mt sink : http://10.111.170.101:8091
2018-12-26 12:06:08,611: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra':
, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
2018-12-26 12:06:08,612: mt source_class: <class 'pump_bfd.BFDSource'>
2018-12-26 12:06:08,613: mt sink_class: <class 'pump_cb.CBSink'>
2018-12-26 12:06:08,618: mt Starting new HTTP connection (1): 10.111.170.101
2018-12-26 12:06:08,624: mt "GET /pools/default/buckets HTTP/1.1" 200 10289
2018-12-26 12:06:08,643: mt source_bucket: beer-sample
2018-12-26 12:06:08,643: mt sink_bucket: beer-sample
2018-12-26 12:06:08,643: mt source_buckets: beer-sample
2018-12-26 12:06:08,643: mt bucket_source: beer-sample
2018-12-26 12:06:08,643: mt source_buckets filtered: beer-sample
2018-12-26 12:06:08,643: mt bucket: beer-sample
transfer design doc only. bucket msgs will be skipped.
2018-12-26 12:06:08,644: mt design_doc row: {u'controllers':
, u'doc': {u'json': {u'_id': u'_design/beer', u'language': u'javascript', u'views': {u'brewery_beers': {u'map': u'function(doc, meta) {\n switch(doc.type) {\n case "brewery":\n emit([meta.id]);\n break;\n case "beer":\n if (doc.brewery_id)
{\n emit([doc.brewery_id, meta.id]);\n }\n break;\n }\n}\n'}, u'by_location': {u'map': u'function (doc, meta) {\n if (doc.country, doc.state, doc.city) {\n emit([doc.country, doc.state, doc.city], 1);\n } else if (doc.country, doc.state) {\n emit([doc.country, doc.state], 1);\n } else if (doc.country) {\n emit([doc.country], 1);\n }\n}', u'reduce': u'_count'}}}, u'meta': {u'rev': u'1-59320655', u'id': u'_design/beer'}}}2018-12-26 12:06:08,644: mt design_doc: {"_id": "_design/beer", "language": "javascript", "views": {"brewery_beers": {"map": "function(doc, meta) {\n switch(doc.type) {\n case \"brewery\":\n emit([meta.id]);\n break;\n case \"beer\":\n if (doc.brewery_id) {n emit([doc.brewery_id, meta.id]);n }
\n break;\n }\n}\n"}, "by_location": {"map": "function (doc, meta) {\n if (doc.country, doc.state, doc.city)
{\n emit([doc.country, doc.state, doc.city], 1);\n }else if (doc.country, doc.state)
{\n emit([doc.country, doc.state], 1);\n }else if (doc.country)
{\n emit([doc.country], 1);\n }\n}", "reduce": "_count"}}}
2018-12-26 12:06:08,644: mt design_doc id: _design/beer at: /beer-sample%2Bd903e75834faa069aed8642b0f4e93b0/_design/beer
2018-12-26 12:06:08,644: mt rest_request: <ud>Administrator</ud>@10.111.170.101:8092/beer-sample%2Bd903e75834faa069aed8642b0f4e93b0/_design/beer; reason: consume_design
2018-12-26 12:06:08,688: mt design_doc created at: /beer-sample%2Bd903e75834faa069aed8642b0f4e93b0/_design/beer
2018-12-26 12:06:08,690: mt Starting new HTTP connection (1): 10.111.170.101
2018-12-26 12:06:08,693: mt "GET /pools/default/nodeServices HTTP/1.1" 200 611
2018-12-26 12:06:08,694: mt Starting new HTTP connection (1): 10.111.170.101
2018-12-26 12:06:08,789: mt "POST /restoreIndexMetadata?bucket=beer-sample HTTP/1.1" 200 18
done
[root@node3-cb500-testing-centos6 bin]#
Build couchbase-server-5.5.4-4307 contains couchbase-cli commit bd24e5e with commit message:
MB-32419 Simplify transfering of design doc
Build couchbase-server-6.0.1-2033 contains couchbase-cli commit 0db2826 with commit message:
MB-32672 Merge branch 'vulcan' into 'alice'
Build couchbase-server-6.0.1-2033 contains couchbase-cli commit bd24e5e with commit message:
MB-32419 Simplify transfering of design doc
Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 0db2826 with commit message:
MB-32672 Merge branch 'vulcan' into 'alice'
Build couchbase-server-6.5.0-2078 contains couchbase-cli commit bd24e5e with commit message:
MB-32419 Simplify transfering of design doc
Build couchbase-server-6.5.0-2078 contains couchbase-cli commit 25816fd with commit message:
MB-32419 Simplify transfering of design doc
Verified in 5.5.4-4321:
[root@node1-mad-hatter-testing-centos7 ~]# /opt/couchbase/bin/cbrestore ~/backup1 http://127.0.0.1:8091 -x design_doc_only=1 -b beer-sample -u Administrator -p password -vvv
2019-02-28 03:25:58,710: mt cbrestore...
2019-02-28 03:25:58,711: mt source : /root/backup1
2019-02-28 03:25:58,711: mt sink : http://127.0.0.1:8091
2019-02-28 03:25:58,711: mt opts : {'username': '<xxx>', 'verbose': 3, 'extra':
, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'beer-sample', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
2019-02-28 03:25:58,711: mt source_class: <class 'pump_bfd.BFDSource'>
2019-02-28 03:25:58,711: mt sink_class: <class 'pump_cb.CBSink'>
2019-02-28 03:25:58,723: mt Starting new HTTP connection (1): 127.0.0.1
2019-02-28 03:25:58,727: mt "GET /pools/default/buckets HTTP/1.1" 200 10065
2019-02-28 03:25:58,734: mt source_bucket: beer-sample
2019-02-28 03:25:58,734: mt sink_bucket: beer-sample
2019-02-28 03:25:58,734: mt source_buckets: beer-sample
2019-02-28 03:25:58,734: mt bucket_source: beer-sample
2019-02-28 03:25:58,734: mt source_buckets filtered: beer-sample
2019-02-28 03:25:58,735: mt bucket: beer-sample
transfer design doc only. bucket msgs will be skipped.
2019-02-28 03:25:58,735: mt design_doc row: {u'controllers':
, u'doc': {u'json': {u'_id': u'_design/beer', u'language': u'javascript', u'views': {u'brewery_beers': {u'map': u'function(doc, meta) {\n switch(doc.type) {\n case "brewery":\n emit([meta.id]);\n break;\n case "beer":\n if (doc.brewery_id)
{\n emit([doc.brewery_id, meta.id]);\n }\n break;\n }\n}\n'}, u'by_location': {u'map': u'function (doc, meta) {\n if (doc.country, doc.state, doc.city) {\n emit([doc.country, doc.state, doc.city], 1);\n } else if (doc.country, doc.state) {\n emit([doc.country, doc.state], 1);\n } else if (doc.country) {\n emit([doc.country], 1);\n }\n}', u'reduce': u'_count'}}}, u'meta': {u'rev': u'1-bd8521cf', u'id': u'_design/beer'}}}2019-02-28 03:25:58,735: mt design_doc: {"_id": "_design/beer", "language": "javascript", "views": {"brewery_beers": {"map": "function(doc, meta) {\n switch(doc.type) {\n case \"brewery\":\n emit([meta.id]);\n break;\n case \"beer\":\n if (doc.brewery_id) {n emit([doc.brewery_id, meta.id]);n }
\n break;\n }\n}\n"}, "by_location": {"map": "function (doc, meta) {\n if (doc.country, doc.state, doc.city)
{\n emit([doc.country, doc.state, doc.city], 1);\n }else if (doc.country, doc.state)
{\n emit([doc.country, doc.state], 1);\n }else if (doc.country)
{\n emit([doc.country], 1);\n }\n}", "reduce": "_count"}}}
2019-02-28 03:25:58,735: mt design_doc id: _design/beer at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer
2019-02-28 03:25:58,735: mt rest_request: <ud>Administrator</ud>@127.0.0.1:8092/beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer; reason: consume_design
2019-02-28 03:25:58,782: mt design_doc created at: /beer-sample%2B5407dd72f4e8f9aa1f16aa8fd422356c/_design/beer
2019-02-28 03:25:58,783: mt Starting new HTTP connection (1): 127.0.0.1
2019-02-28 03:25:58,785: mt "GET /pools/default/nodeServices HTTP/1.1" 200 363
2019-02-28 03:25:58,786: mt Starting new HTTP connection (1): 127.0.0.1
2019-02-28 03:25:58,879: mt "POST /restoreIndexMetadata?bucket=beer-sample HTTP/1.1" 200 18
done
Verified on build 6.5.0-3143, index defs are restored back to restore cluster.
This bug was introduce in 5.5.0. It's related to how design docs are transferred depending on the bucket type. Unfortunately not all sources, in this case backup does have a bucket types.