[MB-20015] CE backup doesn't work in Spock and Watson - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 4.5.0, 5.0.0
Fix Version/s: 4.5.1
Component/s: tools
Labels:
Environment:
4.7.0-779

Triage:
Untriaged
Is this a Regression?:
Yes

Description

[root@hera06-s818 ~]# rm -rf /data/cbbackup_dir

[root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr config --archive /data/cbbackup_dir --repo default

Backup repository `default` created successfully in archive `/data/cbbackup_dir`

[root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr backup --archive /data/cbbackup_dir --repo default --host http://172.23.96.117:8091 --username Administrator --password password --threads 16

Backing up to 2016-06-23T05_23_14.48075409-07_00

Copying at 0B (estimating time remaining)                                                                                                                             0 items / 0B

bucket-1                [                                                                                                                                                ] 100.00%

[root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr backup --archive /data/cbbackup_dir --repo default --host http://172.23.96.117:8091 --username Administrator --password password --threads 16

Error backing up cluster: The most recent backup `2016-06-23T05_23_14.48075409-07_00` did not finish properly. You can either resume this backup from where it left off by re-running the backup command and using the --resume flag. To delete the unfinished backup and start again use the --purge flag.

[root@hera06-s818 ~]#

http://perf.jenkins.couchbase.com/job/hera_backup_restore/1042/consoleFull

Attachments

Issue Links

blocks

MB-19612 4.5.1 Minor Release

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Andrei Baranouski added a comment - 23/Jun/16 5:32 AM

Hi Mike Wiederhold [X], cbbackupwrapper has been excluded in EE?

[22/Jun/2016 23:18:33] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16

[172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16

[172.23.96.123] out: /bin/bash: ./cbbackupwrapper: No such file or directory

[172.23.96.123] out:

[root@hera06-s818 ~]# cd /opt/couchbase/bin

[root@hera06-s818 bin]# ls -la

total 233928

drwxr-xr-x. 5 couchbase couchbase     4096 Jun 22 22:42 .

drwxr-xr-x. 8 couchbase couchbase     4096 Jun 22 22:42 ..

-rwxr-xr-x. 1 couchbase couchbase  8471264 Jun 14 20:40 cbbackupmgr

-rwxr-xr-x. 1 couchbase couchbase      816 Jun 14 20:29 cbbrowse_logs

Andrei Baranouski added a comment - 23/Jun/16 5:32 AM Hi Mike Wiederhold [X] , cbbackupwrapper has been excluded in EE? [22/Jun/2016 23:18:33] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] out: /bin/bash: ./cbbackupwrapper: No such file or directory [172.23.96.123] out: [root@hera06-s818 ~]# cd /opt/couchbase/bin [root@hera06-s818 bin]# ls -la total 233928 drwxr-xr-x. 5 couchbase couchbase 4096 Jun 22 22:42 . drwxr-xr-x. 8 couchbase couchbase 4096 Jun 22 22:42 .. -rwxr-xr-x. 1 couchbase couchbase 8471264 Jun 14 20:40 cbbackupmgr -rwxr-xr-x. 1 couchbase couchbase 816 Jun 14 20:29 cbbrowse_logs

Mike Wiederhold [X] (Inactive) added a comment - 23/Jun/16 10:31 AM

cbbackupwrapper, cbrestorewrapper, cbbackup, and cbrestore are not in the enterprise edition anymore. You will need to install community edition to run tests that use these executables.

Also, I fixed a crash issue last Saturday so please try the latest build. I also need the backup logs if I am going to debug any of these issues and they don't seem to be collected by the perf tests.

Mike Wiederhold [X] (Inactive) added a comment - 23/Jun/16 10:31 AM cbbackupwrapper, cbrestorewrapper, cbbackup, and cbrestore are not in the enterprise edition anymore. You will need to install community edition to run tests that use these executables. Also, I fixed a crash issue last Saturday so please try the latest build. I also need the backup logs if I am going to debug any of these issues and they don't seem to be collected by the perf tests.

Andrei Baranouski added a comment - 24/Jun/16 2:31 AM

HI Mike Wiederhold [X],

in 4.7.0-810 with CE

http://perf.jenkins.couchbase.com/job/hera_backup_restore/1052/console

[172.23.96.123] Executing task 'cbbackup'

[24/Jun/2016 02:11:36] INFO - cbbackup into /data/cbbackup_dir

[172.23.96.123] run: rm -rf /data/cbbackup_dir

[24/Jun/2016 02:11:36] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16

[24/Jun/2016 02:11:37] INFO - Creating new database: ns_serverhera_470-810_a7abucket-1

[24/Jun/2016 02:11:37] INFO - Creating new database: active_taskshera_470-810_a7a

[172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16

[172.23.96.123] out: Traceback (most recent call last):

[172.23.96.123] out:   File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module>

[172.23.96.123] out:     if args.mode:

[172.23.96.123] out: AttributeError: Values instance has no attribute 'mode'

!!! Parallel execution exception under host u'172.23.96.123':

Process 172.23.96.123:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 233, in inner

    submit(task.run(*args, **kwargs))

  File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run

    return self.wrapped(*args, **kwargs)

  File "/tmp/env/local/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner

    return func(*args, **kwargs)

  File "/tmp/env/local/lib/python2.7/site-packages/cbagent/collectors/libstats/iostat.py", line 48, in get_samples

    samples[key] = float(data[metric]) * multiplier

KeyError: 'r/s'

[172.23.96.123] out:

Warning: One or more hosts failed while executing task 'get_samples'

Underlying exception:

    'r/s'

[24/Jun/2016 02:11:38] WARNING - 'exceptions.KeyError' object has no attribute 'keys'

Andrei Baranouski added a comment - 24/Jun/16 2:31 AM HI Mike Wiederhold [X] , in 4.7.0-810 with CE http://perf.jenkins.couchbase.com/job/hera_backup_restore/1052/console [172.23.96.123] Executing task 'cbbackup' [24/Jun/2016 02:11:36] INFO - cbbackup into /data/cbbackup_dir [172.23.96.123] run: rm -rf /data/cbbackup_dir [24/Jun/2016 02:11:36] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [24/Jun/2016 02:11:37] INFO - Creating new database: ns_serverhera_470-810_a7abucket-1 [24/Jun/2016 02:11:37] INFO - Creating new database: active_taskshera_470-810_a7a [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] out: Traceback (most recent call last): [172.23.96.123] out: File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module> [172.23.96.123] out: if args.mode: [172.23.96.123] out: AttributeError: Values instance has no attribute 'mode' !!! Parallel execution exception under host u'172.23.96.123': Process 172.23.96.123: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 233, in inner submit(task.run(*args, **kwargs)) File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run return self.wrapped(*args, **kwargs) File "/tmp/env/local/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner return func(*args, **kwargs) File "/tmp/env/local/lib/python2.7/site-packages/cbagent/collectors/libstats/iostat.py", line 48, in get_samples samples[key] = float(data[metric]) * multiplier KeyError: 'r/s' [172.23.96.123] out: Warning: One or more hosts failed while executing task 'get_samples' Underlying exception: 'r/s' [24/Jun/2016 02:11:38] WARNING - 'exceptions.KeyError' object has no attribute 'keys'

Pavel Paulau (Inactive) added a comment - 07/Jul/16 8:26 PM - edited

This issue happens in Watson as well. The reason is missing "pump_bfd2.py" module.

# rpm -qf /etc/init.d/couchbase-server

couchbase-server-community-4.5.0-2601.x86_64

# /opt/couchbase/bin/cbbackupwrapper  http://172.23.100.29:8091 /workspace/backup -u Administrator -p password

Traceback (most recent call last):

  File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module>

    if args.mode:

AttributeError: Values instance has no attribute 'mode'

This is why "mode" attribute is missing:

    try:

        import pump_bfd2

        parser.add_option("-m", "--mode",

                        action="store", type="string", default="diff",

                        help="backup mode: full, diff or accu [default:%default]")

    except ImportError:

        pass

# find /opt/couchbase -name pump_bfd2.py | wc -l

Pavel Paulau (Inactive) added a comment - 07/Jul/16 8:26 PM - edited This issue happens in Watson as well. The reason is missing "pump_bfd2.py" module. # rpm -qf /etc/init.d/couchbase-server couchbase-server-community-4.5.0-2601.x86_64 # /opt/couchbase/bin/cbbackupwrapper http://172.23.100.29:8091 /workspace/backup -u Administrator -p password Traceback (most recent call last): File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module> if args.mode: AttributeError: Values instance has no attribute 'mode' This is why "mode" attribute is missing: try: import pump_bfd2 parser.add_option("-m", "--mode", action="store", type="string", default="diff", help="backup mode: full, diff or accu [default:%default]") except ImportError: pass # find /opt/couchbase -name pump_bfd2.py | wc -l 0

Wayne Siu added a comment - 27/Jul/16 11:33 AM

Hi Mike,
Do we have an update on this issue? Thanks.

Wayne Siu added a comment - 27/Jul/16 11:33 AM Hi Mike, Do we have an update on this issue? Thanks.

Will Gardner (Inactive) added a comment - 28/Jul/16 1:16 AM - edited

pump_bfd2 is intentionally not included with CE as I believe it's the file which allows incremental backup to work. This would suggest that cbbackupwrapper has never worked on CE.

I believe the fix in this case (To allow cbbackupwrapper to function on CE) would be to change

    mode_options = ''

    if args.mode:

        mode_options = ' -m ' + args.mode

    mode_options = ''

    if hasattr(args, 'mode'):

        mode_options = ' -m ' + args.mode

Edit:

Looks like it was broken by this commit:
https://github.com/couchbase/couchbase-cli/commit/2ff67e7f98f1a9609df7ac0009d9aad54a13ccbf

Will Gardner (Inactive) added a comment - 28/Jul/16 1:16 AM - edited pump_bfd2 is intentionally not included with CE as I believe it's the file which allows incremental backup to work. This would suggest that cbbackupwrapper has never worked on CE. I believe the fix in this case (To allow cbbackupwrapper to function on CE) would be to change mode_options = '' if args.mode: mode_options = ' -m ' + args.mode to mode_options = '' if hasattr (args, 'mode' ): mode_options = ' -m ' + args.mode Edit: Looks like it was broken by this commit: https://github.com/couchbase/couchbase-cli/commit/2ff67e7f98f1a9609df7ac0009d9aad54a13ccbf

Thuan Nguyen added a comment - 19/Aug/16 1:20 PM

Tested on 4.5.1-2817, I could not reproduce issue with cbbackupwrapper

root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackupwrapper  http://12.11.10.132:8091 /tmp/backup -u Administrator -p password

Waiting for the backup to complete...

SUCCESSFULLY COMPLETED!

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/

0-99  1000-1023  100-199  200-299  300-399  400-499  500-599  600-699  700-799  800-899  900-999  logs

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/

2016-08-19T201459Z

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/

2016-08-19T201459Z-full

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/

bucket-travel-sample

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/

design.json  index.json  node-12.11.10.132%3A8091

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/node-12.11.10.132%3A8091/

data-0000.cbb  failover.json  seqno.json  snapshot_markers.json

root@ubuntu:/opt/couchbase/bin# more ../etc/runtime.ini

; Configuration Settings at install-time and runtime.  Couchbase may

; modify this file, for example, at runtime when the server is secured

; and taken out of Admin Party mode.

[product]

title = Couchbase Server

version = 4.5.1-2817

license = community

root@ubuntu:/opt/couchbase/bin#

Thuan Nguyen added a comment - 19/Aug/16 1:20 PM Tested on 4.5.1-2817, I could not reproduce issue with cbbackupwrapper root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackupwrapper http://12.11.10.132:8091 /tmp/backup -u Administrator -p password Waiting for the backup to complete... SUCCESSFULLY COMPLETED! root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/ 0-99 1000-1023 100-199 200-299 300-399 400-499 500-599 600-699 700-799 800-899 900-999 logs root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/ 2016-08-19T201459Z root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/ 2016-08-19T201459Z-full root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/ bucket-travel-sample root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/ design.json index.json node-12.11.10.132%3A8091 root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/node-12.11.10.132%3A8091/ data-0000.cbb failover.json seqno.json snapshot_markers.json root@ubuntu:/opt/couchbase/bin# more ../etc/runtime.ini ; Configuration Settings at install-time and runtime. Couchbase may ; modify this file, for example, at runtime when the server is secured ; and taken out of Admin Party mode. [product] title = Couchbase Server version = 4.5.1-2817 license = community root@ubuntu:/opt/couchbase/bin#

Thuan Nguyen added a comment - 19/Aug/16 1:23 PM

Tested on 4.5.1-2817 CE, I could not reproduce issue with cbbackup

root@ubuntu:/opt/couchbase/bin# mkdir /tmp/backup_1

root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackup  http://12.11.10.132:8091 /tmp/backup_1/ -u Administrator -p password

  [############        ] 58.2% (31591/estimated 54305 msgs)

bucket: travel-sample, msgs transferred...

       :                total |       last |    per sec

 byte  :             36185496 |   36185496 |  6925545.2

done

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/

2016-08-19T202140Z

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/

2016-08-19T202140Z-full

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/

bucket-travel-sample

root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/bucket-travel-sample/

design.json  index.json  node-12.11.10.132%3A8091

root@ubuntu:/opt/couchbase/bin#

Thuan Nguyen added a comment - 19/Aug/16 1:23 PM Tested on 4.5.1-2817 CE, I could not reproduce issue with cbbackup root@ubuntu:/opt/couchbase/bin# mkdir /tmp/backup_1 root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackup http://12.11.10.132:8091 /tmp/backup_1/ -u Administrator -p password [############ ] 58.2% (31591/estimated 54305 msgs) bucket: travel-sample, msgs transferred... : total | last | per sec byte : 36185496 | 36185496 | 6925545.2 done root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/ 2016-08-19T202140Z root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/ 2016-08-19T202140Z-full root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/ bucket-travel-sample root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/bucket-travel-sample/ design.json index.json node-12.11.10.132%3A8091 root@ubuntu:/opt/couchbase/bin#

People

Assignee:: Mike Wiederhold [X] (Inactive)

Reporter:: Andrei Baranouski

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 23/Jun/16 5:25 AM

Updated:: 01/Feb/18 3:49 AM

Resolved:: 04/Aug/16 5:16 PM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-20015 Set mode option in cbbackupwrapper on CE: Gerrit Review:

MB-20015 Set mode option in cbbackupwrapper on CE: Gerrit Review:

PagerDuty