Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20015

CE backup doesn't work in Spock and Watson

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 4.5.0, 5.0.0
    • 4.5.1
    • tools
    • 4.7.0-779
    • Untriaged
    • Yes

    Description

      [root@hera06-s818 ~]# rm -rf /data/cbbackup_dir
      [root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr config --archive /data/cbbackup_dir --repo default
      Backup repository `default` created successfully in archive `/data/cbbackup_dir`
      [root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr backup --archive /data/cbbackup_dir --repo default --host http://172.23.96.117:8091 --username Administrator --password password --threads 16
       
      Backing up to 2016-06-23T05_23_14.48075409-07_00
      Copying at 0B (estimating time remaining)                                                                                                                             0 items / 0B
      bucket-1                [                                                                                                                                                ] 100.00%
      [root@hera06-s818 ~]# /opt/couchbase/bin/cbbackupmgr backup --archive /data/cbbackup_dir --repo default --host http://172.23.96.117:8091 --username Administrator --password password --threads 16
      Error backing up cluster: The most recent backup `2016-06-23T05_23_14.48075409-07_00` did not finish properly. You can either resume this backup from where it left off by re-running the backup command and using the --resume flag. To delete the unfinished backup and start again use the --purge flag.
      [root@hera06-s818 ~]# 
      

      http://perf.jenkins.couchbase.com/job/hera_backup_restore/1042/consoleFull

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Hi Mike Wiederhold [X], cbbackupwrapper has been excluded in EE?

            [22/Jun/2016 23:18:33] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 
            [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 
            [172.23.96.123] out: /bin/bash: ./cbbackupwrapper: No such file or directory
            [172.23.96.123] out: 
             
            [root@hera06-s818 ~]# cd /opt/couchbase/bin
            [root@hera06-s818 bin]# ls -la
            total 233928
            drwxr-xr-x. 5 couchbase couchbase     4096 Jun 22 22:42 .
            drwxr-xr-x. 8 couchbase couchbase     4096 Jun 22 22:42 ..
            -rwxr-xr-x. 1 couchbase couchbase  8471264 Jun 14 20:40 cbbackupmgr
            -rwxr-xr-x. 1 couchbase couchbase      816 Jun 14 20:29 cbbrowse_logs
            
            

            andreibaranouski Andrei Baranouski added a comment - Hi Mike Wiederhold [X] , cbbackupwrapper has been excluded in EE? [22/Jun/2016 23:18:33] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] out: /bin/bash: ./cbbackupwrapper: No such file or directory [172.23.96.123] out:   [root@hera06-s818 ~]# cd /opt/couchbase/bin [root@hera06-s818 bin]# ls -la total 233928 drwxr-xr-x. 5 couchbase couchbase 4096 Jun 22 22:42 . drwxr-xr-x. 8 couchbase couchbase 4096 Jun 22 22:42 .. -rwxr-xr-x. 1 couchbase couchbase 8471264 Jun 14 20:40 cbbackupmgr -rwxr-xr-x. 1 couchbase couchbase 816 Jun 14 20:29 cbbrowse_logs

            cbbackupwrapper, cbrestorewrapper, cbbackup, and cbrestore are not in the enterprise edition anymore. You will need to install community edition to run tests that use these executables.

            Also, I fixed a crash issue last Saturday so please try the latest build. I also need the backup logs if I am going to debug any of these issues and they don't seem to be collected by the perf tests.

            mikew Mike Wiederhold [X] (Inactive) added a comment - cbbackupwrapper, cbrestorewrapper, cbbackup, and cbrestore are not in the enterprise edition anymore. You will need to install community edition to run tests that use these executables. Also, I fixed a crash issue last Saturday so please try the latest build. I also need the backup logs if I am going to debug any of these issues and they don't seem to be collected by the perf tests.

            HI Mike Wiederhold [X],

            in 4.7.0-810 with CE

            http://perf.jenkins.couchbase.com/job/hera_backup_restore/1052/console

            [172.23.96.123] Executing task 'cbbackup'
            [24/Jun/2016 02:11:36] INFO - cbbackup into /data/cbbackup_dir
            [172.23.96.123] run: rm -rf /data/cbbackup_dir
            [24/Jun/2016 02:11:36] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 
            [24/Jun/2016 02:11:37] INFO - Creating new database: ns_serverhera_470-810_a7abucket-1
            [24/Jun/2016 02:11:37] INFO - Creating new database: active_taskshera_470-810_a7a
            [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 
            [172.23.96.123] out: Traceback (most recent call last):
            [172.23.96.123] out:   File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module>
            [172.23.96.123] out:     if args.mode:
            [172.23.96.123] out: AttributeError: Values instance has no attribute 'mode'
            !!! Parallel execution exception under host u'172.23.96.123':
            Process 172.23.96.123:
            Traceback (most recent call last):
              File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
                self.run()
              File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
                self._target(*self._args, **self._kwargs)
              File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 233, in inner
                submit(task.run(*args, **kwargs))
              File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run
                return self.wrapped(*args, **kwargs)
              File "/tmp/env/local/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
                return func(*args, **kwargs)
              File "/tmp/env/local/lib/python2.7/site-packages/cbagent/collectors/libstats/iostat.py", line 48, in get_samples
                samples[key] = float(data[metric]) * multiplier
            KeyError: 'r/s'
            [172.23.96.123] out: 
             
            Warning: One or more hosts failed while executing task 'get_samples'
             
            Underlying exception:
                'r/s'
             
            [24/Jun/2016 02:11:38] WARNING - 'exceptions.KeyError' object has no attribute 'keys'
            

            andreibaranouski Andrei Baranouski added a comment - HI Mike Wiederhold [X] , in 4.7.0-810 with CE http://perf.jenkins.couchbase.com/job/hera_backup_restore/1052/console [172.23.96.123] Executing task 'cbbackup' [24/Jun/2016 02:11:36] INFO - cbbackup into /data/cbbackup_dir [172.23.96.123] run: rm -rf /data/cbbackup_dir [24/Jun/2016 02:11:36] INFO - cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [24/Jun/2016 02:11:37] INFO - Creating new database: ns_serverhera_470-810_a7abucket-1 [24/Jun/2016 02:11:37] INFO - Creating new database: active_taskshera_470-810_a7a [172.23.96.123] run: cd /opt/couchbase/bin && ./cbbackupwrapper http://172.23.96.117:8091 /data/cbbackup_dir -u Administrator -p password -P 16 [172.23.96.123] out: Traceback (most recent call last): [172.23.96.123] out: File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module> [172.23.96.123] out: if args.mode: [172.23.96.123] out: AttributeError: Values instance has no attribute 'mode' !!! Parallel execution exception under host u'172.23.96.123': Process 172.23.96.123: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 233, in inner submit(task.run(*args, **kwargs)) File "/tmp/env/local/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run return self.wrapped(*args, **kwargs) File "/tmp/env/local/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner return func(*args, **kwargs) File "/tmp/env/local/lib/python2.7/site-packages/cbagent/collectors/libstats/iostat.py", line 48, in get_samples samples[key] = float(data[metric]) * multiplier KeyError: 'r/s' [172.23.96.123] out:   Warning: One or more hosts failed while executing task 'get_samples'   Underlying exception: 'r/s'   [24/Jun/2016 02:11:38] WARNING - 'exceptions.KeyError' object has no attribute 'keys'
            pavelpaulau Pavel Paulau (Inactive) added a comment - - edited

            This issue happens in Watson as well. The reason is missing "pump_bfd2.py" module.

            # rpm -qf /etc/init.d/couchbase-server 
            couchbase-server-community-4.5.0-2601.x86_64
            

            # /opt/couchbase/bin/cbbackupwrapper  http://172.23.100.29:8091 /workspace/backup -u Administrator -p password 
            Traceback (most recent call last):
              File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module>
                if args.mode:
            AttributeError: Values instance has no attribute 'mode'
            
            

            This is why "mode" attribute is missing:

                try:
                    import pump_bfd2
                    parser.add_option("-m", "--mode",
                                    action="store", type="string", default="diff",
                                    help="backup mode: full, diff or accu [default:%default]")
                except ImportError:
                    pass
            

            # find /opt/couchbase -name pump_bfd2.py | wc -l
            0
            

            pavelpaulau Pavel Paulau (Inactive) added a comment - - edited This issue happens in Watson as well. The reason is missing "pump_bfd2.py" module. # rpm -qf /etc/init.d/couchbase-server couchbase-server-community-4.5.0-2601.x86_64 # /opt/couchbase/bin/cbbackupwrapper http://172.23.100.29:8091 /workspace/backup -u Administrator -p password Traceback (most recent call last): File "/opt/couchbase/lib/python/cbbackupwrapper", line 343, in <module> if args.mode: AttributeError: Values instance has no attribute 'mode' This is why "mode" attribute is missing: try: import pump_bfd2 parser.add_option("-m", "--mode", action="store", type="string", default="diff", help="backup mode: full, diff or accu [default:%default]") except ImportError: pass # find /opt/couchbase -name pump_bfd2.py | wc -l 0
            wayne Wayne Siu added a comment -

            Hi Mike,
            Do we have an update on this issue? Thanks.

            wayne Wayne Siu added a comment - Hi Mike, Do we have an update on this issue? Thanks.
            will.gardner Will Gardner (Inactive) added a comment - - edited

            pump_bfd2 is intentionally not included with CE as I believe it's the file which allows incremental backup to work. This would suggest that cbbackupwrapper has never worked on CE.

            I believe the fix in this case (To allow cbbackupwrapper to function on CE) would be to change

                mode_options = ''
                if args.mode:
                    mode_options = ' -m ' + args.mode
            

            to

                mode_options = ''
                if hasattr(args, 'mode'):
                    mode_options = ' -m ' + args.mode
            

            Edit:

            Looks like it was broken by this commit:
            https://github.com/couchbase/couchbase-cli/commit/2ff67e7f98f1a9609df7ac0009d9aad54a13ccbf

            will.gardner Will Gardner (Inactive) added a comment - - edited pump_bfd2 is intentionally not included with CE as I believe it's the file which allows incremental backup to work. This would suggest that cbbackupwrapper has never worked on CE. I believe the fix in this case (To allow cbbackupwrapper to function on CE) would be to change mode_options = '' if args.mode: mode_options = ' -m ' + args.mode to mode_options = '' if hasattr (args, 'mode' ): mode_options = ' -m ' + args.mode Edit: Looks like it was broken by this commit: https://github.com/couchbase/couchbase-cli/commit/2ff67e7f98f1a9609df7ac0009d9aad54a13ccbf
            thuan Thuan Nguyen added a comment -

            Tested on 4.5.1-2817, I could not reproduce issue with cbbackupwrapper

            root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackupwrapper  http://12.11.10.132:8091 /tmp/backup -u Administrator -p password
            Waiting for the backup to complete...
            SUCCESSFULLY COMPLETED!
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/
            0-99  1000-1023  100-199  200-299  300-399  400-499  500-599  600-699  700-799  800-899  900-999  logs
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/
            2016-08-19T201459Z
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/
            2016-08-19T201459Z-full
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/
            bucket-travel-sample
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/
            design.json  index.json  node-12.11.10.132%3A8091
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/node-12.11.10.132%3A8091/
            data-0000.cbb  failover.json  seqno.json  snapshot_markers.json
             
            root@ubuntu:/opt/couchbase/bin# more ../etc/runtime.ini 
            ; Configuration Settings at install-time and runtime.  Couchbase may
            ; modify this file, for example, at runtime when the server is secured
            ; and taken out of Admin Party mode.
             
            [product]
            title = Couchbase Server
            version = 4.5.1-2817
            license = community
             
            root@ubuntu:/opt/couchbase/bin#
            

            thuan Thuan Nguyen added a comment - Tested on 4.5.1-2817, I could not reproduce issue with cbbackupwrapper root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackupwrapper http://12.11.10.132:8091 /tmp/backup -u Administrator -p password Waiting for the backup to complete... SUCCESSFULLY COMPLETED! root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/ 0-99 1000-1023 100-199 200-299 300-399 400-499 500-599 600-699 700-799 800-899 900-999 logs root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/ 2016-08-19T201459Z root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/ 2016-08-19T201459Z-full root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/ bucket-travel-sample root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/ design.json index.json node-12.11.10.132%3A8091 root@ubuntu:/opt/couchbase/bin# ls /tmp/backup/0-99/2016-08-19T201459Z/2016-08-19T201459Z-full/bucket-travel-sample/node-12.11.10.132%3A8091/ data-0000.cbb failover.json seqno.json snapshot_markers.json   root@ubuntu:/opt/couchbase/bin# more ../etc/runtime.ini ; Configuration Settings at install-time and runtime. Couchbase may ; modify this file, for example, at runtime when the server is secured ; and taken out of Admin Party mode.   [product] title = Couchbase Server version = 4.5.1-2817 license = community   root@ubuntu:/opt/couchbase/bin#
            thuan Thuan Nguyen added a comment -

            Tested on 4.5.1-2817 CE, I could not reproduce issue with cbbackup

            root@ubuntu:/opt/couchbase/bin# mkdir /tmp/backup_1
            root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackup  http://12.11.10.132:8091 /tmp/backup_1/ -u Administrator -p password
              [############        ] 58.2% (31591/estimated 54305 msgs)
            bucket: travel-sample, msgs transferred...
                   :                total |       last |    per sec
             byte  :             36185496 |   36185496 |  6925545.2
            done
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/
            2016-08-19T202140Z
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/
            2016-08-19T202140Z-full
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/
            bucket-travel-sample
            root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/bucket-travel-sample/
            design.json  index.json  node-12.11.10.132%3A8091
            root@ubuntu:/opt/couchbase/bin# 
            

            thuan Thuan Nguyen added a comment - Tested on 4.5.1-2817 CE, I could not reproduce issue with cbbackup root@ubuntu:/opt/couchbase/bin# mkdir /tmp/backup_1 root@ubuntu:/opt/couchbase/bin# /opt/couchbase/bin/cbbackup http://12.11.10.132:8091 /tmp/backup_1/ -u Administrator -p password [############ ] 58.2% (31591/estimated 54305 msgs) bucket: travel-sample, msgs transferred... : total | last | per sec byte : 36185496 | 36185496 | 6925545.2 done root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/ 2016-08-19T202140Z root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/ 2016-08-19T202140Z-full root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/ bucket-travel-sample root@ubuntu:/opt/couchbase/bin# ls /tmp/backup_1/2016-08-19T202140Z/2016-08-19T202140Z-full/bucket-travel-sample/ design.json index.json node-12.11.10.132%3A8091 root@ubuntu:/opt/couchbase/bin#

            People

              mikew Mike Wiederhold [X] (Inactive)
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty