Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3360

mbrestore tool does not support restoring from WAL'ed database files

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.7 alpha 2, 1.8.0
    • Fix Version/s: 1.7 alpha 2
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None

      Description

      This error is presented:
      tcarpenter@sports-mb-test02:/opt/membase/1.6.4.1/data/ns_1]$ /tmp/python271/Python-2.7.1/python /opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore -a default default-0.mb default-1.mb default-2.mb default-3.mb
      Traceback (most recent call last):
      File "/opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore", line 147, in <module>
      main()
      File "/opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore", line 105, in main
      db.executemany("attach ? as ?", zip(db_filenames, attached_dbs))
      sqlite3.OperationalError: file is encrypted or is not a database

      According to Dustin, this is because the mbrestore tool does not currently support WAL db files and the workaround is to run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete' on all the db files you are trying to restore.

      Please add the above workaround to the mbrestore script itself.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        perry Perry Krug created issue -
        Hide
        sean Sean Lynch (Inactive) added a comment -

        I think it'd be even better to require a newer version of sqlite for Python, or ship our own sqlite module or even Python runtime or frozen scripts.

        Show
        sean Sean Lynch (Inactive) added a comment - I think it'd be even better to require a newer version of sqlite for Python, or ship our own sqlite module or even Python runtime or frozen scripts.
        sean Sean Lynch (Inactive) made changes -
        Field Original Value New Value
        Assignee Dustin Sallings [ dustin ]
        Hide
        perry Perry Krug added a comment -

        A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9309485

        Show
        perry Perry Krug added a comment - A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9309485
        Hide
        perry Perry Krug added a comment -

        This is not a python version problem, it's a problem with WAL.

        Show
        perry Perry Krug added a comment - This is not a python version problem, it's a problem with WAL.
        sharon Sharon Barr (Inactive) made changes -
        Assignee Dustin Sallings [ dustin ] Bin Cui [ bcui ]
        Fix Version/s 1.7 beta [ 10110 ]
        sharon Sharon Barr (Inactive) made changes -
        Fix Version/s 1.7 alpha 1 [ 10170 ]
        Fix Version/s 1.7 beta [ 10110 ]
        sharon Sharon Barr (Inactive) made changes -
        Affects Version/s 1.7 Alpha 2 [ 10180 ]
        Affects Version/s 1.6.4.1 [ 10141 ]
        sharon Sharon Barr (Inactive) made changes -
        Fix Version/s 1.7 Alpha 2 [ 10180 ]
        Fix Version/s 1.7 alpha 1 [ 10170 ]
        bcui Bin Cui made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        farshid Farshid Ghods (Inactive) made changes -
        Flagged [Release Note]
        Hide
        perry Perry Krug added a comment -

        Perry Krug deleted the linked story in Pivotal Tracker

        Show
        perry Perry Krug added a comment - Perry Krug deleted the linked story in Pivotal Tracker
        Hide
        mukherjis Sandeep Mukherjee added a comment -

        Running the workaround:

        run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete'

        gives an error:

        sqlite3.exe: Error: too many options: "journal_mode=delete'"
        Use -help for a list of options.

        Show
        mukherjis Sandeep Mukherjee added a comment - Running the workaround: run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete' gives an error: sqlite3.exe: Error: too many options: "journal_mode=delete'" Use -help for a list of options.
        Hide
        perry Perry Krug added a comment -

        This issue should be resolved and the workaround not needed in post 1.7 releases...can you verify that?

        Show
        perry Perry Krug added a comment - This issue should be resolved and the workaround not needed in post 1.7 releases...can you verify that?
        Hide
        mukherjis Sandeep Mukherjee added a comment -

        Well here is my problem. I have data which ranges upto 57 Gig with 11 million entries. The data was generated using the version 1.6.5. If i try to load the data to the new version will it be backward compatible?

        Show
        mukherjis Sandeep Mukherjee added a comment - Well here is my problem. I have data which ranges upto 57 Gig with 11 million entries. The data was generated using the version 1.6.5. If i try to load the data to the new version will it be backward compatible?
        Hide
        perry Perry Krug added a comment -

        There are a few strategies for converting the data. They're not directly compatible, but you can either upgrade the database (http://www.couchbase.org/wiki/display/membase/Membase+Server+1.7) or use our restore tool to pull the data out and replay it into a new cluster (http://www.couchbase.org/wiki/display/membase/Backup+and+Restore+with+Membase). The latter step is where this bug came from, and if you use the latest mbrestore script (provided with the 1.7 installation) you won't need this workaround.

        Show
        perry Perry Krug added a comment - There are a few strategies for converting the data. They're not directly compatible, but you can either upgrade the database ( http://www.couchbase.org/wiki/display/membase/Membase+Server+1.7 ) or use our restore tool to pull the data out and replay it into a new cluster ( http://www.couchbase.org/wiki/display/membase/Backup+and+Restore+with+Membase ). The latter step is where this bug came from, and if you use the latest mbrestore script (provided with the 1.7 installation) you won't need this workaround.
        Hide
        perry Perry Krug added a comment -

        This has been reported on 1.8:
        "When trying to restore from snapshot with cbrestore, it fails with:

        "Either the metadata db file is not specified
        or the backup files need to be upgraded to version 2.
        Please use cbdbupgrade for upgrade or contact support."

        But using cbbackup on snapshot data and restoring afterwards works."

        Show
        perry Perry Krug added a comment - This has been reported on 1.8: "When trying to restore from snapshot with cbrestore, it fails with: "Either the metadata db file is not specified or the backup files need to be upgraded to version 2. Please use cbdbupgrade for upgrade or contact support." But using cbbackup on snapshot data and restoring afterwards works."
        perry Perry Krug made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        perry Perry Krug made changes -
        Affects Version/s 1.8.0 [ 10248 ]
        steve Steve Yen made changes -
        Assignee Bin Cui [ bcui ] Steve Yen [ steve ]
        peter peter made changes -
        Component/s tools [ 10223 ]
        Component/s cli [ 10011 ]
        Hide
        peter peter added a comment -

        Steve, since the workaround did the trick, can this issue be closed?

        Show
        peter peter added a comment - Steve, since the workaround did the trick, can this issue be closed?
        Hide
        steve Steve Yen added a comment -

        Additionally, as part of the workaround, the user should also run "pragma wal_checkpoint(FULL)" to incorporate any WAL files before changing the journal_mode.

        So...

        pragma wal_checkpoint(FULL);
        pragma journal_mode=delete;

        See...

        http://www.sqlite.org/pragma.html#pragma_wal_checkpoint

        One last workaround, possibly already mentioned, is to run cbrestore on a box with updated python and python sqlite3 binding.

        Show
        steve Steve Yen added a comment - Additionally, as part of the workaround, the user should also run "pragma wal_checkpoint(FULL)" to incorporate any WAL files before changing the journal_mode. So... pragma wal_checkpoint(FULL); pragma journal_mode=delete; See... http://www.sqlite.org/pragma.html#pragma_wal_checkpoint One last workaround, possibly already mentioned, is to run cbrestore on a box with updated python and python sqlite3 binding.
        steve Steve Yen made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        steve Steve Yen made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        steve Steve Yen added a comment -

        Related to Perry's April 13, 2012 comment, another workaround to handle the "backup files need to be upgrade to version 2" issue...

        IF the user can determine that the files definitely came from the right version (e.g., backup of 1.8.1 and trying to restore to 1.8.1), then perhaps the user_version pragma information did not transfer correctly.

        For example, use sqlite3 to run...

        pragma user_version;

        The result should be '2'. If not then a workaround fix would be, in sqlite3 to use the following across all the sqlite files...

        pragma user_version=2;

        Show
        steve Steve Yen added a comment - Related to Perry's April 13, 2012 comment, another workaround to handle the "backup files need to be upgrade to version 2" issue... IF the user can determine that the files definitely came from the right version (e.g., backup of 1.8.1 and trying to restore to 1.8.1), then perhaps the user_version pragma information did not transfer correctly. For example, use sqlite3 to run... pragma user_version; The result should be '2'. If not then a workaround fix would be, in sqlite3 to use the following across all the sqlite files... pragma user_version=2;
        steve Steve Yen made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        mikew Mike Wiederhold made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            steve Steve Yen
            Reporter:
            perry Perry Krug
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes