Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7240

tools - recovery of 2.0 couchstore files corrupts item flags

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.0
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None

      Description

      Based on the learnings from MB-7229, the "data recovery" use case also has a bug. cbtransfer will recover corrupted item flags from couchstore files, as flags are actually stored in network-byte-order in couchstore files.

      (other item fields (like expiration?) should also be checked and fixed for endian-ness corruption while we're at it.)

      For example...

      I set an item of key "a", flag 100001, value "hello" into Couchbase 2.0.0. Then, try to recover it via cbtransfer's support of direct couchstore file access. You can see the flag 100001 is corrupted (network byte order conversion bug)...

      root@ubuntu1104-64:~# /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data stdout:
      set a 2709913856 0 5
      hello
      .
      bucket: default, msgs transferred...
      : total | last | per sec
      batch : 1 | 1 | 8.3
      byte : 5 | 5 | 41.4
      msg : 1 | 1 | 8.3
      2012-11-21 10:02:24,892: mt warning: cannot save bucket design on a stdout destination
      done

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        Related, upgrading from 1.8.1 to 2.0.0 needs a double-check on whether item flags are corrupted due to endianess issues – probably worth a separate bug.

        Show
        steve Steve Yen added a comment - Related, upgrading from 1.8.1 to 2.0.0 needs a double-check on whether item flags are corrupted due to endianess issues – probably worth a separate bug.
        Hide
        bcui Bin Cui added a comment -

        the fix is probably just some socket.ntohl() in pump_sfd.py

        Show
        bcui Bin Cui added a comment - the fix is probably just some socket.ntohl() in pump_sfd.py
        Hide
        steve Steve Yen added a comment -

        Hi Bin,
        This is the issue with the flags that we talked about, where the flags are stored in network byte order in the couchstore files (pump_sfd.py).

        Show
        steve Steve Yen added a comment - Hi Bin, This is the issue with the flags that we talked about, where the flags are stored in network byte order in the couchstore files (pump_sfd.py).
        Hide
        steve Steve Yen added a comment -

        bug-scrub - raised to 2.0 blocker as support 'specially requested a way to recover data

        Show
        steve Steve Yen added a comment - bug-scrub - raised to 2.0 blocker as support 'specially requested a way to recover data
        Hide
        bcui Bin Cui added a comment -

        When i test build 1965 which contains fix of MB-7229, looks like flag values in couchstore files are saved in host byte order and this bug is caused by MB-7229. It is fixed already. I will verify cas and expiration field later.

        c:\Program Files\Couchbase\Server\bin>c:\python27\python.exe cbtransfer couchsto
        re-files://..\var\lib\couchbase\data stdout:
        100002 2726691072
        100004 2760245504
        100001 2709913856
        100000 2693136640
        100003 2743468288
        set bin2 100002 0 11
        hello world
        set bin4 100004 0 11
        hello world
        set bin1 100001 0 11
        hello world
        set bin0 100000 0 11
        hello world
        set bin3 100003 0 11
        hello world

        Show
        bcui Bin Cui added a comment - When i test build 1965 which contains fix of MB-7229 , looks like flag values in couchstore files are saved in host byte order and this bug is caused by MB-7229 . It is fixed already. I will verify cas and expiration field later. c:\Program Files\Couchbase\Server\bin>c:\python27\python.exe cbtransfer couchsto re-files://..\var\lib\couchbase\data stdout: 100002 2726691072 100004 2760245504 100001 2709913856 100000 2693136640 100003 2743468288 set bin2 100002 0 11 hello world set bin4 100004 0 11 hello world set bin1 100001 0 11 hello world set bin0 100000 0 11 hello world set bin3 100003 0 11 hello world
        Hide
        steve Steve Yen added a comment -

        I just tried on linux (2.0.0-1956 (RC1 build)), which also has the ep-engine flags corruption fix (MB-7229) and it's definitely coming back corrupted. The item "a" flags that I used was 100001.

        1. /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data stdout:
          set a 2709913856 0 5
          hello
        Show
        steve Steve Yen added a comment - I just tried on linux (2.0.0-1956 (RC1 build)), which also has the ep-engine flags corruption fix ( MB-7229 ) and it's definitely coming back corrupted. The item "a" flags that I used was 100001. /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data stdout: set a 2709913856 0 5 hello
        Hide
        steve Steve Yen added a comment -

        Just tried on 2.0.0-1965 (first build after RC1), and it (strangely, to me) works correctly...

        root@ubuntu1104-64:~# /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data stdout:
        set a 100001 0 5
        hello
        .
        bucket: default, msgs transferred...
        : total | last | per sec
        batch : 1 | 1 | 8.3
        byte : 5 | 5 | 41.3
        msg : 1 | 1 | 8.3
        2012-11-26 09:20:31,627: mt warning: cannot save bucket design on a stdout destination
        done
        root@ubuntu1104-64:~# cat /opt/couchbase/VERSION.txt
        2.0.0-1965-rel

        Show
        steve Steve Yen added a comment - Just tried on 2.0.0-1965 (first build after RC1), and it (strangely, to me) works correctly... root@ubuntu1104-64:~# /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data stdout: set a 100001 0 5 hello . bucket: default, msgs transferred... : total | last | per sec batch : 1 | 1 | 8.3 byte : 5 | 5 | 41.3 msg : 1 | 1 | 8.3 2012-11-26 09:20:31,627: mt warning: cannot save bucket design on a stdout destination done root@ubuntu1104-64:~# cat /opt/couchbase/VERSION.txt 2.0.0-1965-rel
        Hide
        steve Steve Yen added a comment -

        Turned out to be not a bug - the pump_sfd.py code was doing the right struct.unpack() operation all along and I had forgotten that I had written it correctly.

        Also, as Bin pointed out, the MB-7229 fix was not actually in 2.0.0-1956 after all, but is indeed in 2.0.0-1965.

        Show
        steve Steve Yen added a comment - Turned out to be not a bug - the pump_sfd.py code was doing the right struct.unpack() operation all along and I had forgotten that I had written it correctly. Also, as Bin pointed out, the MB-7229 fix was not actually in 2.0.0-1956 after all, but is indeed in 2.0.0-1965.

          People

          • Assignee:
            bcui Bin Cui
            Reporter:
            steve Steve Yen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes