Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7341

[2.0-hotfix candidate] Offline upgrade hangs at cbtransfer converting sqlite file to couchstorefile

    Details

      Description

      Have a 4 nodes cluster (http://172.23.96.12:8091). 1 default buckets 40G RAM on each node for the bucket. 200M items (920 G) in bucket. Resident is as low as 8.3%.
      Try to do an offline upgrade from 181 to 2.0 build 1971:

      [root@thor05 ~]# rpm -U couchbase-server-enterprise_x86_64_2.0.0-1971-rel.rpm
      Stopping couchbase-server ...
      Stopping couchbase-server
      Upgrading couchbase-server ...
      /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
      Automatic mode: running without interactive questions or confirmations.
      Upgrading your Couchbase Server to 2.0.0-1971-rel.
      The upgrade process might take awhile.
      Analysing...
      Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
      Target node: ns_1@172.23.96.14

      Couchbase should not be running.
      Please use: /etc/init.d/couchbase-server stop

      Database dir: /data2

      Buckets to upgrade: default

      Checking disk space available for buckets in directory:
      /data2
      Free disk bucket space wanted: 494982651904.0
      Free disk bucket space available: 570048225280
      Free disk space factor: 2.0
      Ok.

      Analysis complete.

      Copying /opt/couchbase/var/lib/couchbase/config/config.dat
      cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
      Copying /opt/couchbase/var/lib/couchbase/ip
      cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip
      Ensuring bucket data directories.
      Ensuring bucket data directory: /data2/default
      mkdir -p /data2/default
      Ensuring dbdir owner/group: /data2
      chown -R couchbase:couchbase /data2
      Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
      chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
      Upgrading buckets.
      Upgrading bucket: default
      /opt/couchbase/bin/install/../cbtransfer /data2/default-data/default couchstore-files:///data2 -b default --source-vbucket-state=active --destination-vbucket-state=active
      cbdbupgrade pid: 14416
      [ ] 0.0% (0/49997137 msgs)

      Offline upgrade hangs

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Chisheng Chisheng Hong (Inactive) created issue -
        Hide
        steve Steve Yen added a comment -

        moved to 2.0.1 per bug-scrub.

        And, also recommend users do online upgrade, especially on huge datasizes and highly fragmented sqlite.

        Show
        steve Steve Yen added a comment - moved to 2.0.1 per bug-scrub. And, also recommend users do online upgrade, especially on huge datasizes and highly fragmented sqlite.
        steve Steve Yen made changes -
        Field Original Value New Value
        Labels 2.0-release-notes
        Fix Version/s 2.0.1 [ 10399 ]
        Fix Version/s 2.0 [ 10114 ]
        Hide
        bcui Bin Cui (Inactive) added a comment -

        http://review.couchbase.org/#/c/23073/

        Further observation:
        1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files.
        2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items.

        Show
        bcui Bin Cui (Inactive) added a comment - http://review.couchbase.org/#/c/23073/ Further observation: 1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files. 2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items.
        Hide
        bcui Bin Cui (Inactive) added a comment -

        http://review.couchbase.org/#/c/23073/

        Further observation:
        1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files.
        2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items.

        Show
        bcui Bin Cui (Inactive) added a comment - http://review.couchbase.org/#/c/23073/ Further observation: 1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files. 2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items.
        Hide
        kzeller kzeller added a comment -

        Added to RN as: Performing an upgrade from Couchbase 1.8.1 to Couchbase 2.0.0
        GA build freezes while transfering data from SQLite files
        to Couchstore files using <command>cbtransfer</command>.

        Show
        kzeller kzeller added a comment - Added to RN as: Performing an upgrade from Couchbase 1.8.1 to Couchbase 2.0.0 GA build freezes while transfering data from SQLite files to Couchstore files using <command>cbtransfer</command>.
        Hide
        steve Steve Yen added a comment -

        Hmmm, looks like I put the wrong commit msg on this change, but it went in anyways...

        just a warning msg that cbupgrade could take awhile, asked by Farshid: http://review.couchbase.org/23112

        Show
        steve Steve Yen added a comment - Hmmm, looks like I put the wrong commit msg on this change, but it went in anyways... just a warning msg that cbupgrade could take awhile, asked by Farshid: http://review.couchbase.org/23112
        Hide
        bcui Bin Cui (Inactive) added a comment -

        One one setup to upgrade 50m items, it takes:

        [root@thor08 data]# time /opt/couchbase/bin/install/../cbtransfer /data/thor05/default-data/default couchstore-files:///data/test
        da default --source-vbucket-state=active --destination-vbucket-state=active
        ###### 33.6% (16797698/49997137 msgs))
        ################### 100.0% (49997137/49997137 msgs)
        bucket: default, msgs transferred...
        : total | last | per sec
        batch : 294023 | 294023 | 9.9
        byte : 117679410715 | 117679410715 | 3971062.9
        msg : 49997137 | 49997137 | 1687.1
        done

        real 493m54.341s
        user 177m43.942s
        sys 30m50.346s

        Show
        bcui Bin Cui (Inactive) added a comment - One one setup to upgrade 50m items, it takes: [root@thor08 data] # time /opt/couchbase/bin/install/../cbtransfer /data/thor05/default-data/default couchstore-files:///data/test da default --source-vbucket-state=active --destination-vbucket-state=active ###### 33.6% (16797698/49997137 msgs)) ################### 100.0% (49997137/49997137 msgs) bucket: default, msgs transferred... : total | last | per sec batch : 294023 | 294023 | 9.9 byte : 117679410715 | 117679410715 | 3971062.9 msg : 49997137 | 49997137 | 1687.1 done real 493m54.341s user 177m43.942s sys 30m50.346s
        farshid Farshid Ghods (Inactive) made changes -
        Labels 2.0-release-notes 2.0-hotfix 2.0-release-notes
        farshid Farshid Ghods (Inactive) made changes -
        Summary Offline upgrade hangs at cbtransfer converting sqlite file to couchstorefile [2.0-hotfix candidate] Offline upgrade hangs at cbtransfer converting sqlite file to couchstorefile
        steve Steve Yen made changes -
        Sprint Priority 1
        bcui Bin Cui (Inactive) made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        plabee Phil Labee (Inactive) made changes -
        Attachment couchbase-server-2.0.0-HOTFIX-MB-7341.zip [ 16055 ]
        plabee Phil Labee (Inactive) made changes -
        Link This issue is duplicated by CBQE-892 [ CBQE-892 ]
        Hide
        plabee Phil Labee (Inactive) added a comment -

        Attached hotfix zipfile: couchbase-server-2.0.0-HOTFIX-MB-7341.zip

        Show
        plabee Phil Labee (Inactive) added a comment - Attached hotfix zipfile: couchbase-server-2.0.0-HOTFIX- MB-7341 .zip
        steve Steve Yen made changes -
        Sprint Status Current Sprint
        Sprint Priority 1
        frank Frank Weigel (Inactive) made changes -
        Attachment couchbase-server-2.0.0-HOTFIX-MB-7341.zip [ 16055 ]

          People

          • Assignee:
            bcui Bin Cui (Inactive)
            Reporter:
            Chisheng Chisheng Hong (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes