Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7560

upgrade on centos32 1.8.0r -> 2.0.1: cbdbupgrade err: 11

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: installer
    • Security Level: Public
    • Labels:
      None

      Description

      upgrade on enterprise_x86_2.0.1-139-rel

      http://qa.hq.northscale.net/view/2.0.1/job/centos-32-2.0-upgrade/20/consoleFull

      [2013-01-21 01:37:19,024] - [remote_util:1206] INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_2.0.1-139-rel.rpm
      [2013-01-21 01:37:35,844] - [remote_util:1234] INFO - command executed successfully
      [2013-01-21 01:37:35,844] - [remote_util:1183] INFO - Stopping couchbase-server ...
      [2013-01-21 01:37:35,844] - [remote_util:1183] INFO - Stopping couchbase-server[ OK ]
      [2013-01-21 01:37:35,844] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO - Minimum RAM required : 4 GB
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO - System RAM configured : 2457784 kB
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO - Minimum number of processors required : 4 cores
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO - Number of processors on the system : 4 cores
      [2013-01-21 01:37:35,845] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - Upgrading couchbase-server ...
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - Automatic mode: running without interactive questions or confirmations.
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - Analysing...
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
      [2013-01-21 01:37:35,846] - [remote_util:1183] INFO - Target node: ns_1@127.0.0.1
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - Upgrading your Couchbase Server from 1.8.x to 2.0.1-139-rel.
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - NOTE: all item data will be read from 1.8.x sqlite database files
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - and converted to new files in 2.0 couchstore file format. This can
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - be a lengthy, slow operation for large datasets and/or fragmented
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - database files.
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,847] - [remote_util:1183] INFO - Couchbase should not be running.
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO - Please use: /etc/init.d/couchbase-server stop
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO - Database dir: /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO - Buckets to upgrade: default,bucket0,standard_bucket0
      [2013-01-21 01:37:35,848] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - Checking disk space available for buckets in directory:
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - Free disk bucket space wanted: 68997120.0
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - Free disk bucket space available: 30364876800
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - Free disk space factor: 2.0
      [2013-01-21 01:37:35,849] - [remote_util:1183] INFO - Ok.
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO - Analysis complete.
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
      [2013-01-21 01:37:35,850] - [remote_util:1183] INFO - Ensuring bucket data directories.
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/default
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/bucket0
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/bucket0
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      [2013-01-21 01:37:35,851] - [remote_util:1183] INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - Upgrading buckets.
      [2013-01-21 01:37:35,852] - [remote_util:1183] INFO - Upgrading bucket: default
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO - cbdbupgrade pid: 20893
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO - cbdbupgrade err: 11
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO - ERROR: upgrade failed for bucket: default
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO - Starting couchbase-server[ OK ]
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,853] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO - You have successfully installed Couchbase Server.
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO - Please browse to http://localhost.localdomain:8091/ to configure your server.
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO - Please refer to http://couchbase.com for additional resources.
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO - Please note that you have to update your firewall configuration to
      [2013-01-21 01:37:35,854] - [remote_util:1183] INFO - allow connections to the following ports: 11211, 11210, 11209, 4369,
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO - 8091, 8092 and from 21100 to 21299.
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO - By using this software you agree to the End User License Agreement.
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO - See /opt/couchbase/LICENSE.txt.
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO -
      [2013-01-21 01:37:35,855] - [remote_util:1183] INFO - Stopping couchbase-serverNOTE: shutdown failed
      [2013-01-21 01:37:35,856] - [remote_util:1183] INFO -

      {badrpc,nodedown}

      [2013-01-21 01:37:35,856] - [remote_util:1183] INFO -

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        bcui Bin Cui (Inactive) added a comment -

        Cannot reproduce the problem by following steps:
        1. ./testrunner -i /tmp/upgrade.ini upgrade_version=2.0.1-139-rel -t newupgradetests.SingleNodeUpgradeTests.test_upgrade,initial_version=1.8.1-942-rel,standard_buckets=1,items=100,expire_time=360
        1 default, 1 non-sasl bucket
        Just populate data without upgrade

        2. INSTALL_DONT_AUTO_UPGRADE=1 INSTALL_DONT_START_SERVER=1 rpm -U /tmp/couchbase-server-enterprise_x86_2.0.1-139-rel.rpm

        3. /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --sourcevbucket-state=active --destination-vbucket-state=active

        [root@localhost downloads]# /opt/couchbase/bin/install/../cbtransfer /opt/couc
        hbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couch
        base/var/lib/couchbase/data -b default --source-vbucket-state=active --destinat
        ion-vbucket-state=active
        ################### 100.0% (100/100 msgs)
        bucket: default, msgs transferred...
        : total | last | per sec
        batch : 1025 | 1025 | 1055.6
        byte : 51200 | 51200 | 52729.2
        msg : 100 | 100 | 103.0
        done

        I wonder if if it related to expired time. I set it as 360 instead of 180. Maybe we need to set a much longer time for such upgrade test.

        Show
        bcui Bin Cui (Inactive) added a comment - Cannot reproduce the problem by following steps: 1. ./testrunner -i /tmp/upgrade.ini upgrade_version=2.0.1-139-rel -t newupgradetests.SingleNodeUpgradeTests.test_upgrade,initial_version=1.8.1-942-rel,standard_buckets=1,items=100,expire_time=360 1 default, 1 non-sasl bucket Just populate data without upgrade 2. INSTALL_DONT_AUTO_UPGRADE=1 INSTALL_DONT_START_SERVER=1 rpm -U /tmp/couchbase-server-enterprise_x86_2.0.1-139-rel.rpm 3. /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --sourcevbucket-state=active --destination-vbucket-state=active [root@localhost downloads] # /opt/couchbase/bin/install/../cbtransfer /opt/couc hbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couch base/var/lib/couchbase/data -b default --source-vbucket-state=active --destinat ion-vbucket-state=active ################### 100.0% (100/100 msgs) bucket: default, msgs transferred... : total | last | per sec batch : 1025 | 1025 | 1055.6 byte : 51200 | 51200 | 52729.2 msg : 100 | 100 | 103.0 done I wonder if if it related to expired time. I set it as 360 instead of 180. Maybe we need to set a much longer time for such upgrade test.
        Hide
        steve Steve Yen added a comment -

        Hi Bin,
        Please try with 180 (exact params from Andrei) before we call this a can't reproduce.
        Thanks,
        Steve

        Show
        steve Steve Yen added a comment - Hi Bin, Please try with 180 (exact params from Andrei) before we call this a can't reproduce. Thanks, Steve
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        It seems that this problem has been resolved by 144 build. at the same time it is still reproduced without expiration against 139. see execution logs

        Show
        andreibaranouski Andrei Baranouski added a comment - It seems that this problem has been resolved by 144 build. at the same time it is still reproduced without expiration against 139. see execution logs
        Hide
        bcui Bin Cui (Inactive) added a comment -

        No wonder i tested to upgrade to 142 and i could not reproduce the problem with all identical parameters, including setting expired time to 180.

        Maybe i need to test 139 build.

        Show
        bcui Bin Cui (Inactive) added a comment - No wonder i tested to upgrade to 142 and i could not reproduce the problem with all identical parameters, including setting expired time to 180. Maybe i need to test 139 build.
        Hide
        bcui Bin Cui (Inactive) added a comment -

        Since we really didn't check in anything between 139 and 144, just mark the bug as unreproducible.

        Free to reopen it if you see it happening again.

        Show
        bcui Bin Cui (Inactive) added a comment - Since we really didn't check in anything between 139 and 144, just mark the bug as unreproducible. Free to reopen it if you see it happening again.

          People

          • Assignee:
            bcui Bin Cui (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes