Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8171

centos32: impossible to start server after upgrade on 2.0.2

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.1.0
    • 2.1.0
    • None
    • Security Level: Public
    • None
    • 2.0.2-780, centos32
    • Centos 32-bit

    Description

      http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P0/61/consoleFull

      for example
      upgrade 1.8.0r-55->2.0.2-780

      2013-04-30 01:57:39,076 - root - INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_2.0.2-780-rel.rpm
      2013-04-30 01:58:03,100 - root - INFO - command executed successfully
      2013-04-30 01:58:03,100 - root - INFO - Stopping couchbase-server ...
      2013-04-30 01:58:03,100 - root - INFO - Stopping couchbase-server[ OK ]
      2013-04-30 01:58:03,101 - root - INFO -
      2013-04-30 01:58:03,101 - root - INFO - Minimum RAM required : 4 GB
      2013-04-30 01:58:03,101 - root - INFO - System RAM configured : 4096184 kB
      2013-04-30 01:58:03,101 - root - INFO -
      2013-04-30 01:58:03,101 - root - INFO - Minimum number of processors required : 4 cores
      2013-04-30 01:58:03,101 - root - INFO - Number of processors on the system : 4 cores
      2013-04-30 01:58:03,102 - root - INFO -
      2013-04-30 01:58:03,102 - root - INFO - Upgrading couchbase-server ...
      2013-04-30 01:58:03,102 - root - INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
      2013-04-30 01:58:03,102 - root - INFO - Automatic mode: running without interactive questions or confirmations.
      2013-04-30 01:58:03,102 - root - INFO - Analysing...
      2013-04-30 01:58:03,102 - root - INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
      2013-04-30 01:58:03,102 - root - INFO - Target node: ns_1@127.0.0.1
      2013-04-30 01:58:03,103 - root - INFO - Upgrading your Couchbase Server from 1.8.x to 2.0.2-780-rel.
      2013-04-30 01:58:03,103 - root - INFO - NOTE: all item data will be read from 1.8.x sqlite database files
      2013-04-30 01:58:03,103 - root - INFO - and converted to new files in 2.0 couchstore file format. This can
      2013-04-30 01:58:03,103 - root - INFO - be a lengthy, slow operation for large datasets and/or fragmented
      2013-04-30 01:58:03,103 - root - INFO - database files.
      2013-04-30 01:58:03,103 - root - INFO -
      2013-04-30 01:58:03,104 - root - INFO - Couchbase should not be running.
      2013-04-30 01:58:03,104 - root - INFO - Please use: /etc/init.d/couchbase-server stop
      2013-04-30 01:58:03,104 - root - INFO -
      2013-04-30 01:58:03,104 - root - INFO - Database dir: /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,104 - root - INFO -
      2013-04-30 01:58:03,104 - root - INFO - Buckets to upgrade: default,bucket0,standard_bucket0
      2013-04-30 01:58:03,105 - root - INFO -
      2013-04-30 01:58:03,105 - root - INFO - Checking disk space available for buckets in directory:
      2013-04-30 01:58:03,105 - root - INFO - /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,105 - root - INFO - Free disk bucket space wanted: 7686144.0
      2013-04-30 01:58:03,105 - root - INFO - Free disk bucket space available: 29204688896
      2013-04-30 01:58:03,105 - root - INFO - Free disk space factor: 2.0
      2013-04-30 01:58:03,105 - root - INFO - Ok.
      2013-04-30 01:58:03,106 - root - INFO -
      2013-04-30 01:58:03,106 - root - INFO - Analysis complete.
      2013-04-30 01:58:03,106 - root - INFO -
      2013-04-30 01:58:03,106 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat
      2013-04-30 01:58:03,107 - root - INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
      2013-04-30 01:58:03,107 - root - INFO - Ensuring bucket data directories.
      2013-04-30 01:58:03,107 - root - INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default
      2013-04-30 01:58:03,107 - root - INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/default
      2013-04-30 01:58:03,107 - root - INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/bucket0
      2013-04-30 01:58:03,108 - root - INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/bucket0
      2013-04-30 01:58:03,108 - root - INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      2013-04-30 01:58:03,108 - root - INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      2013-04-30 01:58:03,108 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,109 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,109 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,109 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
      2013-04-30 01:58:03,109 - root - INFO - Upgrading buckets.
      2013-04-30 01:58:03,109 - root - INFO - Upgrading bucket: default
      2013-04-30 01:58:03,110 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active
      2013-04-30 01:58:03,110 - root - INFO - cbdbupgrade pid: 22040
      2013-04-30 01:58:03,110 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,110 - root - INFO - bucket: default, msgs transferred...
      2013-04-30 01:58:03,111 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,111 - root - INFO - batch : 1025 | 1025 | 2172.9
      2013-04-30 01:58:03,111 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,111 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,111 - root - INFO - done
      2013-04-30 01:58:03,112 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,112 - root - INFO - Upgrading bucket: default
      2013-04-30 01:58:03,112 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=replica --destination-vbucket-state=replica
      2013-04-30 01:58:03,112 - root - INFO - cbdbupgrade pid: 22049
      2013-04-30 01:58:03,113 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,113 - root - INFO - bucket: default, msgs transferred...
      2013-04-30 01:58:03,113 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,113 - root - INFO - batch : 1025 | 1025 | 3510.0
      2013-04-30 01:58:03,114 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,114 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,114 - root - INFO - done
      2013-04-30 01:58:03,114 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,114 - root - INFO - Upgrading bucket: default
      2013-04-30 01:58:03,115 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=pending --destination-vbucket-state=pending
      2013-04-30 01:58:03,115 - root - INFO - cbdbupgrade pid: 22058
      2013-04-30 01:58:03,115 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,115 - root - INFO - bucket: default, msgs transferred...
      2013-04-30 01:58:03,116 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,116 - root - INFO - batch : 1025 | 1025 | 2803.8
      2013-04-30 01:58:03,116 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,116 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,116 - root - INFO - done
      2013-04-30 01:58:03,117 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,117 - root - INFO - Upgrading bucket: default
      2013-04-30 01:58:03,117 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead
      2013-04-30 01:58:03,117 - root - INFO - cbdbupgrade pid: 22067
      2013-04-30 01:58:03,118 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,118 - root - INFO - bucket: default, msgs transferred...
      2013-04-30 01:58:03,118 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,118 - root - INFO - batch : 1025 | 1025 | 3612.0
      2013-04-30 01:58:03,118 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,119 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,119 - root - INFO - done
      2013-04-30 01:58:03,119 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,119 - root - INFO - Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/default
      2013-04-30 01:58:03,120 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/default
      2013-04-30 01:58:03,120 - root - INFO - Upgrading bucket: bucket0
      2013-04-30 01:58:03,120 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/bucket0-data/bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b bucket0 --source-vbucket-state=active --destination-vbucket-state=active
      2013-04-30 01:58:03,120 - root - INFO - cbdbupgrade pid: 22087
      2013-04-30 01:58:03,121 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,121 - root - INFO - bucket: bucket0, msgs transferred...
      2013-04-30 01:58:03,121 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,121 - root - INFO - batch : 1025 | 1025 | 2104.8
      2013-04-30 01:58:03,121 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,122 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,122 - root - INFO - done
      2013-04-30 01:58:03,122 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,122 - root - INFO - Upgrading bucket: bucket0
      2013-04-30 01:58:03,123 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/bucket0-data/bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica
      2013-04-30 01:58:03,123 - root - INFO - cbdbupgrade pid: 22096
      2013-04-30 01:58:03,123 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,123 - root - INFO - bucket: bucket0, msgs transferred...
      2013-04-30 01:58:03,124 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,124 - root - INFO - batch : 1025 | 1025 | 2784.5
      2013-04-30 01:58:03,124 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,124 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,125 - root - INFO - done
      2013-04-30 01:58:03,125 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,125 - root - INFO - Upgrading bucket: bucket0
      2013-04-30 01:58:03,125 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/bucket0-data/bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending
      2013-04-30 01:58:03,125 - root - INFO - cbdbupgrade pid: 22105
      2013-04-30 01:58:03,126 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,126 - root - INFO - bucket: bucket0, msgs transferred...
      2013-04-30 01:58:03,126 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,126 - root - INFO - batch : 1025 | 1025 | 3204.2
      2013-04-30 01:58:03,127 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,127 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,127 - root - INFO - done
      2013-04-30 01:58:03,127 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,128 - root - INFO - Upgrading bucket: bucket0
      2013-04-30 01:58:03,128 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/bucket0-data/bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead
      2013-04-30 01:58:03,128 - root - INFO - cbdbupgrade pid: 22114
      2013-04-30 01:58:03,128 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,129 - root - INFO - bucket: bucket0, msgs transferred...
      2013-04-30 01:58:03,129 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,129 - root - INFO - batch : 1025 | 1025 | 2781.4
      2013-04-30 01:58:03,129 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,129 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,130 - root - INFO - done
      2013-04-30 01:58:03,130 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,130 - root - INFO - Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/bucket0
      2013-04-30 01:58:03,130 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/bucket0
      2013-04-30 01:58:03,131 - root - INFO - Upgrading bucket: standard_bucket0
      2013-04-30 01:58:03,131 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/standard_bucket0-data/standard_bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b standard_bucket0 --source-vbucket-state=active --destination-vbucket-state=active
      2013-04-30 01:58:03,131 - root - INFO - cbdbupgrade pid: 22134
      2013-04-30 01:58:03,131 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,132 - root - INFO - bucket: standard_bucket0, msgs transferred...
      2013-04-30 01:58:03,132 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,132 - root - INFO - batch : 1025 | 1025 | 2157.6
      2013-04-30 01:58:03,132 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,133 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,133 - root - INFO - done
      2013-04-30 01:58:03,133 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,133 - root - INFO - Upgrading bucket: standard_bucket0
      2013-04-30 01:58:03,133 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/standard_bucket0-data/standard_bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b standard_bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica
      2013-04-30 01:58:03,134 - root - INFO - cbdbupgrade pid: 22143
      2013-04-30 01:58:03,134 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,134 - root - INFO - bucket: standard_bucket0, msgs transferred...
      2013-04-30 01:58:03,134 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,135 - root - INFO - batch : 1025 | 1025 | 3525.8
      2013-04-30 01:58:03,135 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,135 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,135 - root - INFO - done
      2013-04-30 01:58:03,136 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,136 - root - INFO - Upgrading bucket: standard_bucket0
      2013-04-30 01:58:03,136 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/standard_bucket0-data/standard_bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b standard_bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending
      2013-04-30 01:58:03,136 - root - INFO - cbdbupgrade pid: 22152
      2013-04-30 01:58:03,137 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,137 - root - INFO - bucket: standard_bucket0, msgs transferred...
      2013-04-30 01:58:03,137 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,137 - root - INFO - batch : 1025 | 1025 | 2804.4
      2013-04-30 01:58:03,138 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,138 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,138 - root - INFO - done
      2013-04-30 01:58:03,138 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,139 - root - INFO - Upgrading bucket: standard_bucket0
      2013-04-30 01:58:03,139 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/standard_bucket0-data/standard_bucket0 couchstore-files:///opt/couchbase/var/lib/couchbase/data -b standard_bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead
      2013-04-30 01:58:03,139 - root - INFO - cbdbupgrade pid: 22161
      2013-04-30 01:58:03,139 - root - INFO - ..............................................................................................................................................................................................................
      2013-04-30 01:58:03,140 - root - INFO - bucket: standard_bucket0, msgs transferred...
      2013-04-30 01:58:03,140 - root - INFO - : total | last | per sec
      2013-04-30 01:58:03,140 - root - INFO - batch : 1025 | 1025 | 2810.6
      2013-04-30 01:58:03,140 - root - INFO - byte : 0 | 0 | 0.0
      2013-04-30 01:58:03,141 - root - INFO - msg : 0 | 0 | 0.0
      2013-04-30 01:58:03,141 - root - INFO - done
      2013-04-30 01:58:03,141 - root - INFO - cbdbupgrade err: 0
      2013-04-30 01:58:03,141 - root - INFO - Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      2013-04-30 01:58:03,141 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/standard_bucket0
      2013-04-30 01:58:03,142 - root - INFO -
      2013-04-30 01:58:03,142 - root - INFO - Done.
      2013-04-30 01:58:03,142 - root - INFO - Starting couchbase-server[ OK ]
      2013-04-30 01:58:03,142 - root - INFO -
      2013-04-30 01:58:03,143 - root - INFO -
      2013-04-30 01:58:03,143 - root - INFO - You have successfully installed Couchbase Server.
      2013-04-30 01:58:03,143 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server.
      2013-04-30 01:58:03,143 - root - INFO - Please refer to http://couchbase.com for additional resources.
      2013-04-30 01:58:03,144 - root - INFO -
      2013-04-30 01:58:03,144 - root - INFO - Please note that you have to update your firewall configuration to
      2013-04-30 01:58:03,144 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369,
      2013-04-30 01:58:03,145 - root - INFO - 8091, 8092 and from 21100 to 21299.
      2013-04-30 01:58:03,145 - root - INFO -
      2013-04-30 01:58:03,145 - root - INFO - By using this software you agree to the End User License Agreement.
      2013-04-30 01:58:03,145 - root - INFO - See /opt/couchbase/LICENSE.txt.
      2013-04-30 01:58:03,146 - root - INFO -
      2013-04-30 01:58:03,146 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed
      2013-04-30 01:58:03,146 - root - INFO -

      {badrpc,nodedown}

      2013-04-30 01:58:03,146 - root - INFO -
      2013-04-30 01:58:03,146 - root - INFO - upgrade 10.1.4.24 to version 2.0.2-780-rel is completed
      2013-04-30 01:58:03,195 - root - INFO - sleep for 10 secs. .
      2013-04-30 01:58:13,207 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused
      2013-04-30 01:58:14,210 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused ......................

      then I tried to start server from terminal and got;

      [root@localhost ~]# /etc/init.d/couchbase-server status
      couchbase-server is not running
      [root@localhost ~]# /etc/init.d/couchbase-server staке
      Usage: /etc/init.d/couchbase-server

      {start|stop|restart|status}

      [root@localhost ~]# /etc/init.d/couchbase-server start
      Starting couchbase-server [ OK ]
      [root@localhost ~]# /etc/init.d/couchbase-server status
      couchbase-server is not running

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty