Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7817

offline upgrade 1.8.1->2.0.1: impossible to create RestConnection when some nodes are upgraded first: couchApiBase doesn't exist in nodes/self

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: installer, ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      centos/ubuntu 32/64

      Description

      build 164
      run as a example:
      http://qa.hq.northscale.net/job/ubuntu-64-2.0-upgrade/88/consoleFull
      ./testrunner -i /tmp/upgrade.ini upgrade_version=2.0.1-164-rel,initial_vbuckets=64,get-cbcollect-info=True -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=1.8.1-942-rel,nodes_init=3,sasl_buckets=1,standard_buckets=1,GROUP=1_8;OFFLINE

      steps:
      1. 3 1.8.1-942 node in cluster
      2. async upgrade all nodes
      3. after upgrade we are tying to create rest connection:
      request nodes/self to get couchApiBase but it is not there for a while

      in test when we create RestConnection we use loop to get API for 2.* version if something is wrong

      for iteration in xrange(5):
      if "couchApiBase" not in http_res.keys():
      if self.is_cluster_mixed():
      self.capiBaseUrl = self.baseUrl + "/couchBase"
      return
      http_res, success = self.init_http_request(self.baseUrl + 'nodes/self')
      else:
      self.capiBaseUrl = http_res["couchApiBase"]
      return

      I tried to set 5 * 5 seconds but it does not help.

      at the same time, the 'nodes/self' has couchApiBase after all servers were updated( checked it manually)

      I believe that couchApiBase is not set until all the servers in the cluster will not come back after the upgrade

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        andreibaranouski Andrei Baranouski added a comment - - edited

        tried with 2 cluster node

        when first node was upgraded, tried to get couchApiBase 5*5 seconds:

        2013-02-25 17:37:38,876 - root - INFO -

        {badrpc,nodedown}
        2013-02-25 17:37:38,877 - root - INFO -
        2013-02-25 17:37:38,915 - root - ERROR - socket error while connecting to http://172.27.33.4:8091/nodes/self error [Errno 111] Connection refused
        2013-02-25 17:37:39,917 - root - ERROR - socket error while connecting to http://172.27.33.4:8091/nodes/self error [Errno 111] Connection refused
        SLEEP
        Mon Feb 25 17:37:45 2013
        {u'status': u'warmup', u'uptime': u'5', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}}
        SLEEP
        Mon Feb 25 17:37:50 2013
        {u'status': u'warmup', u'uptime': u'9', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}}
        SLEEP
        Mon Feb 25 17:37:55 2013
        {u'status': u'warmup', u'uptime': u'14', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}}
        SLEEP
        Mon Feb 25 17:38:00 2013
        {u'status': u'warmup', u'uptime': u'19', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615504, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.24937655860349128}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615504, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}}
        SLEEP
        2013-02-25 17:38:01,117 - root - INFO - command executed successfully
        2013-02-25 17:38:01,117 - root - INFO - 1d0
        2013-02-25 17:38:01,117 - root - INFO - < efc4b36b015426f01460d6e20f4d81b5 couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm
        2013-02-25 17:38:01,475 - root - INFO - running command.raw sudo cat /proc/cpuinfo
        2013-02-25 17:38:01,588 - root - INFO - command executed successfully
        2013-02-25 17:38:01,588 - root - INFO - running command.raw sudo df -Th
        2013-02-25 17:38:01,696 - root - INFO - command executed successfully
        2013-02-25 17:38:01,696 - root - INFO - running command.raw sudo cat /proc/meminfo
        2013-02-25 17:38:01,808 - root - INFO - command executed successfully
        2013-02-25 17:38:01,808 - root - INFO - running command.raw hostname
        2013-02-25 17:38:01,920 - root - INFO - command executed successfully
        2013-02-25 17:38:01,921 - root - INFO - deliverable_type : rpm
        2013-02-25 17:38:01,921 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm or /tmp/couchbase-server-enterprise
        2013-02-25 17:38:01,921 - root - INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm
        Mon Feb 25 17:38:06 2013
        {u'status': u'warmup', u'uptime': u'24', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615504, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.25}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615504, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}}
        Exception in thread upgrade_thread172.27.33.4:
        Traceback (most recent call last):
        File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
        self.run()
        File "/usr/lib64/python2.6/threading.py", line 484, in run
        self._target(*self.args, **self._kwargs)
        File "pytests/newupgradebasetest.py", line 140, in _upgrade
        raise e
        Exception: couchApiBase doesn't exist in nodes/self: {u'status': u'warmup', u'uptime': u'29', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615408, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.0}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615408, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}}


        upgrade for the second node is still in progress:


        2013-02-25 17:38:21,980 - root - INFO - command executed successfully
        2013-02-25 17:38:21,980 - root - INFO - Stopping couchbase-server ...
        2013-02-25 17:38:21,980 - root - INFO - Stopping couchbase-server
        2013-02-25 17:38:21,980 - root - INFO - Minimum RAM required : 4 GB
        2013-02-25 17:38:21,980 - root - INFO - System RAM configured : 4194304 kB
        2013-02-25 17:38:21,980 - root - INFO -
        2013-02-25 17:38:21,980 - root - INFO - Minimum number of processors required : 4 cores
        2013-02-25 17:38:21,980 - root - INFO - Number of processors on the system : 4 cores
        2013-02-25 17:38:21,980 - root - INFO -
        2013-02-25 17:38:21,980 - root - INFO - Upgrading couchbase-server ...
        2013-02-25 17:38:21,981 - root - INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
        2013-02-25 17:38:21,981 - root - INFO - Automatic mode: running without interactive questions or confirmations.
        2013-02-25 17:38:21,981 - root - INFO - Analysing...
        2013-02-25 17:38:21,981 - root - INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
        2013-02-25 17:38:21,981 - root - INFO - Target node: ns_1@172.27.33.2
        2013-02-25 17:38:21,981 - root - INFO - Upgrading your Couchbase Server from 1.8.x to 2.0.1-164-rel.
        2013-02-25 17:38:21,981 - root - INFO - NOTE: all item data will be read from 1.8.x sqlite database files
        2013-02-25 17:38:21,981 - root - INFO - and converted to new files in 2.0 couchstore file format. This can
        2013-02-25 17:38:21,981 - root - INFO - be a lengthy, slow operation for large datasets and/or fragmented
        2013-02-25 17:38:21,981 - root - INFO - database files.
        2013-02-25 17:38:21,981 - root - INFO -
        2013-02-25 17:38:21,982 - root - INFO - Couchbase should not be running.
        2013-02-25 17:38:21,982 - root - INFO - Please use: /etc/init.d/couchbase-server stop
        2013-02-25 17:38:21,982 - root - INFO -
        2013-02-25 17:38:21,982 - root - INFO - Database dir: /tmp/
        2013-02-25 17:38:21,982 - root - INFO -
        2013-02-25 17:38:21,982 - root - INFO - Buckets to upgrade: default,bucket0,standard_bucket0
        2013-02-25 17:38:21,982 - root - INFO -
        2013-02-25 17:38:21,982 - root - INFO - Checking disk space available for buckets in directory:
        2013-02-25 17:38:21,982 - root - INFO - /tmp/
        2013-02-25 17:38:21,982 - root - INFO - Free disk bucket space wanted: 528384.0
        2013-02-25 17:38:21,982 - root - INFO - Free disk bucket space available: 26522152960
        2013-02-25 17:38:21,983 - root - INFO - Free disk space factor: 2.0
        2013-02-25 17:38:21,983 - root - INFO - Ok.
        2013-02-25 17:38:21,983 - root - INFO -
        2013-02-25 17:38:21,983 - root - INFO - Analysis complete.
        2013-02-25 17:38:21,983 - root - INFO -
        2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat
        2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
        2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/ip
        2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip
        2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/etc/couchdb/local.ini
        2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/etc/couchdb/local.ini /opt/couchbase/bin/install/../../etc/couchdb/local.ini
        2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directories.
        2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//default
        2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//default
        2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//bucket0
        2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//bucket0
        2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//standard_bucket0
        2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//standard_bucket0
        2013-02-25 17:38:21,984 - root - INFO - Ensuring dbdir owner/group: /tmp/
        2013-02-25 17:38:21,984 - root - INFO - chown -R couchbase:couchbase /tmp/
        2013-02-25 17:38:21,984 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
        2013-02-25 17:38:21,984 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
        2013-02-25 17:38:21,984 - root - INFO - Upgrading buckets.
        2013-02-25 17:38:21,985 - root - INFO - Upgrading bucket: default
        2013-02-25 17:38:21,985 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=active --destination-vbucket-state=active
        2013-02-25 17:38:21,985 - root - INFO - cbdbupgrade pid: 2646
        2013-02-25 17:38:21,985 - root - INFO - ..............
        2013-02-25 17:38:21,985 - root - INFO - bucket: default, msgs transferred...
        2013-02-25 17:38:21,985 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,985 - root - INFO - batch : 65 | 65 | 524.0
        2013-02-25 17:38:21,985 - root - INFO - byte : 3072 | 3072 | 24764.0
        2013-02-25 17:38:21,985 - root - INFO - msg : 6 | 6 | 48.4
        2013-02-25 17:38:21,985 - root - INFO - done
        2013-02-25 17:38:21,985 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,986 - root - INFO - Upgrading bucket: default
        2013-02-25 17:38:21,986 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=replica --destination-vbucket-state=replica
        2013-02-25 17:38:21,986 - root - INFO - cbdbupgrade pid: 2655
        2013-02-25 17:38:21,986 - root - INFO - ..............
        2013-02-25 17:38:21,986 - root - INFO - bucket: default, msgs transferred...
        2013-02-25 17:38:21,986 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,986 - root - INFO - batch : 65 | 65 | 1015.0
        2013-02-25 17:38:21,986 - root - INFO - byte : 2048 | 2048 | 31981.1
        2013-02-25 17:38:21,986 - root - INFO - msg : 4 | 4 | 62.5
        2013-02-25 17:38:21,986 - root - INFO - done
        2013-02-25 17:38:21,986 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,987 - root - INFO - Upgrading bucket: default
        2013-02-25 17:38:21,987 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=pending --destination-vbucket-state=pending
        2013-02-25 17:38:21,987 - root - INFO - cbdbupgrade pid: 2664
        2013-02-25 17:38:21,987 - root - INFO - ..............
        2013-02-25 17:38:21,987 - root - INFO - bucket: default, msgs transferred...
        2013-02-25 17:38:21,987 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,987 - root - INFO - batch : 65 | 65 | 1012.6
        2013-02-25 17:38:21,987 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,987 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,987 - root - INFO - done
        2013-02-25 17:38:21,987 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,987 - root - INFO - Upgrading bucket: default
        2013-02-25 17:38:21,988 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=dead --destination-vbucket-state=dead
        2013-02-25 17:38:21,988 - root - INFO - cbdbupgrade pid: 2673
        2013-02-25 17:38:21,988 - root - INFO - ..............
        2013-02-25 17:38:21,988 - root - INFO - bucket: default, msgs transferred...
        2013-02-25 17:38:21,988 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,988 - root - INFO - batch : 65 | 65 | 1012.8
        2013-02-25 17:38:21,988 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,988 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,988 - root - INFO - done
        2013-02-25 17:38:21,988 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,988 - root - INFO - Ensuring bucket owner/group: /tmp//default
        2013-02-25 17:38:21,989 - root - INFO - chown -R couchbase:couchbase /tmp//default
        2013-02-25 17:38:21,989 - root - INFO - Upgrading bucket: bucket0
        2013-02-25 17:38:21,989 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=active --destination-vbucket-state=active
        2013-02-25 17:38:21,989 - root - INFO - cbdbupgrade pid: 2693
        2013-02-25 17:38:21,989 - root - INFO - ..............
        2013-02-25 17:38:21,989 - root - INFO - bucket: bucket0, msgs transferred...
        2013-02-25 17:38:21,989 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,989 - root - INFO - batch : 65 | 65 | 1014.9
        2013-02-25 17:38:21,989 - root - INFO - byte : 3072 | 3072 | 47965.6
        2013-02-25 17:38:21,989 - root - INFO - msg : 6 | 6 | 93.7
        2013-02-25 17:38:21,989 - root - INFO - done
        2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,990 - root - INFO - Upgrading bucket: bucket0
        2013-02-25 17:38:21,990 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica
        2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade pid: 2702
        2013-02-25 17:38:21,990 - root - INFO - ..............
        2013-02-25 17:38:21,990 - root - INFO - bucket: bucket0, msgs transferred...
        2013-02-25 17:38:21,990 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,990 - root - INFO - batch : 65 | 65 | 1015.0
        2013-02-25 17:38:21,990 - root - INFO - byte : 2048 | 2048 | 31981.5
        2013-02-25 17:38:21,990 - root - INFO - msg : 4 | 4 | 62.5
        2013-02-25 17:38:21,990 - root - INFO - done
        2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,991 - root - INFO - Upgrading bucket: bucket0
        2013-02-25 17:38:21,991 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending
        2013-02-25 17:38:21,991 - root - INFO - cbdbupgrade pid: 2711
        2013-02-25 17:38:21,991 - root - INFO - ..............
        2013-02-25 17:38:21,991 - root - INFO - bucket: bucket0, msgs transferred...
        2013-02-25 17:38:21,991 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,991 - root - INFO - batch : 65 | 65 | 1012.7
        2013-02-25 17:38:21,991 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,991 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,991 - root - INFO - done
        2013-02-25 17:38:21,991 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,992 - root - INFO - Upgrading bucket: bucket0
        2013-02-25 17:38:21,992 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead
        2013-02-25 17:38:21,992 - root - INFO - cbdbupgrade pid: 2720
        2013-02-25 17:38:21,992 - root - INFO - ..............
        2013-02-25 17:38:21,992 - root - INFO - bucket: bucket0, msgs transferred...
        2013-02-25 17:38:21,992 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,992 - root - INFO - batch : 65 | 65 | 1015.1
        2013-02-25 17:38:21,992 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,992 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,992 - root - INFO - done
        2013-02-25 17:38:21,992 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,992 - root - INFO - Ensuring bucket owner/group: /tmp//bucket0
        2013-02-25 17:38:21,993 - root - INFO - chown -R couchbase:couchbase /tmp//bucket0
        2013-02-25 17:38:21,993 - root - INFO - Upgrading bucket: standard_bucket0
        2013-02-25 17:38:21,993 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=active --destination-vbucket-state=active
        2013-02-25 17:38:21,993 - root - INFO - cbdbupgrade pid: 2740
        2013-02-25 17:38:21,993 - root - INFO - ..............
        2013-02-25 17:38:21,993 - root - INFO - bucket: standard_bucket0, msgs transferred...
        2013-02-25 17:38:21,993 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,993 - root - INFO - batch : 65 | 65 | 524.0
        2013-02-25 17:38:21,993 - root - INFO - byte : 3072 | 3072 | 24767.4
        2013-02-25 17:38:21,993 - root - INFO - msg : 6 | 6 | 48.4
        2013-02-25 17:38:21,993 - root - INFO - done
        2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,994 - root - INFO - Upgrading bucket: standard_bucket0
        2013-02-25 17:38:21,994 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica
        2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade pid: 2749
        2013-02-25 17:38:21,994 - root - INFO - ..............
        2013-02-25 17:38:21,994 - root - INFO - bucket: standard_bucket0, msgs transferred...
        2013-02-25 17:38:21,994 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,994 - root - INFO - batch : 65 | 65 | 1012.7
        2013-02-25 17:38:21,994 - root - INFO - byte : 2048 | 2048 | 31908.3
        2013-02-25 17:38:21,994 - root - INFO - msg : 4 | 4 | 62.3
        2013-02-25 17:38:21,994 - root - INFO - done
        2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,995 - root - INFO - Upgrading bucket: standard_bucket0
        2013-02-25 17:38:21,995 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending
        2013-02-25 17:38:21,995 - root - INFO - cbdbupgrade pid: 2758
        2013-02-25 17:38:21,995 - root - INFO - ..............
        2013-02-25 17:38:21,995 - root - INFO - bucket: standard_bucket0, msgs transferred...
        2013-02-25 17:38:21,995 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,995 - root - INFO - batch : 65 | 65 | 1012.8
        2013-02-25 17:38:21,995 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,995 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,995 - root - INFO - done
        2013-02-25 17:38:21,995 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,996 - root - INFO - Upgrading bucket: standard_bucket0
        2013-02-25 17:38:21,996 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead
        2013-02-25 17:38:21,996 - root - INFO - cbdbupgrade pid: 2767
        2013-02-25 17:38:21,996 - root - INFO - ..............
        2013-02-25 17:38:21,996 - root - INFO - bucket: standard_bucket0, msgs transferred...
        2013-02-25 17:38:21,996 - root - INFO - : total | last | per sec
        2013-02-25 17:38:21,996 - root - INFO - batch : 65 | 65 | 1012.9
        2013-02-25 17:38:21,996 - root - INFO - byte : 0 | 0 | 0.0
        2013-02-25 17:38:21,996 - root - INFO - msg : 0 | 0 | 0.0
        2013-02-25 17:38:21,996 - root - INFO - done
        2013-02-25 17:38:21,996 - root - INFO - cbdbupgrade err: 0
        2013-02-25 17:38:21,997 - root - INFO - Ensuring bucket owner/group: /tmp//standard_bucket0
        2013-02-25 17:38:21,997 - root - INFO - chown -R couchbase:couchbase /tmp//standard_bucket0
        2013-02-25 17:38:21,997 - root - INFO -
        2013-02-25 17:38:21,997 - root - INFO - Done.
        2013-02-25 17:38:21,997 - root - INFO - Starting couchbase-server[ OK ]
        2013-02-25 17:38:21,997 - root - INFO -
        2013-02-25 17:38:21,997 - root - INFO -
        2013-02-25 17:38:21,997 - root - INFO - You have successfully installed Couchbase Server.
        2013-02-25 17:38:21,997 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server.
        2013-02-25 17:38:21,997 - root - INFO - Please refer to http://couchbase.com for additional resources.
        2013-02-25 17:38:21,997 - root - INFO -
        2013-02-25 17:38:21,997 - root - INFO - Please note that you have to update your firewall configuration to
        2013-02-25 17:38:21,998 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369,
        2013-02-25 17:38:21,998 - root - INFO - 8091, 8092 and from 21100 to 21299.
        2013-02-25 17:38:21,998 - root - INFO -
        2013-02-25 17:38:21,998 - root - INFO - By using this software you agree to the End User License Agreement.
        2013-02-25 17:38:21,998 - root - INFO - See /opt/couchbase/LICENSE.txt.
        2013-02-25 17:38:21,998 - root - INFO -
        2013-02-25 17:38:21,998 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed
        2013-02-25 17:38:21,998 - root - INFO - {badrpc,nodedown}

        2013-02-25 17:38:21,998 - root - INFO -

        got couchApiBase from the third attempt for the second node( after 10 seconds)

        2013-02-25 17:38:22,038 - root - ERROR - socket error while connecting to http://172.27.33.2:8091/nodes/self error [Errno 111] Connection refused
        2013-02-25 17:38:23,040 - root - ERROR - socket error while connecting to http://172.27.33.2:8091/nodes/self error [Errno 111] Connection refused
        SLEEP
        Mon Feb 25 17:38:29 2013
        {u'status': u'warmup', u'uptime': u'4', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2405048320, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.2:8091', u'storage': {u'ssd': [], u'hdd': [

        {u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}

        ]}, u'storageTotals': {u'hdd':

        {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}

        , u'ram': {u'used': 1889918976, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13775768, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.2', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [

        {u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}

        ,

        {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}

        ,

        {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}

        ]}, u'systemStats':

        {u'swap_total': 4959834112, u'swap_used': 120999936, u'cpu_utilization_rate': 5.2631578947368425}

        , u'os': u'x86_64-unknown-linux-gnu', u'ports':

        {u'proxy': 11211, u'direct': 11210}

        , u'interestingStats': {u'vb_replica_curr_items': 12, u'mem_used': 13775768, u'curr_items_tot': 30, u'curr_items': 18}}
        SLEEP
        Mon Feb 25 17:38:34 2013
        {u'status': u'warmup', u'uptime': u'9', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2405048320, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.2:8091', u'storage': {u'ssd': [], u'hdd': [

        {u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}

        ]}, u'storageTotals': {u'hdd':

        {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 146556, u'used': 3778554101}

        , u'ram': {u'used': 1889918976, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1613440, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.2', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [

        {u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}

        ,

        {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}

        ,

        {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}

        ]}, u'systemStats':

        {u'swap_total': 4959834112, u'swap_used': 120999936, u'cpu_utilization_rate': 0.0}

        , u'os': u'x86_64-unknown-linux-gnu', u'ports':

        {u'proxy': 11211, u'direct': 11210}

        , u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 146556, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1613440, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 133802}}
        2013-02-25 17:38:34,099 - root - INFO - settings/web params on 172.27.33.2:8091 :username=Administrator&password=password&port=8091
        2013-02-25 17:38:34,102 - root - INFO - sleep for 10 secs. ...
        FAIL
        2013-02-25 17:38:44,124 - root - WARNING - CLEANUP WAS SKIPPED

        ======================================================================
        FAIL: offline_cluster_upgrade (newupgradetests.MultiNodesUpgradeTests)
        ----------------------------------------------------------------------
        Traceback (most recent call last):
        File "pytests/newupgradetests.py", line 80, in offline_cluster_upgrade
        self.fail("Upgrade failed!")
        AssertionError: Upgrade failed!

        ----------------------------------------------------------------------
        Ran 1 test in 862.160s

        FAILED (failures=1)
        summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1

        https://s3.amazonaws.com/bugdb/jira/MB-7817/172.27.33.2-2252013-1752-diag.zip
        https://s3.amazonaws.com/bugdb/jira/MB-7817/172.27.33.4-2252013-1753-diag.zip

        Show
        andreibaranouski Andrei Baranouski added a comment - - edited tried with 2 cluster node when first node was upgraded, tried to get couchApiBase 5*5 seconds: 2013-02-25 17:37:38,876 - root - INFO - {badrpc,nodedown} 2013-02-25 17:37:38,877 - root - INFO - 2013-02-25 17:37:38,915 - root - ERROR - socket error while connecting to http://172.27.33.4:8091/nodes/self error [Errno 111] Connection refused 2013-02-25 17:37:39,917 - root - ERROR - socket error while connecting to http://172.27.33.4:8091/nodes/self error [Errno 111] Connection refused SLEEP Mon Feb 25 17:37:45 2013 {u'status': u'warmup', u'uptime': u'5', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}} SLEEP Mon Feb 25 17:37:50 2013 {u'status': u'warmup', u'uptime': u'9', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}} SLEEP Mon Feb 25 17:37:55 2013 {u'status': u'warmup', u'uptime': u'14', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13774328, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 4.2821158690176322}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'vb_replica_curr_items': 18, u'mem_used': 13774328, u'curr_items_tot': 30, u'curr_items': 12}} SLEEP Mon Feb 25 17:38:00 2013 {u'status': u'warmup', u'uptime': u'19', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615504, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.24937655860349128}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615504, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}} SLEEP 2013-02-25 17:38:01,117 - root - INFO - command executed successfully 2013-02-25 17:38:01,117 - root - INFO - 1d0 2013-02-25 17:38:01,117 - root - INFO - < efc4b36b015426f01460d6e20f4d81b5 couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm 2013-02-25 17:38:01,475 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-25 17:38:01,588 - root - INFO - command executed successfully 2013-02-25 17:38:01,588 - root - INFO - running command.raw sudo df -Th 2013-02-25 17:38:01,696 - root - INFO - command executed successfully 2013-02-25 17:38:01,696 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-25 17:38:01,808 - root - INFO - command executed successfully 2013-02-25 17:38:01,808 - root - INFO - running command.raw hostname 2013-02-25 17:38:01,920 - root - INFO - command executed successfully 2013-02-25 17:38:01,921 - root - INFO - deliverable_type : rpm 2013-02-25 17:38:01,921 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm or /tmp/couchbase-server-enterprise 2013-02-25 17:38:01,921 - root - INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_64_2.0.1-164-rel.rpm Mon Feb 25 17:38:06 2013 {u'status': u'warmup', u'uptime': u'24', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615504, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.25}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615504, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}} Exception in thread upgrade_thread172.27.33.4: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self._ target(*self. args, **self. _kwargs) File "pytests/newupgradebasetest.py", line 140, in _upgrade raise e Exception: couchApiBase doesn't exist in nodes/self: {u'status': u'warmup', u'uptime': u'29', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2337271808, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.4:8091', u'storage': {u'ssd': [], u'hdd': [{u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'}] }, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 269442, u'used': 3778554101}, u'ram': {u'used': 1957695488, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1615408, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.4', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [{u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952}, {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086}, {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152}] }, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 132427776, u'cpu_utilization_rate': 0.0}, u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210}, u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 269442, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1615408, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 248490}} upgrade for the second node is still in progress: 2013-02-25 17:38:21,980 - root - INFO - command executed successfully 2013-02-25 17:38:21,980 - root - INFO - Stopping couchbase-server ... 2013-02-25 17:38:21,980 - root - INFO - Stopping couchbase-server 2013-02-25 17:38:21,980 - root - INFO - Minimum RAM required : 4 GB 2013-02-25 17:38:21,980 - root - INFO - System RAM configured : 4194304 kB 2013-02-25 17:38:21,980 - root - INFO - 2013-02-25 17:38:21,980 - root - INFO - Minimum number of processors required : 4 cores 2013-02-25 17:38:21,980 - root - INFO - Number of processors on the system : 4 cores 2013-02-25 17:38:21,980 - root - INFO - 2013-02-25 17:38:21,980 - root - INFO - Upgrading couchbase-server ... 2013-02-25 17:38:21,981 - root - INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 2013-02-25 17:38:21,981 - root - INFO - Automatic mode: running without interactive questions or confirmations. 2013-02-25 17:38:21,981 - root - INFO - Analysing... 2013-02-25 17:38:21,981 - root - INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-25 17:38:21,981 - root - INFO - Target node: ns_1@172.27.33.2 2013-02-25 17:38:21,981 - root - INFO - Upgrading your Couchbase Server from 1.8.x to 2.0.1-164-rel. 2013-02-25 17:38:21,981 - root - INFO - NOTE: all item data will be read from 1.8.x sqlite database files 2013-02-25 17:38:21,981 - root - INFO - and converted to new files in 2.0 couchstore file format. This can 2013-02-25 17:38:21,981 - root - INFO - be a lengthy, slow operation for large datasets and/or fragmented 2013-02-25 17:38:21,981 - root - INFO - database files. 2013-02-25 17:38:21,981 - root - INFO - 2013-02-25 17:38:21,982 - root - INFO - Couchbase should not be running. 2013-02-25 17:38:21,982 - root - INFO - Please use: /etc/init.d/couchbase-server stop 2013-02-25 17:38:21,982 - root - INFO - 2013-02-25 17:38:21,982 - root - INFO - Database dir: /tmp/ 2013-02-25 17:38:21,982 - root - INFO - 2013-02-25 17:38:21,982 - root - INFO - Buckets to upgrade: default,bucket0,standard_bucket0 2013-02-25 17:38:21,982 - root - INFO - 2013-02-25 17:38:21,982 - root - INFO - Checking disk space available for buckets in directory: 2013-02-25 17:38:21,982 - root - INFO - /tmp/ 2013-02-25 17:38:21,982 - root - INFO - Free disk bucket space wanted: 528384.0 2013-02-25 17:38:21,982 - root - INFO - Free disk bucket space available: 26522152960 2013-02-25 17:38:21,983 - root - INFO - Free disk space factor: 2.0 2013-02-25 17:38:21,983 - root - INFO - Ok. 2013-02-25 17:38:21,983 - root - INFO - 2013-02-25 17:38:21,983 - root - INFO - Analysis complete. 2013-02-25 17:38:21,983 - root - INFO - 2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat 2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/ip 2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip 2013-02-25 17:38:21,983 - root - INFO - Copying /opt/couchbase/etc/couchdb/local.ini 2013-02-25 17:38:21,983 - root - INFO - cp /opt/couchbase/etc/couchdb/local.ini /opt/couchbase/bin/install/../../etc/couchdb/local.ini 2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directories. 2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//default 2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//default 2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//bucket0 2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//bucket0 2013-02-25 17:38:21,984 - root - INFO - Ensuring bucket data directory: /tmp//standard_bucket0 2013-02-25 17:38:21,984 - root - INFO - mkdir -p /tmp//standard_bucket0 2013-02-25 17:38:21,984 - root - INFO - Ensuring dbdir owner/group: /tmp/ 2013-02-25 17:38:21,984 - root - INFO - chown -R couchbase:couchbase /tmp/ 2013-02-25 17:38:21,984 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data 2013-02-25 17:38:21,984 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data 2013-02-25 17:38:21,984 - root - INFO - Upgrading buckets. 2013-02-25 17:38:21,985 - root - INFO - Upgrading bucket: default 2013-02-25 17:38:21,985 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=active --destination-vbucket-state=active 2013-02-25 17:38:21,985 - root - INFO - cbdbupgrade pid: 2646 2013-02-25 17:38:21,985 - root - INFO - .............. 2013-02-25 17:38:21,985 - root - INFO - bucket: default, msgs transferred... 2013-02-25 17:38:21,985 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,985 - root - INFO - batch : 65 | 65 | 524.0 2013-02-25 17:38:21,985 - root - INFO - byte : 3072 | 3072 | 24764.0 2013-02-25 17:38:21,985 - root - INFO - msg : 6 | 6 | 48.4 2013-02-25 17:38:21,985 - root - INFO - done 2013-02-25 17:38:21,985 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,986 - root - INFO - Upgrading bucket: default 2013-02-25 17:38:21,986 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=replica --destination-vbucket-state=replica 2013-02-25 17:38:21,986 - root - INFO - cbdbupgrade pid: 2655 2013-02-25 17:38:21,986 - root - INFO - .............. 2013-02-25 17:38:21,986 - root - INFO - bucket: default, msgs transferred... 2013-02-25 17:38:21,986 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,986 - root - INFO - batch : 65 | 65 | 1015.0 2013-02-25 17:38:21,986 - root - INFO - byte : 2048 | 2048 | 31981.1 2013-02-25 17:38:21,986 - root - INFO - msg : 4 | 4 | 62.5 2013-02-25 17:38:21,986 - root - INFO - done 2013-02-25 17:38:21,986 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,987 - root - INFO - Upgrading bucket: default 2013-02-25 17:38:21,987 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=pending --destination-vbucket-state=pending 2013-02-25 17:38:21,987 - root - INFO - cbdbupgrade pid: 2664 2013-02-25 17:38:21,987 - root - INFO - .............. 2013-02-25 17:38:21,987 - root - INFO - bucket: default, msgs transferred... 2013-02-25 17:38:21,987 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,987 - root - INFO - batch : 65 | 65 | 1012.6 2013-02-25 17:38:21,987 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,987 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,987 - root - INFO - done 2013-02-25 17:38:21,987 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,987 - root - INFO - Upgrading bucket: default 2013-02-25 17:38:21,988 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//default-data/default couchstore-files:///tmp/ -b default --source-vbucket-state=dead --destination-vbucket-state=dead 2013-02-25 17:38:21,988 - root - INFO - cbdbupgrade pid: 2673 2013-02-25 17:38:21,988 - root - INFO - .............. 2013-02-25 17:38:21,988 - root - INFO - bucket: default, msgs transferred... 2013-02-25 17:38:21,988 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,988 - root - INFO - batch : 65 | 65 | 1012.8 2013-02-25 17:38:21,988 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,988 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,988 - root - INFO - done 2013-02-25 17:38:21,988 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,988 - root - INFO - Ensuring bucket owner/group: /tmp//default 2013-02-25 17:38:21,989 - root - INFO - chown -R couchbase:couchbase /tmp//default 2013-02-25 17:38:21,989 - root - INFO - Upgrading bucket: bucket0 2013-02-25 17:38:21,989 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=active --destination-vbucket-state=active 2013-02-25 17:38:21,989 - root - INFO - cbdbupgrade pid: 2693 2013-02-25 17:38:21,989 - root - INFO - .............. 2013-02-25 17:38:21,989 - root - INFO - bucket: bucket0, msgs transferred... 2013-02-25 17:38:21,989 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,989 - root - INFO - batch : 65 | 65 | 1014.9 2013-02-25 17:38:21,989 - root - INFO - byte : 3072 | 3072 | 47965.6 2013-02-25 17:38:21,989 - root - INFO - msg : 6 | 6 | 93.7 2013-02-25 17:38:21,989 - root - INFO - done 2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,990 - root - INFO - Upgrading bucket: bucket0 2013-02-25 17:38:21,990 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica 2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade pid: 2702 2013-02-25 17:38:21,990 - root - INFO - .............. 2013-02-25 17:38:21,990 - root - INFO - bucket: bucket0, msgs transferred... 2013-02-25 17:38:21,990 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,990 - root - INFO - batch : 65 | 65 | 1015.0 2013-02-25 17:38:21,990 - root - INFO - byte : 2048 | 2048 | 31981.5 2013-02-25 17:38:21,990 - root - INFO - msg : 4 | 4 | 62.5 2013-02-25 17:38:21,990 - root - INFO - done 2013-02-25 17:38:21,990 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,991 - root - INFO - Upgrading bucket: bucket0 2013-02-25 17:38:21,991 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending 2013-02-25 17:38:21,991 - root - INFO - cbdbupgrade pid: 2711 2013-02-25 17:38:21,991 - root - INFO - .............. 2013-02-25 17:38:21,991 - root - INFO - bucket: bucket0, msgs transferred... 2013-02-25 17:38:21,991 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,991 - root - INFO - batch : 65 | 65 | 1012.7 2013-02-25 17:38:21,991 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,991 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,991 - root - INFO - done 2013-02-25 17:38:21,991 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,992 - root - INFO - Upgrading bucket: bucket0 2013-02-25 17:38:21,992 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//bucket0-data/bucket0 couchstore-files:///tmp/ -b bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead 2013-02-25 17:38:21,992 - root - INFO - cbdbupgrade pid: 2720 2013-02-25 17:38:21,992 - root - INFO - .............. 2013-02-25 17:38:21,992 - root - INFO - bucket: bucket0, msgs transferred... 2013-02-25 17:38:21,992 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,992 - root - INFO - batch : 65 | 65 | 1015.1 2013-02-25 17:38:21,992 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,992 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,992 - root - INFO - done 2013-02-25 17:38:21,992 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,992 - root - INFO - Ensuring bucket owner/group: /tmp//bucket0 2013-02-25 17:38:21,993 - root - INFO - chown -R couchbase:couchbase /tmp//bucket0 2013-02-25 17:38:21,993 - root - INFO - Upgrading bucket: standard_bucket0 2013-02-25 17:38:21,993 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=active --destination-vbucket-state=active 2013-02-25 17:38:21,993 - root - INFO - cbdbupgrade pid: 2740 2013-02-25 17:38:21,993 - root - INFO - .............. 2013-02-25 17:38:21,993 - root - INFO - bucket: standard_bucket0, msgs transferred... 2013-02-25 17:38:21,993 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,993 - root - INFO - batch : 65 | 65 | 524.0 2013-02-25 17:38:21,993 - root - INFO - byte : 3072 | 3072 | 24767.4 2013-02-25 17:38:21,993 - root - INFO - msg : 6 | 6 | 48.4 2013-02-25 17:38:21,993 - root - INFO - done 2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,994 - root - INFO - Upgrading bucket: standard_bucket0 2013-02-25 17:38:21,994 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=replica --destination-vbucket-state=replica 2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade pid: 2749 2013-02-25 17:38:21,994 - root - INFO - .............. 2013-02-25 17:38:21,994 - root - INFO - bucket: standard_bucket0, msgs transferred... 2013-02-25 17:38:21,994 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,994 - root - INFO - batch : 65 | 65 | 1012.7 2013-02-25 17:38:21,994 - root - INFO - byte : 2048 | 2048 | 31908.3 2013-02-25 17:38:21,994 - root - INFO - msg : 4 | 4 | 62.3 2013-02-25 17:38:21,994 - root - INFO - done 2013-02-25 17:38:21,994 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,995 - root - INFO - Upgrading bucket: standard_bucket0 2013-02-25 17:38:21,995 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=pending --destination-vbucket-state=pending 2013-02-25 17:38:21,995 - root - INFO - cbdbupgrade pid: 2758 2013-02-25 17:38:21,995 - root - INFO - .............. 2013-02-25 17:38:21,995 - root - INFO - bucket: standard_bucket0, msgs transferred... 2013-02-25 17:38:21,995 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,995 - root - INFO - batch : 65 | 65 | 1012.8 2013-02-25 17:38:21,995 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,995 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,995 - root - INFO - done 2013-02-25 17:38:21,995 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,996 - root - INFO - Upgrading bucket: standard_bucket0 2013-02-25 17:38:21,996 - root - INFO - /opt/couchbase/bin/install/../cbtransfer /tmp//standard_bucket0-data/standard_bucket0 couchstore-files:///tmp/ -b standard_bucket0 --source-vbucket-state=dead --destination-vbucket-state=dead 2013-02-25 17:38:21,996 - root - INFO - cbdbupgrade pid: 2767 2013-02-25 17:38:21,996 - root - INFO - .............. 2013-02-25 17:38:21,996 - root - INFO - bucket: standard_bucket0, msgs transferred... 2013-02-25 17:38:21,996 - root - INFO - : total | last | per sec 2013-02-25 17:38:21,996 - root - INFO - batch : 65 | 65 | 1012.9 2013-02-25 17:38:21,996 - root - INFO - byte : 0 | 0 | 0.0 2013-02-25 17:38:21,996 - root - INFO - msg : 0 | 0 | 0.0 2013-02-25 17:38:21,996 - root - INFO - done 2013-02-25 17:38:21,996 - root - INFO - cbdbupgrade err: 0 2013-02-25 17:38:21,997 - root - INFO - Ensuring bucket owner/group: /tmp//standard_bucket0 2013-02-25 17:38:21,997 - root - INFO - chown -R couchbase:couchbase /tmp//standard_bucket0 2013-02-25 17:38:21,997 - root - INFO - 2013-02-25 17:38:21,997 - root - INFO - Done. 2013-02-25 17:38:21,997 - root - INFO - Starting couchbase-server[ OK ] 2013-02-25 17:38:21,997 - root - INFO - 2013-02-25 17:38:21,997 - root - INFO - 2013-02-25 17:38:21,997 - root - INFO - You have successfully installed Couchbase Server. 2013-02-25 17:38:21,997 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server. 2013-02-25 17:38:21,997 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-02-25 17:38:21,997 - root - INFO - 2013-02-25 17:38:21,997 - root - INFO - Please note that you have to update your firewall configuration to 2013-02-25 17:38:21,998 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-02-25 17:38:21,998 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-02-25 17:38:21,998 - root - INFO - 2013-02-25 17:38:21,998 - root - INFO - By using this software you agree to the End User License Agreement. 2013-02-25 17:38:21,998 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-02-25 17:38:21,998 - root - INFO - 2013-02-25 17:38:21,998 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-02-25 17:38:21,998 - root - INFO - {badrpc,nodedown} 2013-02-25 17:38:21,998 - root - INFO - got couchApiBase from the third attempt for the second node( after 10 seconds) 2013-02-25 17:38:22,038 - root - ERROR - socket error while connecting to http://172.27.33.2:8091/nodes/self error [Errno 111] Connection refused 2013-02-25 17:38:23,040 - root - ERROR - socket error while connecting to http://172.27.33.2:8091/nodes/self error [Errno 111] Connection refused SLEEP Mon Feb 25 17:38:29 2013 {u'status': u'warmup', u'uptime': u'4', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2405048320, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.2:8091', u'storage': {u'ssd': [], u'hdd': [ {u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'} ]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 0, u'used': 3778554101} , u'ram': {u'used': 1889918976, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 13775768, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.2', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [ {u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952} , {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086} , {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152} ]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 120999936, u'cpu_utilization_rate': 5.2631578947368425} , u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210} , u'interestingStats': {u'vb_replica_curr_items': 12, u'mem_used': 13775768, u'curr_items_tot': 30, u'curr_items': 18}} SLEEP Mon Feb 25 17:38:34 2013 {u'status': u'warmup', u'uptime': u'9', u'clusterMembership': u'active', u'otpCookie': u'ujfppjvxnxwyjkxo', u'memoryQuota': 2184, u'memoryFree': 2405048320, u'memoryTotal': 4294967296, u'hostname': u'172.27.33.2:8091', u'storage': {u'ssd': [], u'hdd': [ {u'path': u'/tmp', u'quotaMb': u'none', u'index_path': u'/tmp', u'state': u'ok'} ]}, u'storageTotals': {u'hdd': {u'free': 27709396747, u'total': 31487950848, u'quotaTotal': 31487950848, u'usedByData': 146556, u'used': 3778554101} , u'ram': {u'used': 1889918976, u'total': 4294967296, u'quotaTotal': 2290089984, u'usedByData': 1613440, u'quotaUsed': 1525678080}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.0.1-164-rel-enterprise', u'otpNode': u'ns_1@172.27.33.2', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [ {u'usagePercent': 12, u'path': u'/', u'sizeKBytes': 30749952} , {u'usagePercent': 15, u'path': u'/boot', u'sizeKBytes': 101086} , {u'usagePercent': 0, u'path': u'/dev/shm', u'sizeKBytes': 2097152} ]}, u'systemStats': {u'swap_total': 4959834112, u'swap_used': 120999936, u'cpu_utilization_rate': 0.0} , u'os': u'x86_64-unknown-linux-gnu', u'ports': {u'proxy': 11211, u'direct': 11210} , u'interestingStats': {u'couch_views_data_size': 0, u'couch_docs_actual_disk_size': 146556, u'couch_views_actual_disk_size': 0, u'vb_replica_curr_items': 0, u'mem_used': 1613440, u'curr_items': 0, u'curr_items_tot': 0, u'couch_docs_data_size': 133802}} 2013-02-25 17:38:34,099 - root - INFO - settings/web params on 172.27.33.2:8091 :username=Administrator&password=password&port=8091 2013-02-25 17:38:34,102 - root - INFO - sleep for 10 secs. ... FAIL 2013-02-25 17:38:44,124 - root - WARNING - CLEANUP WAS SKIPPED ====================================================================== FAIL: offline_cluster_upgrade (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 80, in offline_cluster_upgrade self.fail("Upgrade failed!") AssertionError: Upgrade failed! ---------------------------------------------------------------------- Ran 1 test in 862.160s FAILED (failures=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 https://s3.amazonaws.com/bugdb/jira/MB-7817/172.27.33.2-2252013-1752-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7817/172.27.33.4-2252013-1753-diag.zip
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        spoke with andrei about this issue and andrei is going to confirm this :
        1- this occurs only when some nodes are partially offline upgraded
        2- even though this is a bug would the user be able to get around this issue by restart the nodes or running some command ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - spoke with andrei about this issue and andrei is going to confirm this : 1- this occurs only when some nodes are partially offline upgraded 2- even though this is a bug would the user be able to get around this issue by restart the nodes or running some command ?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        ns_server code or other components have not changed since build 161 so this can not be a regression, mostly its a timing thing

        Show
        farshid Farshid Ghods (Inactive) added a comment - ns_server code or other components have not changed since build 161 so this can not be a regression, mostly its a timing thing
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        2.0 mode is enabled only after all nodes are 2.0 or higher. That hasn't changed since GA.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - 2.0 mode is enabled only after all nodes are 2.0 or higher. That hasn't changed since GA.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/)
        MB-7817: wait while all nodes will be updated (Revision 673daabf465b81dfd5f2b4ce5a2c6ee2beaeebe1)

        Result = SUCCESS
        andrei :
        Files :

        • pytests/newupgradebasetest.py
        Show
        thuan Thuan Nguyen added a comment - Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/ ) MB-7817 : wait while all nodes will be updated (Revision 673daabf465b81dfd5f2b4ce5a2c6ee2beaeebe1) Result = SUCCESS andrei : Files : pytests/newupgradebasetest.py

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes