Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36940

Upgrade fails : Namespace upgrade: upgradeCallback couchstore_open_doc_with_docinfo errcode:-5

    XMLWordPrintable

Details

    • Triaged
    • Unknown
    • KV-Engine Mad-Hatter GA

    Description

      With the data generated by FuzzyTestGenerator, upgrade from 5.5.5/5.5.6 to 6.5.0 shows errors. The bucket remains in warmup state forever.

      Steps followed:

      • Cleaned up the machine including /opt/couchbase
      • Setup one cluster with 5.5.6-4730 with services kv,n1ql,index
      • Create a bucket
      • Load documents using the attached exported docs (using cbimport). This dataset is generated by FuzzyTestGenerator
      • Upgrade to 6.5.0-4821 with command:
      • rpm -U couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm

      Upgrade from 5.5.6-4730 to 6.5.0-4821 errored out with below with the message "You have successfully installed Couchbase Server":

      [root@localhost tmp]# rpm -U couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm 
      Minimum RAM required  : 4 GB
      System RAM configured : 4.07 GB
       
      Minimum number of processors required : 4 cores
      Number of processors on the system    : 4 cores
       
      Upgrading couchbase-server ...
        /opt/couchbase/bin/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 
      Automatic mode: running without interactive questions or confirmations.
      Analysing...
      Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
       
      Database dir: /opt/couchbase/var/lib/couchbase/data
       
      Buckets to upgrade: default
       
      Checking disk space available for buckets in directory:
        /opt/couchbase/var/lib/couchbase/data
        Free disk bucket space wanted: 57344.0
        Free disk bucket space available: 22704865280
        Free disk space factor: 2.0
        Ok.
       
      Analysis complete.
       
      Upgrading existing datafiles to be namespace compatible for 1 bucket
      warning this may take time if not already done
      Namespace upgrade: bucket default /opt/couchbase/var/lib/couchbase/data/default
      Namespace upgrade: bucket default files to process 1024
      Namespace upgrade of file: /opt/couchbase/var/lib/couchbase/data/default/899.couch.3
          /opt/couchbase/bin/couchfile_upgrade --input /opt/couchbase/var/lib/couchbase/data/default/899.couch.3 --output /opt/couchbase/var/lib/couchbase/data/default/899.couch.4 --tolerate
      ERROR: problem executing `/opt/couchbase/bin/couchfile_upgrade --input /opt/couchbase/var/lib/couchbase/data/default/899.couch.3 --output /opt/couchbase/var/lib/couchbase/data/default/899.couch.4 --tolerate` RC: 1, STDOUT: Input:/opt/couchbase/var/lib/couchbase/data/default/899.couch.3
      Output:/opt/couchbase/var/lib/couchbase/data/default/899.couch.4
      exit(0) for already upgraded files
      , STDERR: An exception occurred: InputCouchFile::upgradeCallback couchstore_open_doc_with_docinfo errcode:-5
      Terminating with exit code 1
       
      warning: %post(couchbase-server-6.5.0-4821.x86_64) scriptlet failed, exit status 1
       
      You have successfully installed Couchbase Server.
      Please browse to http://localhost.localdomain:8091/ to configure your server.
      Refer to http://docs.couchbase.com for additional resources.
       
      Please note that you have to update your firewall configuration to
      allow external connections to a number of network ports for full
      operation. Refer to the documentation for the current list:
      https://docs.couchbase.com/server/6.5/install/install-ports.html
       
      By using this software you agree to the End User License Agreement.
      See /opt/couchbase/LICENSE.txt.
       
      [root@localhost tmp]# 
      
      

      Won't be able to see the documents after upgrade.

      Attachments

        1. 994.couch.2
          4 kB
        2. backup.tar
          51.01 MB
        3. export.zip
          4 kB
        4. logs.tar
          39.41 MB
        For Gerrit Dashboard: MB-36940
        # Subject Branch Project Status CR V

        Activity

          jwalker Jim Walker added a comment -

          The cbupgrade part of this fix is in review - http://review.couchbase.org/#/c/118134/

          jwalker Jim Walker added a comment - The cbupgrade part of this fix is in review - http://review.couchbase.org/#/c/118134/

          Build couchbase-server-6.5.0-4851 contains voltron commit ab02e12 with commit message:
          MB-36940: cbupgrade (couchfile_upgrade) must remove output file on error

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4851 contains voltron commit ab02e12 with commit message: MB-36940 : cbupgrade (couchfile_upgrade) must remove output file on error

          Build couchbase-server-7.0.0-1051 contains kv_engine commit ead473d with commit message:
          MB-36940: Handle tombstones properly in couchfile_upgrade

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1051 contains kv_engine commit ead473d with commit message: MB-36940 : Handle tombstones properly in couchfile_upgrade

          Build couchbase-server-7.0.0-1051 contains voltron commit ab02e12 with commit message:
          MB-36940: cbupgrade (couchfile_upgrade) must remove output file on error

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1051 contains voltron commit ab02e12 with commit message: MB-36940 : cbupgrade (couchfile_upgrade) must remove output file on error

          Closing as we did not see this issue with 6.5.0-4851, with the below steps:

          • install 5.5.6-4730: rpm -i couchbase-server-enterprise-5.5.6-4730-centos7.x86_64.rpm
          • Create a cluster with kv,index,query services
          • Create a bucket naming : default
          • create a backup repo: /opt/couchbase/bin/cbbackupmgr config -a /backup -r default_bkup
          • Untar the contents of backup.tar in /backup/default_bkup
          • So directory looks like:
          • [root@localhost default_bkup]# ls -ltr /backup/default_bkup
          • total 52236
          • rw-rw---. 1 root root 491 Nov 18 10:07 backup-meta.json
          • drwxrwx---. 3 root root 102 Nov 18 10:07 2019-11-18T10_07_45.171795846-08_00
          • run restore command: /opt/couchbase/bin/cbbackupmgr restore -a /backup -r default_bkup -c localhost -u Administrator -p password
          • upgrade to 6.5.0-4821 : rpm -U couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm
          girish.benakappa Girish Benakappa added a comment - Closing as we did not see this issue with 6.5.0-4851, with the below steps: install 5.5.6-4730: rpm -i couchbase-server-enterprise-5.5.6-4730-centos7.x86_64.rpm Create a cluster with kv,index,query services Create a bucket naming : default create a backup repo: /opt/couchbase/bin/cbbackupmgr config -a /backup -r default_bkup Untar the contents of backup.tar in /backup/default_bkup So directory looks like: [root@localhost default_bkup] # ls -ltr /backup/default_bkup total 52236 rw-rw---. 1 root root 491 Nov 18 10:07 backup-meta.json drwxrwx---. 3 root root 102 Nov 18 10:07 2019-11-18T10_07_45.171795846-08_00 run restore command: /opt/couchbase/bin/cbbackupmgr restore -a /backup -r default_bkup -c localhost -u Administrator -p password upgrade to 6.5.0-4821 : rpm -U couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty