Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19727

Upgrade from 4.1 - 4.5 - Indexer crashing - [FDB ERR] Failed to read the superblock:

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 4.5.0
    • 4.5.0
    • forestdb
    • None
    • 2585

    Description

      1. Install 4.1.0-5005 and install travel-bucket
      2. Upgrade to 4.5.0-2585, node by node, using rpm -U
      3. Check log file.

      Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit()
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820043fb0 sp=0xc820043fa8
      created by net/http.(*Transport).dialConn
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/net/http/transport.go:854 +0x10cb
      [goport] 2016/05/24 03:14:23 /opt/couchbase/bin/indexer terminated: signal: aborted (core dumped)
      

      2016-05-24T03:05:24.742-07:00 [Error] updateIndexerList(): Cannot locate cluster node.
      [FDB INFO] Forestdb blockcache size 268435456 initialized in 15508 us
       
      [FDB ERR] Failed to read the superblock: incorrect block marker (marker: dd, SB No.: 0)
      [FDB ERR] Failed to read the superblock: incorrect block marker (marker: ee, SB No.: 1)
      [FDB ERR] Failed to read the superblock: incorrect block marker (marker: dd, SB No.: 2)
      [FDB ERR] Failed to read the superblock: incorrect block marker (marker: ee, SB No.: 3)
      [FDB INFO] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore
      [FDB ERR] Cannot open KV store instance 'MAIN' because multi-KV store instance mode is disabled.
      [FDB INFO] Forestdb closed database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore
      2016-05-24T03:05:24.778-07:00 [Error] EmbeddedServer.boostrap: error : FDB_RESULT_INVALID_CONFIG
      panic: close of nil channel
       
      goroutine 1 [running]:
      panic(0xced3c0, 0xc8231a0a00)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:464 +0x3e6 fp=0xc82013ccb8 sp=0xc82013cc38
      runtime.closechan(0x0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/chan.go:294 +0x75 fp=0xc82013cd08 sp=0xc82013ccb8
      github.com/couchbase/indexing/secondary/manager.(*IndexManager).Close(0xc8200e62c0)
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:271 +0x143 fp=0xc82013cd30 sp=0xc82013cd08
      github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal(0x7f24e6c956a0, 0xc821187c80, 0x7f24e6c95740, 0xc823208120, 0xc8200183f0, 0xc8200e62c0, 0x7f24e6c957e0, 0xc8231a09d0)
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:185 +0xcec fp=0xc82013d070 sp=0xc82013cd30
      github.com/couchbase/indexing/secondary/manager.NewIndexManager(0x7f24e6c956a0, 0xc821187c80, 0xc8200183f0, 0x7f24e6c956a0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:122 +0x222 fp=0xc82013d148 sp=0xc82013d070
      github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent(0xc8211a23c0, 0xc8211a2000, 0xc8200183f0, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:71 +0x7e1 fp=0xc82013d2f8 sp=0xc82013d148
      github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc8201522d0, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:240 +0x20d1 fp=0xc82013dc20 sp=0xc82013d2f8
      main.main()
              /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:153 +0x15b5 fp=0xc82013df40 sp=0xc82013dc20
      runtime.main()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:188 +0x2b0 fp=0xc82013df90 sp=0xc82013df40
      runtime.goexit()
      
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-19727
          # Subject Branch Project Status CR V

          Activity

            I talked to James Mauss. He reviewed the scenario and said this is something customer would be doing in a multi-node environment. This is a supported scenario. So, it's critical for us to fix it.

            keshav Keshav Murthy added a comment - I talked to James Mauss . He reviewed the scenario and said this is something customer would be doing in a multi-node environment. This is a supported scenario. So, it's critical for us to fix it.

            Quick updates:

            We identified a bug in setting a commit header in the upgrade scenario and will push the fix very soon.

            chiyoung Chiyoung Seo (Inactive) added a comment - Quick updates: We identified a bug in setting a commit header in the upgrade scenario and will push the fix very soon.

            To give a bit more context - the bug happens in the MetadataStore which is never compacted. As a result right after upgrade, forestdb is expected to continue using the MetadataStore with the same version that it was opened with (in this case 4.1.0).
            However due to a bug while updating the file opened with old version, we were incorrectly writing some 'newer information' into MetadataStore header, causing future restart to end up reading an older version number but trip on the 'newer info' wrongly written.
            Fix soon to be merged will be to not write new info when file belongs to older version.
            thanks

            sundar Sundar Sridharan (Inactive) added a comment - To give a bit more context - the bug happens in the MetadataStore which is never compacted. As a result right after upgrade, forestdb is expected to continue using the MetadataStore with the same version that it was opened with (in this case 4.1.0). However due to a bug while updating the file opened with old version, we were incorrectly writing some 'newer information' into MetadataStore header, causing future restart to end up reading an older version number but trip on the 'newer info' wrongly written. Fix soon to be merged will be to not write new info when file belongs to older version. thanks

            The fix was merged:

            http://review.couchbase.org/#/c/64369/

            Please help us verify it when the new 4.5.0 build is ready.

            chiyoung Chiyoung Seo (Inactive) added a comment - The fix was merged: http://review.couchbase.org/#/c/64369/ Please help us verify it when the new 4.5.0 build is ready.

            Hi Chiyoung

            I verified the scenario with latest build: 2592 which has the fix:
            changed projects :

            forestdb changed from 05b1f487b4ee6e512de170bc086ab8ae4016971f to c36a0ed074e1bbeb2679dc110cbfb5c89189bc36
            [+] c36a0ed MB-19727 Do not add # deleted docs field for 4.1.0 commit header

            I dont see the error in logs.

            Thanks
            Prerna

            Prerna.Manaktala Prerna Manaktala (Inactive) added a comment - Hi Chiyoung I verified the scenario with latest build: 2592 which has the fix: changed projects : forestdb changed from 05b1f487b4ee6e512de170bc086ab8ae4016971f to c36a0ed074e1bbeb2679dc110cbfb5c89189bc36 [+] c36a0ed MB-19727 Do not add # deleted docs field for 4.1.0 commit header I dont see the error in logs. Thanks Prerna

            People

              ritam.sharma Ritam Sharma
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty