Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25408

CLONE - Panic seen while running cbbackupmgr compact cmd

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 5.0.0-3255

      Originally seen in the new system test that compacts all incremental backups, but can be easily reproduced with the following steps.

      1. Create a bucket and load 1M items using cbworkloadgen
      ./cbworkloadgen -n 127.0.0.1:8091 -j -b test -i 1000000 -s 1100 -u Administrator -p password
      2. Take a backup of the cluster
      3. Compact this backup.

      Compact fails, there is no msg on the console. The backup log has the following panic : 

      2017-07-14T09:55:52.355-07:00 (Cmd) compact -a /tmp/backup -r backup --backup 2017-07-14T09_54_13.205096025-07_00 
      2017-07-14T09:55:52.394-07:00 [INFO][FDB] Forestdb blockcache size 2147483648 initialized in 37860 us
      		 
       
      		 
      2017-07-14T09:55:52.394-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_0.fdb
      		 
      2017-07-14T09:55:52.400-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_1.fdb
      		 
      2017-07-14T09:55:52.403-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_10.fdb
      		 
      2017-07-14T09:55:52.408-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_11.fdb
      		 
      2017-07-14T09:55:52.411-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_12.fdb
      		 
      2017-07-14T09:55:52.414-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_13.fdb
      		 
      2017-07-14T09:55:52.417-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_14.fdb
      		 
      2017-07-14T09:55:52.421-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_15.fdb
      		 
      2017-07-14T09:55:52.424-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_2.fdb
      		 
      2017-07-14T09:55:52.428-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_3.fdb
      		 
      2017-07-14T09:55:52.432-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_4.fdb
      		 
      2017-07-14T09:55:52.435-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_5.fdb
      		 
      2017-07-14T09:55:52.438-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_6.fdb
      		 
      2017-07-14T09:55:52.441-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_7.fdb
      		 
      2017-07-14T09:55:52.445-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_8.fdb
      		 
      2017-07-14T09:55:52.449-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_9.fdb
      		 
      2017-07-14T09:55:52.453-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_15.fdb.compact
      		 
      2017-07-14T09:55:52.453-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_7.fdb.compact
      		 
      2017-07-14T09:55:52.453-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_3.fdb.compact
      		 
      2017-07-14T09:55:52.453-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_11.fdb.compact
      		 
      2017-07-14T09:55:52.494-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_12.fdb.compact
      		 
      2017-07-14T09:55:52.544-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_0.fdb.compact
      		 
      2017-07-14T09:55:52.594-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_4.fdb.compact
      		 
      2017-07-14T09:55:52.687-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_8.fdb.compact
      		 
      2017-07-14T09:55:52.731-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_9.fdb.compact
      		 
      2017-07-14T09:55:52.734-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_1.fdb.compact
      		 
      2017-07-14T09:55:52.739-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_13.fdb.compact
      		 
      2017-07-14T09:55:52.778-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_6.fdb.compact
      		 
      2017-07-14T09:55:52.790-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_10.fdb.compact
      		 
      2017-07-14T09:55:52.843-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_2.fdb.compact
      		 
      2017-07-14T09:55:52.931-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_14.fdb.compact
      		 
      2017-07-14T09:55:52.951-07:00 [INFO][FDB] Forestdb opened database file /tmp/backup/backup/2017-07-14T09_54_13.205096025-07_00/test-136d64add18ef3bf5dda8c9eed913106/data/shard_5.fdb.compact
      		 
      panic: runtime error: index out of range
      		 
       
      		 
      goroutine 39 [running]:
      		 
      github.com/couchbase/goforestdb.LogCallbackInternal(0xc400000000, 0x7efdc55fd2b0, 0xc420114c70)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/goforestdb/log.go:13 +0x110
      		 
      github.com/couchbase/goforestdb._cgoexpwrap_512a988f7ab7_LogCallbackInternal(0x0, 0x7efdc55fd2b0, 0xc420114c70)
      		 
      	github.com/couchbase/goforestdb/_obj/_cgo_gotypes.go:1162 +0x3d
      		 
      github.com/couchbase/goforestdb._Cfunc_fdb_close(0x7eff25016110, 0x7eff00000000)
      		 
      	github.com/couchbase/goforestdb/_obj/_cgo_gotypes.go:404 +0x49
      		 
      github.com/couchbase/goforestdb.(*File).Close.func1(0x7eff25016110, 0x7cf415)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/goforestdb/file.go:145 +0x60
      		 
      github.com/couchbase/goforestdb.(*File).Close(0xc42000e278, 0x0, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/goforestdb/file.go:145 +0xd9
      		 
      github.com/couchbase/backup/storage.(*ForestDBFile).Close(0xc4200570e0)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/storage/forestdb.go:172 +0x16d
      		 
      github.com/couchbase/backup/storage.(*ForestDBFile).Compact(0xc4200570e0, 0xc42001b920, 0xf, 0xc420072f28)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/storage/forestdb.go:485 +0xf1
      		 
      github.com/couchbase/backup/storage.(*DataBackup).Compact.func1(0xc4201156e0, 0xc42001b950, 0xc4201156d8, 0xf)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/storage/databackup.go:291 +0x85
      		 
      created by github.com/couchbase/backup/storage.(*DataBackup).Compact
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/storage/databackup.go:293 +0xc7

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty