Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25159

cbbackmgr : panic seen during an incremental backup. Cannot resume.

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 4.6.3-4051

      During the Incremental system test, the incremental backup in the 3rd iteration after failing over a data node and rebalancing it out failed, and there is a panic in the backup.log :
      The src cluster would have had 120M+ items at the time of failure.

      panic: runtime error: invalid memory address or nil pointer dereference
      		 
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4981ec]
      		 
       
      		 
      goroutine 1 [running]:
      		 
      panic(0x7079a0, 0xc4200100a0)
      		 
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1
      		 
      github.com/couchbase/backup/archive.(*backupDir).nextIncrBackupStartSeqnos(0xc420013170, 0xc42014e820, 0x1e, 0xc42014d326, 0x7, 0xc420aaccf0, 0xc42000ab00, 0xc42081ed20, 0xc420804750)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/archive/dir_backup.go:346 +0x85c
      		 
      github.com/couchbase/backup/archive.(*archiveSink).DataRange(0xc42014f3c0, 0xc42014d326, 0x7, 0xc42014f140, 0x20, 0xc420aaccf0, 0x0, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/archive/sink.go:98 +0x94
      		 
      github.com/couchbase/backup/plan.(*dataRange).execute(0xc420166740, 0xad1060, 0xc420015d40, 0xad0ee0, 0xc42014f3c0, 0xc42014d790, 0x0, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/plan/range.go:53 +0x19c
      		 
      github.com/couchbase/backup/plan.(*serial).execute(0xc4201742a0, 0xad1060, 0xc420015d40, 0xad0ee0, 0xc42014f3c0, 0xc42014d790, 0x0, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/plan/serial.go:67 +0xa0
      		 
      github.com/couchbase/backup/plan.(*parallel).execute(0xc420174270, 0xad1060, 0xc420015d40, 0xad0ee0, 0xc42014f3c0, 0xc42014d790, 0x0, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/plan/parallel.go:67 +0xa0
      		 
      github.com/couchbase/backup/plan.(*serial).execute(0xc420174210, 0xad1060, 0xc420015d40, 0xad0ee0, 0xc42014f3c0, 0xc42014d790, 0xc42014d7a0, 0xc42014d7a0)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/plan/serial.go:67 +0xa0
      		 
      github.com/couchbase/backup/plan.(*Plan).ExecutePlan(0xc42014d790, 0xad1060, 0xc420015d40, 0xad0ee0, 0xc42014f3c0, 0xc4200f3248, 0x403b87)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/plan/plan.go:112 +0xa4
      		 
      github.com/couchbase/backup.(*Transferable).Execute(0xc420166a00, 0x794a70, 0xc420166a00)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/transferable.go:30 +0x53
      		 
      main.(*BackupContext).Run(0xc4200a2090)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/cmd/cbbackupmgr/backup.go:87 +0x309
      		 
      main.(*BackupContext).Run-fm()
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/cmd/cbbackupmgr/main.go:79 +0x2a
      		 
      github.com/couchbase/cbflag.(*Command).parseFlags(0xc420078070, 0xc4200f3850, 0xc42000c2c0, 0xc, 0xc)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/cbflag/command.go:218 +0x570
      		 
      github.com/couchbase/cbflag.(*Command).parse(0xc420078070, 0xc4200f3850, 0xc42000c2c0, 0xc, 0xc)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/cbflag/command.go:89 +0x109
      		 
      github.com/couchbase/cbflag.(*Command).parseCommands(0xc420078850, 0xc4200f3850, 0xc42000c2b0, 0xd, 0xd)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/cbflag/command.go:101 +0x340
      		 
      github.com/couchbase/cbflag.(*Command).parse(0xc420078850, 0xc4200f3850, 0xc42000c2b0, 0xd, 0xd)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/cbflag/command.go:87 +0x13e
      		 
      github.com/couchbase/cbflag.(*CLI).Parse(0xc4200a2120, 0xc42000c2a0, 0xe, 0xe)
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/cbflag/cli.go:59 +0x1ee
      		 
      main.main()
      		 
      	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/backup/cmd/cbbackupmgr/main.go:818 +0x7f96

      Post this, I tried to resume the backup, but it too fails with the same error. I deliberately didnt try the purge option so that the environment can be looked at. I have aborted the test.

      Environment is live currently for debugging:
      Src cluster : 35.163.219.174, 34.210.232.120, 52.41.131.68
      Dest cluster : 35.165.121.2

      Backup location on src cluster : /backupdata
      Backup logs are at : /backupdata/logs

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-25238 Rebalance fails when adding back a failed over node with delta recovery

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-25140 cbbackupmgr does not resume once hung due to low memory

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty