Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33779

CB Restore with GSI indexes failing

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-2943

      cbbackupmgr restore fails when the backup has gsi indexes to restore.

      On the console, the output is like this -

      [root@localhost bin]# ./cbbackupmgr restore -c localhost -u Administrator -p password -a /tmp/backup -r fresh650bkp
       
      (1/1) Restoring backup 2019-04-12T13_59_46.304926829-07_00
      Copying at 0B (estimating time remaining)                                                                                                                                                                                                                           0 items / 0B
      Error restoring cluster: EOF
      Restore bucket "travel-sample" succeeded
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 0, Deletions failed to restore: 0
      Skipped due to purge number or conflict resolution: Muatations: 0 Deletions: 0
      

      The backup log has the following :

      2019-04-12T14:05:59.318-07:00 (Cmd) restore -c localhost -u <ud>Administrator</ud> -p ******** -a /tmp/backup -r fresh650bkp
      2019-04-12T14:05:59.323-07:00 (Rest) GET http://localhost:8091/pools 200
      2019-04-12T14:05:59.326-07:00 (Rest) GET http://localhost:8091/pools 200
      2019-04-12T14:05:59.326-07:00 (Cmd) Restoring to cluster d09849738c751775d71e58d53212fa09
      2019-04-12T14:05:59.326-07:00 Restoring data from backup 2019-04-12T13_59_46.304926829-07_00
      2019-04-12T14:05:59.326-07:00 (Plan) Executing transfer plan
      2019-04-12T14:05:59.326-07:00 (Plan) Checking for data movement restrictions between travel-sample and travel-sample
      2019-04-12T14:05:59.351-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.361-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.361-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.361-07:00 (Plan) Checking for data movement restrictions between beer-sample and beer-sample
      2019-04-12T14:05:59.380-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.385-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.386-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.386-07:00 (Plan) Checking for data movement restrictions between gamesim-sample and gamesim-sample
      2019-04-12T14:05:59.409-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.413-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.414-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.414-07:00 (Plan) Transferring views definitions for travel-sample to travel-sample
      2019-04-12T14:05:59.415-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
      2019-04-12T14:05:59.415-07:00 (Plan) Transferring GSI index definitions for travel-sample to travel-sample
      2019-04-12T14:05:59.418-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
      2019-04-12T14:05:59.458-07:00 ERRO: EOF -- plan.(*gsi).execute() at gsi.go:64
      2019-04-12T14:05:59.458-07:00 (Plan) Transfer plan failed due to error EOF
      2019-04-12T14:05:59.458-07:00 (Cmd) Error restoring cluster: EOF
      2019-04-12T14:05:59.458-07:00 (Cmd) Restore bucket "travel-sample" succeeded
      2019-04-12T14:05:59.458-07:00 (Cmd) Mutations restored: 0, Mutations failed to restore: 0
      2019-04-12T14:05:59.458-07:00 (Cmd) Deletions restored: 0, Deletions failed to restore: 0
      2019-04-12T14:05:59.458-07:00 (Cmd) Skipped due to purge number or conflict resolution: Muatations: 0 Deletions: 0
      

      The indexer log has the following panic:

      2019-04-12T14:05:59.434-07:00 [Info] restore to target bucket travel-sample
      2019-04-12T14:05:59.449-07:00 [Info] StreamState::adjustNonSnapAlignedVbs
      	Bucket gamesim-sample StreamId MAINT_STREAM Vbuckets [].
      2019-04-12T14:05:59.449-07:00 [Info] Timekeeper::repairStream Nothing to repair for Stream MAINT_STREAM and Bucket gamesim-sample
      2019/04/12 14:05:59 http: panic serving [::1]:34639: runtime error: invalid memory address or nil pointer dereference
      goroutine 96172 [running]:
      net/http.(*conn).serve.func1(0xc006e38640)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1746 +0xd0
      panic(0xebf1c0, 0x1d636a0)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/indexing/secondary/planner.findTopologyByBucket(0x0, 0xc00f4e4780, 0x1, 0xc005261550, 0xd, 0xe2adc0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/planner/proxy.go:816 +0x67
      github.com/couchbase/indexing/secondary/planner.ConvertToIndexUsage(0xc0102f5c80, 0xc004b5c1a0, 0xc0116cf120, 0x0, 0xf90960, 0x4, 0xc0116cefc8, 0x30)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/planner/proxy.go:269 +0x73
      github.com/couchbase/indexing/secondary/manager.(*RestoreContext).convertImage(0xc0116cf7d8, 0x0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/restore.go:131 +0x948
      github.com/couchbase/indexing/secondary/manager.(*RestoreContext).computeIndexLayout(0xc0116cf7d8, 0x1b, 0xc0116cf728, 0x1)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/restore.go:67 +0x50
      github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleRestoreIndexMetadataRequest(0x1dca620, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1096 +0x38f
      github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleRestoreIndexMetadataRequest-fm(0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:189 +0x48
      net/http.HandlerFunc.ServeHTTP(0xc000142460, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1964 +0x44
      net/http.(*ServeMux).ServeHTTP(0xc0001342a0, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2361 +0x127
      net/http.serverHandler.ServeHTTP(0xc0001228f0, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc006e38640, 0x119e700, 0xc006962500)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2851 +0x2f5
      

      Steps to reproduce :
      1. Create a 1 node cluster with kv+query+index services
      2. Install all 3 sample buckets.
      3. Take a full backup.
      4. Delete the buckets
      5. Create empty buckets with the same names
      6. Restore from backup.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-33779
          # Subject Branch Project Status CR V

          Activity

            People

              jliang John Liang
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty