Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33779

CB Restore with GSI indexes failing

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-2943

      cbbackupmgr restore fails when the backup has gsi indexes to restore.

      On the console, the output is like this -

      [root@localhost bin]# ./cbbackupmgr restore -c localhost -u Administrator -p password -a /tmp/backup -r fresh650bkp
       
      (1/1) Restoring backup 2019-04-12T13_59_46.304926829-07_00
      Copying at 0B (estimating time remaining)                                                                                                                                                                                                                           0 items / 0B
      Error restoring cluster: EOF
      Restore bucket "travel-sample" succeeded
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 0, Deletions failed to restore: 0
      Skipped due to purge number or conflict resolution: Muatations: 0 Deletions: 0
      

      The backup log has the following :

      2019-04-12T14:05:59.318-07:00 (Cmd) restore -c localhost -u <ud>Administrator</ud> -p ******** -a /tmp/backup -r fresh650bkp
      2019-04-12T14:05:59.323-07:00 (Rest) GET http://localhost:8091/pools 200
      2019-04-12T14:05:59.326-07:00 (Rest) GET http://localhost:8091/pools 200
      2019-04-12T14:05:59.326-07:00 (Cmd) Restoring to cluster d09849738c751775d71e58d53212fa09
      2019-04-12T14:05:59.326-07:00 Restoring data from backup 2019-04-12T13_59_46.304926829-07_00
      2019-04-12T14:05:59.326-07:00 (Plan) Executing transfer plan
      2019-04-12T14:05:59.326-07:00 (Plan) Checking for data movement restrictions between travel-sample and travel-sample
      2019-04-12T14:05:59.351-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.361-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.361-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.361-07:00 (Plan) Checking for data movement restrictions between beer-sample and beer-sample
      2019-04-12T14:05:59.380-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.385-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.386-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.386-07:00 (Plan) Checking for data movement restrictions between gamesim-sample and gamesim-sample
      2019-04-12T14:05:59.409-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
      2019-04-12T14:05:59.413-07:00 (Rest) GET http://localhost:8091/pools/default 200
      2019-04-12T14:05:59.414-07:00 Transferring to Couchbase Server 6.5.0
      2019-04-12T14:05:59.414-07:00 (Plan) Transferring views definitions for travel-sample to travel-sample
      2019-04-12T14:05:59.415-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
      2019-04-12T14:05:59.415-07:00 (Plan) Transferring GSI index definitions for travel-sample to travel-sample
      2019-04-12T14:05:59.418-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
      2019-04-12T14:05:59.458-07:00 ERRO: EOF -- plan.(*gsi).execute() at gsi.go:64
      2019-04-12T14:05:59.458-07:00 (Plan) Transfer plan failed due to error EOF
      2019-04-12T14:05:59.458-07:00 (Cmd) Error restoring cluster: EOF
      2019-04-12T14:05:59.458-07:00 (Cmd) Restore bucket "travel-sample" succeeded
      2019-04-12T14:05:59.458-07:00 (Cmd) Mutations restored: 0, Mutations failed to restore: 0
      2019-04-12T14:05:59.458-07:00 (Cmd) Deletions restored: 0, Deletions failed to restore: 0
      2019-04-12T14:05:59.458-07:00 (Cmd) Skipped due to purge number or conflict resolution: Muatations: 0 Deletions: 0
      

      The indexer log has the following panic:

      2019-04-12T14:05:59.434-07:00 [Info] restore to target bucket travel-sample
      2019-04-12T14:05:59.449-07:00 [Info] StreamState::adjustNonSnapAlignedVbs
      	Bucket gamesim-sample StreamId MAINT_STREAM Vbuckets [].
      2019-04-12T14:05:59.449-07:00 [Info] Timekeeper::repairStream Nothing to repair for Stream MAINT_STREAM and Bucket gamesim-sample
      2019/04/12 14:05:59 http: panic serving [::1]:34639: runtime error: invalid memory address or nil pointer dereference
      goroutine 96172 [running]:
      net/http.(*conn).serve.func1(0xc006e38640)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1746 +0xd0
      panic(0xebf1c0, 0x1d636a0)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/indexing/secondary/planner.findTopologyByBucket(0x0, 0xc00f4e4780, 0x1, 0xc005261550, 0xd, 0xe2adc0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/planner/proxy.go:816 +0x67
      github.com/couchbase/indexing/secondary/planner.ConvertToIndexUsage(0xc0102f5c80, 0xc004b5c1a0, 0xc0116cf120, 0x0, 0xf90960, 0x4, 0xc0116cefc8, 0x30)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/planner/proxy.go:269 +0x73
      github.com/couchbase/indexing/secondary/manager.(*RestoreContext).convertImage(0xc0116cf7d8, 0x0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/restore.go:131 +0x948
      github.com/couchbase/indexing/secondary/manager.(*RestoreContext).computeIndexLayout(0xc0116cf7d8, 0x1b, 0xc0116cf728, 0x1)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/restore.go:67 +0x50
      github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleRestoreIndexMetadataRequest(0x1dca620, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1096 +0x38f
      github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleRestoreIndexMetadataRequest-fm(0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:189 +0x48
      net/http.HandlerFunc.ServeHTTP(0xc000142460, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1964 +0x44
      net/http.(*ServeMux).ServeHTTP(0xc0001342a0, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2361 +0x127
      net/http.serverHandler.ServeHTTP(0xc0001228f0, 0x1199780, 0xc0064c60e0, 0xc005e99c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc006e38640, 0x119e700, 0xc006962500)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/net/http/server.go:2851 +0x2f5
      

      Steps to reproduce :
      1. Create a 1 node cluster with kv+query+index services
      2. Install all 3 sample buckets.
      3. Take a full backup.
      4. Delete the buckets
      5. Create empty buckets with the same names
      6. Restore from backup.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jliang John Liang
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty