Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45882

[Windows] - Indexes are missing after doing offline upgrade from 6.6.2 -> 7.0.0

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      Steps to Repro
      1. Create a 3 node cluster of 6.6.2-9588 with all the nodes having kv-index-n1ql services.
      2. Install all 3 sample buckets: beer-sample, gamesim-sample and travel-sample
      3. Wait for all the docs to be created and indexes to be created.
      4. Do a offline upgrade one after the other.

      I had enabled Auto failover initially, so after the first upgrade it had Failed over, so did a delta recovery and did a rebalance. For the other 2 nodes I disabled Auto failover and did and offline upgrade serially.

      After upgrade I noticed only one index and all others were missing.
      Post upgrade

      Balakumarans-MacBook-Pro-2:testrunner balakumaran.g$ curl -u Administrator:password http://172.23.136.115:9102/getIndexStatus | jq
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   680  100   680    0     0   1320      0 --:--:-- --:--:-- --:--:--  1322
      {
        "code": "success",
        "status": [
          {
            "defnId": 16361052539014170000,
            "instId": 16799931116015390000,
            "name": "def_schedule_utc",
            "bucket": "travel-sample",
            "scope": "_default",
            "collection": "_default",
            "secExprs": [
              "array (`s`.`utc`) for `s` in `schedule` end"
            ],
            "indexType": "plasma",
            "status": "Ready",
            "definition": "CREATE INDEX `def_schedule_utc` ON `travel-sample`(array (`s`.`utc`) for `s` in `schedule` end) WITH {  \"defer_build\":true }",
            "hosts": [
              "172.23.136.251:8091"
            ],
            "completion": 100,
            "progress": 100,
            "scheduled": false,
            "partitioned": false,
            "numPartition": 1,
            "partitionMap": {
              "172.23.136.251:8091": [
                0
              ]
            },
            "numReplica": 0,
            "indexName": "def_schedule_utc",
            "replicaId": 0,
            "stale": false,
            "lastScanTime": "NA"
          }
        ]
      }
      Balakumarans-MacBook-Pro-2:testrunner balakumaran.g$ 
      

      Found lot of suspicious messages like the following.
      172.23.136.114

      $ grep 'Found orphan index' *
      indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\gamesim-sample_gamesim_primary_14287640395995850841_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_city_14691844915152536168_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_name_type_2503638052479337442_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_type_11033268586742495615_0.index. Scheduling it for cleanup.
      grep: rebalance: Is a directory
      

      172.23.136.115

      $ grep 'Found orphan index' *
      indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\beer-sample_beer_primary_13318156936120593294_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_airportname_3347637866298838521_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_faa_17557541258858760224_0.index. Scheduling it for cleanup.
      grep: rebalance: Is a directory
      

      172.23.136.251

      $ grep 'Found orphan index' *
      indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_icao_13200693036253856112_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_primary_17489209872933999265_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_route_src_dst_day_6596347943015610479_0.index. Scheduling it for cleanup.
      indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_sourceairport_5729675667190842624_0.index. Scheduling it for cleanup.
      grep: rebalance: Is a directory
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Hi Srinath Duvuru

          Bala can give a full spin of the test with the fix.
          I had simple windows VM running to gave it a quick try and found that the fix works
          Attached the logs and data dir for reference.

          2021-05-05T19:57:35.878+00:00 [Info] Repo.upgradeAndOpenDBFile(c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore): Cannot open with COMPACT_AUTO mode. Trying with COMPACT_MANUAL mode.
          2021-05-05T19:57:35.887+00:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore
          2021-05-05T19:57:36.446+00:00 [Info] Repo.upgradeAndOpenDBFile(c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore): Switching to COMPACT_AUTO mode
          2021-05-05T19:57:36.446+00:00 [INFO][FDB] Forestdb closed database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore
          2021-05-05T19:57:36.450+00:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore.1
          

          2021-05-05T19:57:37.232+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 
          	InstanceId: 1798238460450055546 Name: def_name_type Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 17468912019050442230 Name: def_type Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 15808607274263720670 Name: def_route_src_dst_day Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 17959832328128226959 Name: def_airportname Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 4273994875517253534 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 2739763579406926261 Name: def_schedule_utc Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 709142411737105488 Name: def_icao Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 2306834522883516331 Name: def_sourceairport Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 13427002854782164621 Name: def_city Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          	InstanceId: 2029960645492317771 Name: def_faa Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 
          , elapsed: 22.9007ms
          

          Thanks
          Sai

          [^win_withFix_collectinfo-2021-05-05T195835-ns_1@127.0.0.1.zip] win_withFix_data.zip

          sai.teja Sai Krishna Teja added a comment - Hi Srinath Duvuru Bala can give a full spin of the test with the fix. I had simple windows VM running to gave it a quick try and found that the fix works Attached the logs and data dir for reference. 2021-05-05T19:57:35.878+00:00 [Info] Repo.upgradeAndOpenDBFile(c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore): Cannot open with COMPACT_AUTO mode. Trying with COMPACT_MANUAL mode. 2021-05-05T19:57:35.887+00:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore 2021-05-05T19:57:36.446+00:00 [Info] Repo.upgradeAndOpenDBFile(c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore): Switching to COMPACT_AUTO mode 2021-05-05T19:57:36.446+00:00 [INFO][FDB] Forestdb closed database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore 2021-05-05T19:57:36.450+00:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\MetadataStore.1 2021-05-05T19:57:37.232+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 1798238460450055546 Name: def_name_type Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 17468912019050442230 Name: def_type Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 15808607274263720670 Name: def_route_src_dst_day Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 17959832328128226959 Name: def_airportname Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 4273994875517253534 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 2739763579406926261 Name: def_schedule_utc Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 709142411737105488 Name: def_icao Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 2306834522883516331 Name: def_sourceairport Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 13427002854782164621 Name: def_city Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 InstanceId: 2029960645492317771 Name: def_faa Keyspace: travel-sample:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 22.9007ms Thanks Sai [^win_withFix_collectinfo-2021-05-05T195835-ns_1@127.0.0.1.zip] win_withFix_data.zip

          Sai Krishna Teja Thank you. Appreciate the help.

          srinath.duvuru Srinath Duvuru added a comment - Sai Krishna Teja  Thank you. Appreciate the help.
          srinath.duvuru Srinath Duvuru added a comment - - edited

          I was also able to verify the basic upgrade works with the fix. Will push the changes and let Balakumaran Gopal verify the regular build.

          srinath.duvuru Srinath Duvuru added a comment - - edited I was also able to verify the basic upgrade works with the fix. Will push the changes and let Balakumaran Gopal  verify the regular build.

          Build couchbase-server-7.0.0-5111 contains forestdb commit 43c6128 with commit message:
          MB-45882: [Windows] - Indexes are missing

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5111 contains forestdb commit 43c6128 with commit message: MB-45882 : [Windows] - Indexes are missing

          Validated this on Enterprise Edition 7.0.0 build 5112 ‧ IPv4 © 2021 Couchbase, Inc.
          See below for the o/p of the rest api for the results post upgrade. Marking this closed.

          Balakumarans-MacBook-Pro-2:TAF balakumaran.g$ curl -u Administrator:password http://172.23.106.120:9102/getIndexStatus | jq | grep name | wc -l
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100  5417    0  5417    0     0  10663      0 --:--:-- --:--:-- --:--:-- 10663
                12
          

          Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on Enterprise Edition 7.0.0 build 5112 ‧ IPv4 © 2021 Couchbase, Inc. See below for the o/p of the rest api for the results post upgrade. Marking this closed. Balakumarans-MacBook-Pro-2:TAF balakumaran.g$ curl -u Administrator:password http://172.23.106.120:9102/getIndexStatus | jq | grep name | wc -l % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5417 0 5417 0 0 10663 0 --:--:-- --:--:-- --:--:-- 10663 12

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty