Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
6.6.2-9588 -> 7.0.0-5016
-
Untriaged
-
1
-
No
Description
Steps to Repro
1. Create a 3 node cluster of 6.6.2-9588 with all the nodes having kv-index-n1ql services.
2. Install all 3 sample buckets: beer-sample, gamesim-sample and travel-sample
3. Wait for all the docs to be created and indexes to be created.
4. Do a offline upgrade one after the other.
I had enabled Auto failover initially, so after the first upgrade it had Failed over, so did a delta recovery and did a rebalance. For the other 2 nodes I disabled Auto failover and did and offline upgrade serially.
After upgrade I noticed only one index and all others were missing.
Post upgrade
Balakumarans-MacBook-Pro-2:testrunner balakumaran.g$ curl -u Administrator:password http://172.23.136.115:9102/getIndexStatus | jq
|
% Total % Received % Xferd Average Speed Time Time Time Current
|
Dload Upload Total Spent Left Speed
|
100 680 100 680 0 0 1320 0 --:--:-- --:--:-- --:--:-- 1322
|
{
|
"code": "success",
|
"status": [
|
{
|
"defnId": 16361052539014170000,
|
"instId": 16799931116015390000,
|
"name": "def_schedule_utc",
|
"bucket": "travel-sample",
|
"scope": "_default",
|
"collection": "_default",
|
"secExprs": [
|
"array (`s`.`utc`) for `s` in `schedule` end"
|
],
|
"indexType": "plasma",
|
"status": "Ready",
|
"definition": "CREATE INDEX `def_schedule_utc` ON `travel-sample`(array (`s`.`utc`) for `s` in `schedule` end) WITH { \"defer_build\":true }",
|
"hosts": [
|
"172.23.136.251:8091"
|
],
|
"completion": 100,
|
"progress": 100,
|
"scheduled": false,
|
"partitioned": false,
|
"numPartition": 1,
|
"partitionMap": {
|
"172.23.136.251:8091": [
|
0
|
]
|
},
|
"numReplica": 0,
|
"indexName": "def_schedule_utc",
|
"replicaId": 0,
|
"stale": false,
|
"lastScanTime": "NA"
|
}
|
]
|
}
|
Balakumarans-MacBook-Pro-2:testrunner balakumaran.g$
|
Found lot of suspicious messages like the following.
172.23.136.114
$ grep 'Found orphan index' *
|
indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\gamesim-sample_gamesim_primary_14287640395995850841_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_city_14691844915152536168_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_name_type_2503638052479337442_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:54:53.749-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_type_11033268586742495615_0.index. Scheduling it for cleanup.
|
grep: rebalance: Is a directory
|
172.23.136.115
$ grep 'Found orphan index' *
|
indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\beer-sample_beer_primary_13318156936120593294_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_airportname_3347637866298838521_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:47:50.401-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_faa_17557541258858760224_0.index. Scheduling it for cleanup.
|
grep: rebalance: Is a directory
|
172.23.136.251
$ grep 'Found orphan index' *
|
indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_icao_13200693036253856112_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_primary_17489209872933999265_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_route_src_dst_day_6596347943015610479_0.index. Scheduling it for cleanup.
|
indexer.log:2021-04-23T04:42:42.071-07:00 [Info] Found orphan index slice c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\travel-sample_def_sourceairport_5729675667190842624_0.index. Scheduling it for cleanup.
|
grep: rebalance: Is a directory
|
Attachments
| For Gerrit Dashboard: MB-45882 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 152448,2 | MB-45882 : [For-Testing] Check if MetadataStore is restored correctly | unstable | indexing | Status: ABANDONED | 0 | 0 |
| 152927,2 | MB-45882: [Windows] - Indexes are missing | cb-master | forestdb | Status: MERGED | +2 | +1 |