Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4122
- Cluster with 2 kv+n1ql, 1 index
- Create a bucket, one scope and 2 collections in it.
- Create 10 indexes: 5 indexes on each collections
- Load documents to both collections till the indexes are in DGM
[2021-01-17 17:16:41,767] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_1job_title1 , resident_ratio : 0.1883
[2021-01-17 17:16:41,768] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_1job_title4 , resident_ratio : 0.14815
[2021-01-17 17:16:41,768] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_2job_title0 , resident_ratio : 0.1246
[2021-01-17 17:16:41,770] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_2job_title2 , resident_ratio : 0.11084
[2021-01-17 17:16:41,770] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_1job_title3 , resident_ratio : 0.17011
[2021-01-17 17:16:41,770] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_2job_title1 , resident_ratio : 0.13468
[2021-01-17 17:16:41,770] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_1job_title2 , resident_ratio : 0.17741
[2021-01-17 17:16:41,771] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_1job_title0 , resident_ratio : 0.17685
[2021-01-17 17:16:41,771] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_2job_title4 , resident_ratio : 0.21078
[2021-01-17 17:16:41,771] - [base_gsi:1273] INFO - Index : idx_test_scope_1_test_collection_2job_title3 , resident_ratio : 0.13722
- Once they are in DGM, unmount the disk on index node and continue to load documents
[2021-01-17 17:16:52,994] - [remote_util:3395] INFO - running command.raw on 172.23.121.58: umount -l /data; df -Th
- Mount the disk back after around 20 mins:
[2021-01-17 17:41:26,075] - [remote_util:3395] INFO - running command.raw on 172.23.121.58: mount -o loop,rw,usrquota,grpquota /usr/disk-img/disk-quota.ext3 /data; df -Th
- Indexes created before the disk_failure are cleaned up
2021-01-17T17:41:29.888-08:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
2021-01-17T17:41:29.888-08:00 [Info] ClustMgr:handleUpdateTopologyForIndex &{56 [] {true true false false true false false} 0 false <nil>}
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title0_845567171095108806_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title1_10782895665027593881_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title2_10403861901164089539_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title3_15278000801494084819_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title4_2697525028955822799_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title0_4985990153750169428_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title1_13211418559035801781_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title2_7455346437267424134_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title3_17458332207778686093_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Found orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title4_5515611106417479574_0.index. Scheduling it for cleanup.
2021-01-17T17:41:29.888-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title0_845567171095108806_0.index.
2021-01-17T17:41:29.889-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title1_10782895665027593881_0.index.
2021-01-17T17:41:29.889-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title2_10403861901164089539_0.index.
2021-01-17T17:41:29.889-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title3_15278000801494084819_0.index.
2021-01-17T17:41:29.889-08:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set
2021-01-17T17:41:29.889-08:00 [Info] DDLServiceMgr: intialized. Local nodeUUID c688852077cc16da57d484ff2d7269d2
2021-01-17T17:41:29.889-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_1job_title4_2697525028955822799_0.index.
2021-01-17T17:41:29.890-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title0_4985990153750169428_0.index.
2021-01-17T17:41:29.890-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title1_13211418559035801781_0.index.
2021-01-17T17:41:29.890-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title2_7455346437267424134_0.index.
2021-01-17T17:41:29.891-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title3_17458332207778686093_0.index.
2021-01-17T17:41:29.891-08:00 [Info] Cleaned up the orphan index slice /data/@2i/test_bucket_idx_test_scope_1_test_collection_2job_title4_5515611106417479574_0.index.
2021-01-17T17:41:29.893-08:00 [Info] Indexer::local storage mode plasma
In the test, after the disk failure resolved, we expect these indexes to come online and serve the mutations came in during system failure.
Logs are attached. 172.23.121.58 is the index node.