Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
7.0.4
-
None
-
7.0.4 build 7265
-
Untriaged
-
-
1
-
No
Description
Note: This is found while testing for MB-51751
Steps:
- Create a 2 KV and 1 index/query node cluster.
- Create couchstore bucket and collections.
- Create 500000000 items sequentially
- Update 500000000 RandonKey keys to create 50 percent fragmentation
- Create 500000000 items sequentially
- Update 500000000 RandonKey keys to create 50 percent fragmentation
- Create an index idx0. Wait for index building.
- Rebalance in KV with Loading of docs. Rebalance completed with progress: 100% in 6609.36799979 sec
- Rebalance Out KV with Loading of docs. Rebalance completed with progress: 100% in 6881.47799993 sec
- Rebalance In_Out KV with Loading of docs. Rebalance completed with progress: 100% in 9201.84899998 sec
- Swap KV node with Loading of docs. Rebalance completed with progress: 100% in 8248.52999997 sec
- Failover 1 KV node and RebalanceOut that node with loading in parallel. Rebalance completed in 17713.875 sec
- Failover 1 KV node and FullRecovery that node. While rebalance is around 15% create an index on body field. Index creating was running fine upto around 2TB index disk size then crashes are observed and index is stuck in warmup.
First Crash:
Service 'indexer' exited with status 137. Restarting. Messages:
|
2022-04-29T12:43:29.466-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 108000 Milliseconds KeyspaceId GleamBookUsers0:_default:VolumeCollection0 Vbucket 939
|
2022-04-29T12:43:29.470-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 108000 Milliseconds KeyspaceId GleamBookUsers0:_default:VolumeCollection0 Vbucket 598
|
2022-04-29T12:43:29.524-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 108000 Milliseconds KeyspaceId GleamBookUsers0:_default:VolumeCollection0 Vbucket 861
|
2022-04-29T12:43:29.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 108000 Milliseconds KeyspaceId GleamBookUsers0:_default:VolumeCollection0 Vbucket 916
|
2022-04-29T12:43:29.715-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 108000 Milliseconds KeyspaceId GleamBookUsers0:_default:VolumeCollection0 Vbucket 614
|
2022-04-29T12:43:29.966-07:00 [Info] ServiceMgr::rebalanceJanitor Running Periodic Cleanup
|
2022-04-29T12:43:32.045-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 96000 Milliseconds KeyspaceId GleamBookUsers0 Vbucket 460
|
Subsequent Crash:
Service 'indexer' exited with status 137. Restarting. Messages:
|
2022-04-29T13:11:16.618-07:00 [Info] Plasma: Adaptive memory quota tuning RSS:256687058944, freePercent:2.383766351077545, currentQuota=38842505625
|
2022-04-29T13:11:21.618-07:00 [Info] Plasma: Adaptive memory quota tuning RSS:257139044352, freePercent:2.2195230649016877, currentQuota=38842505625
|
2022-04-29T13:11:26.619-07:00 [Info] Plasma: Adaptive memory quota tuning RSS:257566330880, freePercent:2.056442385756443, currentQuota=38842505625
|
2022-04-29T13:11:27.000-07:00 [Info] [Queryport ":9101"] connection "172.23.110.66:43548" doAuth() ...
|
2022-04-29T13:11:27.000-07:00 [Info] [Queryport ":9101"] connection "172.23.110.66:43548" doReceive() ...
|
2022-04-29T13:11:31.619-07:00 [Info] Plasma: Adaptive memory quota tuning RSS:259607367680, freePercent:1.3007113895283438, currentQuota=38842505625
|
2022-04-29T13:11:36.620-07:00 [Info] Plasma: Adaptive memory quota tuning RSS:259969609728, freePercent:1.1643175503560241, currentQuota=38842505625
|
2022-04-29T13:11:38.984-07:00 [Info] [Queryport ":9101"] connection "172.23.110.66:43552" doAuth() ...
|
2022-04-29T13:11:38.985-07:00 [Info] [Queryport ":9101"] connection "172.23.110.66:43552" doReceive() ...
|
|
Doc body |
{
|
"name": "DJohn",
|
"age": 9,
|
"animals": ["China rose", "Chinese red"],
|
"attributes": {
|
"hair": "Dark magenta",
|
"dimensions": {
|
"height": 93,
|
"weight": 81
|
},
|
"hobbies": [{
|
"type": "Men",
|
"name": "Women",
|
"details": {
|
"location": {
|
"lat": 99.93546078930339,
|
"lon": 58.01158976753777
|
}
|
}
|
}]
|
},
|
"gender": "F",
|
"marital": "W",
|
"mutated": 0,
|
"body": "PAF9J3XGOtuXwVIf9ktRZmWZWtTHrDcyEv0KmLB6EKSyX8mdUWDRNfFsTDQlt3h6HvLyDxcBfFKZWrNdBccR7TddipfgTu7NIAtsfqU3npe6VJDDNlHpXcaefzXgE1X9yrTb2lC6XXhzqCfzAJyAejnq5fg0tdVoZm27n7wF1rOdTHtPy7pP3cb3pndlMlgjSs7fPtmqO9zV00ct0YmWm85hPdmXYu7Y9pd4LgASshbbZlJ2jkaiVjk5RhWTUtmdwGwSpvUnltsoVxl4mKUh9sKhdDg7NTx2kdfTMp5BbTfBif1vl1s66QkkhTSFzt9nBSimvwPf6pkW80X7Jv2tueyz23HQfZaUZxh4q1GQLjnigN2xCiIX53ei8DX93QIEiWuNIqagrp1FZvMgpK7FxG0xjdswCV1tXFl0uH79oWErfNpmSdUhIMg2maYMR7LvkUIYM3gq1lfwZs4I5hbg2QT4FTVWeMu6PPkbT8VZHKEEo33Ux4M2xT14Sq2WPHrBBMonSSaYVmCTHSd8eNOdDfiwYWblG44Hcfy93YnsvpeyJlBV01BYwrGuHTHp3CzCYxo0NRcIHoUVkpzwNiESeRZw9ftMaa6ZsmxEbIjIBHdne8dxwHUCvgMlbl8cpRpsmDNm1UH0GVKXuF6oChzMZwxefhTjqQP7QIH24Ow9ZaS5pfq2YmgACcupt2a5DOwlpk6w9fW0n8qMMebLAPHXTOjoMQAx8QO3LNACm0ZLiXmZIXBtjcK2OSUECdoUtM7VSgqKqid2LzLS5k8lkjrWdNaW9vLx8nv70WoaGT3DX25togCOPgOFdGLIaeQGiXiKkV4ud6thCuO5h4V2IhyrrucUjOVpzLz8DpDcKTFDo3MZZ74ZZ6ltsqM64WzYhPgjIW48PNRjj3DDNSd8sZIuQxUp9ofWVwj6xQ9ZKgtRpL8JrUe1qWKcUEBEaxkCESKeDJpL85wJj73szo6eHfY5H5frK1dpNNbJzxhPgZ4mTcTTRw9c40p8DahqKSgSW27pnRkm9L9VwKFnP1giNx96fY9K8NaV1FqkJkTckKDqQEacp5fUAqNP8IblIWiWaKm9tWtTbHCP5M3Gz5I6UQKehRqdQ45VGKDfzeYOoL9lgTSkJVTJJt0a8JSA6WE1z2XeQdRkwNQrtq8DXiDZF46LNFIgbA21i3roRbdBMg89IpkikSPUPHz7bgrZWQ5D8oADhYUY201Apoctk0npOCjlNvbaDo41RGxQgujc32wehhJl32dl7js27lOT2mu8A2YsLQ1XreAvcKHqu1cgk3p4UHhoLGtVD8kLcQzf8DliKZIeekBWwESaYUO40sNZXPeswGK34mklO8FmJQwUHU7TjxtqaSdvT3MxBMb0ZeWygu7ngCWgJjdflNulTChXPVD2iO7GlpmyS6yZgrbsN7kIaxVwbvwJDbfSBSWCWYP7rsyI2cVYf2ZfASsU70Ng1UfmC2dpfufxpXMqE8vqbWHnHFOXZTylnh3S1YntjYjLmaqfzkJc6NyoN7TrPbQUUpyg86daVUJedDQXsvKm0FbdGkLaMi7RTaFlOoAhMufzKV3NHonR9NHjgDyEShVV5SYpi7VpCtfP5CTwt9AUZshGzM0pNUl9EIjmsdkU7gMiq8xuD1xw5IQ4xUUPcKWst9vh2dcJhOS0aMNZjI1cvGw4rhkXNzOo5ohVG3CiOMOLpqsz8ripJYObY6EJWjG1QGzAqiB3p4D53j9e3EsBYCgFQgKLqY9UJDUslYqAtaGd7JCy8EtQsDgQAS22XcUuhT0wVit9QdShhamiA23VtBmRwYuH5JITFNqsdrLsVANqtc1g2FaZaJMVnMl4FbA5TwesBDHbqeA0e7FJRIoG1dh7y3RTNg6m0LjT32SIEX3hMCcGVmcsuhpLAOWuB6zOnmKuIQ5vWueYJBwFFScsI7jZbwAQNH5hguBRAz5nR3ybObWXerIIbla6MShc3E6akeoCHdqYy8LVbM4syafCVxP7OkQ8sxIorJaUP6PHb0i4cohGUYRfHzRjdeL6xz7ZQV3DWzvb2rCEZKihPKJC12II4hmBeIx12ADkD3eEuRjWbWw5B4jtF2fYaQ1HtpPQE1CMbUX60d93OHZCF56CQ2ra8wA6c6S6bpiWe5mmLkL3GzaN0576kmpySnWQdQWDThzxiPIBeNlYMxdt2LqFDWsJy6Z1VMFE8wLyhGP75n7tqpWVsFeAP1OYf6DHtc2VtaL0Rdnfx4JaHPPp1e4fwpijruocBKT1T1tYZGFStAdahDFB91UIjx53siMiLeVdxGDdiZVuDg5A8X2wlDesPc6SkEW9tqRXccNF9CgX7s8zeNTOKn9C6IyFuLqDTehqwqZbs30ocMa4xZAASMdg89D0jHayI4WnWYcJ1iIEOjjsSDbDYD6IvPKQA66NEUCqRPmDUhL9rrf4urJJvvaRDUmg9AW2rENPDLHGQaDgNxsVXGw5sdR57oCUDNGZ8Z1ZJfJ9eRfGy2D6htVVbshjUHbnpSRtlqEQE5xuh6ADuvol0scE07yxXqnBnZZ3aHlLOmN1DHDZd0V38kxqxHdJTKDuWKyrYzYhDLmtm8B5zFQ6ufIR9NSC9fadb8CFPOkw6Z5eyHSyW7jO6TI5uXP2Cy1dgjdITysvnR1H5IFouscOh5b5U88CwQ53y6u6hhZKcmr80uUytUeLbxkrmsRPq5JJKlNHzE4im53bcJPUE1J10YI9IcTvZsajLPW6pFpOsvwbWSYGhfbE84ovOs28p3sGTcPaCNXAUvcqIW3egTKddNGqwdIkmoEN6783kMDmZv9rqvJYcPs4MpzxJa1s0Ch9QLBh5eyBRkJ4Y0Qwm18VmS9PaIvFQHXUeDrVeHdWFO7ZR0deQyuFv2s8KebVKIrYeEiJVQAvAa5u6dTepa2jbjqoVZfcefrCy3mCenoH0ybNcJAkqYOna8XQckTMRWttb9AwZ9CLSPXdDSjnKiSNqHiFamp2FBBHw1BpwI1F5zvh3y41Gnucg7zmRKNxsH9WAQjicUo9cZkEmOs7STZo0QPTtUMTEaQJOJj9435C2LxBtNjO6xR7Zs3qhUYkGvWDwhPjogxvDE8sA1vktSCYLAgUjRZhzAUGC6V1Sffgzb4DtD9d2pjugyTsLFlHqtZb3BeDY3mcxHiZvuJgm3xFhnocryGQ5uEDWocHEjdgwNgise9eoB03W48WTxOYH8S88CgJl3i8jyXAsqmvAwVfr3idlcerzxxKWld6OzyEn1Y6W2B65f4J6KgW20oifSkIaqLthFpH8kRKWXgQQ3EQwz1LZsUOXMkB96kMwRpgCTukIXB7hQPiXSR9Wh6fodsp2wDnpjUpw5t8XdWn3PCTJhb9uC8dr3jBNhefYtWgqQIObDgVrOoSxSDzHarVlIFWzOrA0lE6AuOmijomrU3AHlB76WFedxhMNgS4LnI24ez4KDKs9p1pVKsAOnSaaJUtxSNyrYrR70hL0jwO0qcPrKOFdeSLLdrKC3kZzxw9UdewYVTL6jkhw8qYsmHPOUnWZgE7fWvEioo0qfflnjKDZEokk8ON1m7dgM5OxApP5W2sXO3Mhwu62JnWsHZEdHlG8oJV0UG3KUYZjSU5MPAEtRa2dQ1yAJGAY1iC8KD9QFClkeqdRF8tUKtgJR5HftJtKv8IDVqxP1E1DAWke3rqdjvTO32GsbftdTOOWRdPEokRewpbLoRzCC1wbKTDLcx98zWjbYzKowdXhFUqrd2LocGddIqiBs3uhvgEUWqb73Pg93qrdpKfUzLzveYWMaEcBeOwIfmgCoEc4N4kra5NSXEs4yJsoyhl76KWtnaXYSwYguT92Nw5ke8iIcva9INIC7g55QZlTWXR7ELA3kmZ0EJzWTHSzAfm2Ur6lIxpEKApMVF97idAJvvHRCxfpwT8bKnldRu3sP1iVfeGMpefpZG8DEvNkcSylWeF1V4IU3L2aA251eYggxc0KYtHciSqhd5bvKgR3hnOi8Y7bp3VnLQ7x3dQ5gtvLE1UZMXm5zeCZ5m6M3mUiSKhOgBu1IjGlKiHumXzSO70QHGNvu5gTQXp99vvoaB8SFdbhecBF38MPAF9J3XGOtuXwVIf9ktRZmWZWtTHrDcyEv0KmLB6EKSyX8mdUWDRNfFsTDQlt3h6HvLyDxcBfFKZWrNdBccR7TddipfgTu7NIAtsfqU3npe6VJDDNlHpXcaefzXgE1X9yrTb2lC6XXhzqCfzAJyAejnq5fg0tdVoZm27n7wF1rOdTHtPy7pP3cb3pndlMlgjSs7fPtmqO9zV00ct0YmWm85hPdmXYu7Y9pd4LgASshbbZlJ2jkaiVjk5RhWTUtmdwGwSpvUnltsoVxl4mKUh9sKhdDg7NTx2kdfTMp5BbTfBif1vl1s66QkkhTSFzt9nBSimvwPf6pkW80X7Jv2tueyz23HQfZaUZxh4q1GQLjnigN2xCiIX53ei8DX93QIEiWuNIqagrp1FZvMgpK7FxG0xjdswCV1tXFl0uH79oWErfNpmSdUhIMg2maYMR7LvkUIYM3gq1lfwZs4I5hbg2QT4FTVWeMu6PPkbT8VZHKEEo33Ux4M2xT14Sq2WPHrBBMonSSaYVmCTHSd8eNOdDfiwYWblG44Hcfy93YnsvpeyJlBV01BYwrGuHT"
|
}
|