Details
Major Description
I created a 2 node cluster (kv,fts,n1ql : kv,fts,index) running Couchbase Server Enterprise 7.2.5 7596.
The fts quota is set to 2000Mb.
Now loaded 5000 documents and created 3 FTS indexes over it.
i. index_1 with 18 partitions
ii. index_2 with 3 partitions
iii. index_3 with 3 partitions
Now, wait_for_indexing_to_complete function is triggered along with additional 5 mins wait time to ensure all the partitions are created and indexing is complete.
Now, ran 20 queries against each index and the queries are successful.
Sample query results :
index 1 (18 partitions):
2024-07-22 04:34:58,076 - root - INFO - Running query {"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.129 : Administrator: |
2024-07-22 04:34:58,076 - root - INFO - Making a rest request api=http://172.23.106.129:8094/api/index/index_1/query verb=POST params=b'{"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:34:58,091 - root - INFO - Status: {'total': 18, 'failed': 0, 'successful': 18} |
2024-07-22 04:34:58,091 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 4.411881ms) |
index 2 :
2024-07-22 04:34:59,558 - root - INFO - Running query {"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.129 : Administrator: |
2024-07-22 04:34:59,558 - root - INFO - Making a rest request api=http://172.23.106.129:8094/api/index/index_2/query verb=POST params=b'{"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:34:59,573 - root - INFO - Status: {'total': 3, 'failed': 0, 'successful': 3} |
2024-07-22 04:34:59,573 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 4.278342ms) |
index 3 :
2024-07-22 04:35:00,983 - root - INFO - Running query {"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.129 : Administrator: |
2024-07-22 04:35:00,983 - root - INFO - Making a rest request api=http://172.23.106.129:8094/api/index/index_3/query verb=POST params=b'{"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:35:00,999 - root - INFO - Status: {'total': 3, 'failed': 0, 'successful': 3} |
2024-07-22 04:35:00,999 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 6.609102ms) |
Now upgrade starts, and one of the node is upgraded to 7.6.2 3721
Now ran the same queries against all the indexes and these were the results
index 1:
2024-07-22 04:41:04,825 - root - INFO - Running query {"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.104.74 : Administrator: |
2024-07-22 04:41:04,825 - root - INFO - Making a rest request api=http://172.23.104.74:8094/api/index/index_1/query verb=POST params=b'{"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:41:04,838 - root - INFO - Status: {'total': 20, 'failed': 1, 'successful': 19, 'errors': {'index_1_5d27ac67b4bb3475_22f98827': 'pindex not available'}} |
2024-07-22 04:41:04,839 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 703 (took 4.31447ms) |
index 2:
2024-07-22 04:41:06,572 - root - INFO - Running query {"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.104.74 : Administrator: |
2024-07-22 04:41:06,572 - root - INFO - Making a rest request api=http://172.23.104.74:8094/api/index/index_2/query verb=POST params=b'{"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:41:06,588 - root - INFO - Status: {'total': 5, 'failed': 1, 'successful': 4, 'errors': {'index_2_408f8af93d6927b9_a652076d': 'pindex not available'}} |
2024-07-22 04:41:06,588 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 903 (took 4.928466ms) |
index 3:
2024-07-22 04:41:08,439 - root - INFO - Running query {"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.104.74 : Administrator: |
2024-07-22 04:41:08,439 - root - INFO - Making a rest request api=http://172.23.104.74:8094/api/index/index_3/query verb=POST params=b'{"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:41:08,454 - root - INFO - Status: {'total': 5, 'failed': 1, 'successful': 4, 'errors': {'index_3_255bd481c79f14b8_6a5f0298': 'pindex not available'}} |
2024-07-22 04:41:08,455 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 899 (took 4.924573ms) |
As seen in status of the above query results, one of the index partitions is missing in all the indexes.
After upgrade: ( both the nodes are in 7.6.2 3721 version now ) and same queries are ran against all the 3 indexes
index 1:
2024-07-22 04:49:12,807 - root - INFO - Running query {"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.130 : Administrator: |
2024-07-22 04:49:12,807 - root - INFO - Making a rest request api=http://172.23.106.130:8094/api/index/index_1/query verb=POST params=b'{"indexName": "index_1", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:49:12,820 - root - INFO - Status: {'total': 20, 'failed': 2, 'successful': 18, 'errors': {'index_1_5d27ac67b4bb3475_22f98827': 'pindex not available', 'index_1_5d27ac67b4bb3475_eded1b4d': 'pindex not available'}} |
2024-07-22 04:49:12,820 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 4.038832ms) |
index 2:
2024-07-22 04:49:14,974 - root - INFO - Running query {"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.130 : Administrator: |
2024-07-22 04:49:14,975 - root - INFO - Making a rest request api=http://172.23.106.130:8094/api/index/index_2/query verb=POST params=b'{"indexName": "index_2", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:49:14,986 - root - INFO - Status: {'total': 5, 'failed': 2, 'successful': 3, 'errors': {'index_2_408f8af93d6927b9_6a5f0298': 'pindex not available', 'index_2_408f8af93d6927b9_a652076d': 'pindex not available'}} |
2024-07-22 04:49:14,986 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 3.35056ms) |
index 3:
2024-07-22 04:49:17,047 - root - INFO - Running query {"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.106.130 : Administrator: |
2024-07-22 04:49:17,047 - root - INFO - Making a rest request api=http://172.23.106.130:8094/api/index/index_3/query verb=POST params=b'{"indexName": "index_3", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-07-22 04:49:17,060 - root - INFO - Status: {'total': 5, 'failed': 2, 'successful': 3, 'errors': {'index_3_255bd481c79f14b8_6a5f0298': 'pindex not available', 'index_3_255bd481c79f14b8_a652076d': 'pindex not available'}} |
2024-07-22 04:49:17,060 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 672 (took 4.905551ms) |
One more thing here is, the partitions are increased as well.
index_1 partition count is increased from 18 to 20 and index_2, index_3 partitions are increased from 3 to 5 and this issue is addressed here.
To verify it further, I performed the same scenario manually with different partitions and same issues were observed.
