Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
centos-launcher-2
-
Untriaged
-
-
Yes
Description
Build : 5.5.0-2938
Test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan_new.yml
Scale : 2
Iteration : 1
The test has a step to rebalance out 2 indexer nodes - 172.23.96.48 & 172.23.96.254. This rebalance operation is stuck for almost 20 hrs now because on node 172.23.96.122, index building activity for other-1.#primary is not getting completed.
From the stats, another discrepancy is observed. When I collected indexer stats first, following were the numbers for other-1.#primary :
"other-1:#primary:num_docs_indexed":96656563,
"other-1:#primary:num_docs_pending":50,
"other-1:#primary:num_docs_processed":157305700,
"other-1:#primary:num_docs_queued":56,
Then, after 1 min, when I collected stats again, the same stats show totally different numbers:
"other-1:#primary:num_docs_indexed":82702150,
"other-1:#primary:num_docs_pending":4419513,
"other-1:#primary:num_docs_processed":152954092,
"other-1:#primary:num_docs_queued":3,
How can the value for num_docs_processed go down? Also the ops rate for other-1 bucket is around 1700-1800 ops / sec, and the total item count is around 1.1M
The environment is available for debugging : http://172.23.96.206:8091
Attachments
| For Gerrit Dashboard: MB-30327 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 96419,1 | MB-30327 terminate projector on opaque mismatch | unstable | indexing | Status: MERGED | +2 | +1 |
| 96527,1 | MB-30327: Do not overwrite reqTs | unstable | indexing | Status: MERGED | +2 | +1 |