Details
-
Bug
-
Resolution: Unresolved
-
Major
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-5274 (RC4)
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 3
Iteration : 3rd
Day : 5th
Towards the end of the iteration, when all indexes are being dropped, for index idx1_A1gBCTz5 on bucket6.scope_5.coll_3, there was an error returned to query :
[{u'completed_requests': {u'node': u'172.23.97.150:8091', u'users': u'Administrator', u'errors': [{u'icause': u'Fail to Drop Index due to internal errors. Cleanup may happen in the background. Error=DeleteScheduleCreateToken:15002607395196254441:Rev mismatch.', u'message': u'GSI Drop()', u'code': 5000, u'key': u'Internal Error'}], u'scanConsistency': u'unbounded', u'state': u'completed', u'phaseOperators': {u'authorize': 1}, u'serviceTime': u'1.487378867s', u'remoteAddr': u'172.23.104.254:44208', u'elapsedTime': u'1.487459992s', u'resultSize': 0, u'requestTime': u'2021-06-09T13:57:35.247-07:00', u'statement': u'DROP INDEX `idx1_A1gBCTz5` on `bucket6`.`scope_5`.`coll_3`;', u'requestId': u'9dfd451b-b25f-4899-91d5-f3d3824ecb38', u'clientContextID': u'81723b18b94ca925', u'userAgent': u'libcouchbase/3.0.6 (Linux-3.10.0-1062.9.1.el7.x86_64; x86_64; GNU 9.3.0) PYCBC/3.0.10', u'useCBO': True, u'resultCount':
|
0, u'errorCount': 1, u'phaseTimes': {u'authorize': u'36.042\xb5s', u'parse': u'584.27\xb5s', u'instantiate': u'20.384\xb5s', u'run': u'1.485203958s', u'plan': u'1.537114ms'}}}]
|
The following can be seen on the query logs on 172.23.97.150:
2021-06-09T13:57:36.734-07:00 [Error] MetakvRecurciveDel: Failed to delete /indexing/ddl/commandToken/schedule/15002607395196254441/: Rev mismatch
|
2021-06-09T13:57:36.734-07:00 [Info] DropIndex 15002607395196254441 - elapsed(1.484941097s), err(Fail to Drop Index due to internal errors. Cleanup may happen in the background. Error=DeleteScheduleCreateToken:15002607395196254441:Rev mismatch.)
|
Also in the metakv logs on 172.23.97.150, the following can be seen -
[metakv:debug,2021-06-09T13:57:35.342-07:00,ns_1@172.23.97.150:<0.10951.1785>:menelaus_metakv:handle_iterate:127]Starting iteration of /indexing/ddl/commandToken/schedule/15002607395196254441/. Continuous = false
|
...
|
[metakv:debug,2021-06-09T13:57:35.477-07:00,ns_1@172.23.97.150:<0.13530.1785>:menelaus_metakv:handle_recursive_delete:114]handle_recursive_delete_post for <<"/indexing/ddl/commandToken/schedule/15002607395196254441/">>
|
...
|
[metakv:debug,2021-06-09T13:57:36.733-07:00,ns_1@172.23.97.150:<0.13530.1785>:menelaus_metakv:handle_recursive_delete:120]Recursive deletion failed for <<"/indexing/ddl/commandToken/schedule/15002607395196254441/">> with error retry_needed.
|
On indexer node 172.23.120.75, following error can be seen in the indexer logs much later -
2021-06-09T14:54:02.061-07:00 [Error] MetakvRecurciveDel: Failed to delete /indexing/ddl/commandToken/schedule/15002607395196254441/: Rev mismatch
|
2021-06-09T14:54:02.061-07:00 [Error] janitor: Failed to delete scheduled tokens upon cleanup for /indexing/ddl/commandToken/delete/15002607395196254441. Internal Error = DeleteScheduleCreateToken:15002607395196254441:Rev mismatch.
|
Indexer nodes : 172.23.120.75, 172.23.120.81, 172.23.96.243, 172.23.97.105, 172.23.97.110
Attachments
Issue Links
- relates to
-
-
- Closed
-