n1ql transaction failing due to unambiguous timeout

This one is a bit tricky because our test repros it consistently, but we haven't had a manual reproduction of the problem, will upload the logs and describe the test as well as provide the test, can create a live environment upon request. Here is the test command

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.94155.ini rerun=False,data_spec_name=initial_load_N1ql,GROUP=P1,upgrade_version=7.0.2-6683 -t N1qlTransaction.basic_ops.BasicOps.test_n1ql_query,num_stmt_txn=10,upgrade_version=7.0.2-6683,rerun=False,num_savepoints=7,num_collection=10,GROUP=P1,rollback_to_savepoint=True,bucket_spec=multi_bucket.buckets_for_N1ql_tests,data_spec_name=initial_load_N1ql,get-cbcollect-info=False,num_buckets=2,bucket_storage=couchstore,nodes_init=4,txntimeout=5'

4 node environment -
kv,n1ql,index - 172.23.104.110
kv -172.23.104.173
kv -172.23.104.174
kv -172.23.104.175

Basically the test creates 100s of indexes across multiple buckets/collections then runs a basic n1ql transaction, the transaction itself is randomized but so far we have seen the failure on a DELETE an UPDATE and an INSERT, it doesn't seem to matter what the dml is, and when I run the same transaction manually on the affected cluster the transaction seems to pass.

Here is an example of the error message we are seeing

{"signature":

{"id": "json"}

, "requestID": "c3da1cbe-4698-4da1-88e7-3e625ea8087e", "metrics":

{"transactionRemainingTime": "4m54.748017303s", "resultCount": 3, "executionTime": "4.639067492s", "serviceLoad": 1, "resultSize": 90, "mutationCount": 3, "transactionElapsedTime": "5.160604112s", "errorCount": 1, "elapsedTime": "4.639209211s"}

, "results": [

{"id": "test_collections-1038"}

,

{"id": "test_collections-1051"}

,

{"id": "test_collections-1062"}

], "errors": [{"msg": "Transaction fetch error", "code": 17017, "cause": {"rollback": true, "cause": {"c": "59c469f7f1a756bb/e2a4370af578fec1", "i": "0x8", "l": "172.23.104.90:48678", "-": {"InnerError": {"Message": "unambiguous timeout", "InnerError": {}}}, "r": "172.23.104.90:11210", "s": "LookupIn", "t": 2542012}, "raise": "failed", "retry": true}}], "status": "errors"}

The logs look slightly different for different runs that encounter this error, so i will attach the logs from my run and pierre will attach the logs from his run as well. It does not appear this problem was seen in 7.0.0 and we are checking 7.0.1 right now, those run results should be added here when we get them. One common thing we see in the query logs is this

_time=2021-09-24T13:28:53.506-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
_time=2021-09-24T13:28:53.506-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
_time=2021-09-24T13:29:10.125-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
_time=2021-09-24T13:29:10.609-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
_time=2021-09-24T13:29:24.178-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
_time=2021-09-24T13:29:24.180-07:00 _level=WARN _msg=(GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout

here is one of the transactions that hit the error, as I said each one was different, and it seems like running these transactions manually does not lead to the error

BEGIN WORK

DELETE FROM default:`6_E-8-534000`.`1s89-8-535000`.`8qsmo5-8-535000` WHERE citizen_of='INDIA' LIMIT 200 RETURNING meta().id

SELECT META().id,* from default:`6_E-8-534000`.`1s89-8-535000`.`8qsmo5-8-535000` WHERE META().id in []

SAVEPOINT a2

SAVEPOINT a6

SAVEPOINT a1

SELECT META().id,* from default:`9cVUXhUQTH-IzwJSps-8-536000`.`_default`.`_default` WHERE META().id in ['test_collections-0', 'test_collections-1', 'test_collections-1030', 'test_collections-1031', 'test_collections-1032', 'test_collections-1033', 'test_collections-1034', 'test_collections-1035', 'test_collections-1036', 'test_collections-1037', 'test_collections-1038', 'test_collections-1039', 'test_collections-1040', 'test_collections-1041', 'test_collections-1042', 'test_collections-1043', 'test_collections-1044', 'test_collections-1045', 'test_collections-1046', 'test_collections-1047', 'test_collections-1048', 'test_collections-1049', 'test_collections-1050', 'test_collections-1051', 'test_collections-1052', 'test_collections-1053', 'test_collections-1054', 'test_collections-1055', 'test_collections-1056', 'test_collections-1057', 'test_collections-1058', 'test_collections-1059', 'test_collections-1060', 'test_collections-1061', 'test_collections-1062', 'test_collections-1063', 'test_collections-1064', 'test_collections-1065', 'test_collections-1066', 'test_collections-1067', 'test_collections-1068', 'test_collections-1069', 'test_collections-1070', 'test_collections-1071', 'test_collections-1072', 'test_collections-1073', 'test_collections-1074', 'test_collections-1075', 'test_collections-1076', 'test_collections-1077', 'test_collections-1078', 'test_collections-1079', 'test_collections-1080', 'test_collections-1081', 'test_collections-1082', 'test_collections-1083', 'test_collections-1084', 'test_collections-1085', 'test_collections-1086', 'test_collections-1087', 'test_collections-1088', 'test_collections-1089', 'test_collections-1090', 'test_collections-1091', 'test_collections-1092', 'test_collections-1093', 'test_collections-1094', 'test_collections-1095', 'test_collections-1096', 'test_collections-1097', 'test_collections-1098', 'test_collections-1099', 'test_collections-2', 'test_collections-220', 'test_collections-2200', 'test_collections-2201', 'test_collections-2202', 'test_collections-2203', 'test_collections-2204', 'test_collections-2205', 'test_collections-2206', 'test_collections-2207', 'test_collections-2208', 'test_collections-2209', 'test_collections-221', 'test_collections-2210', 'test_collections-2211', 'test_collections-2212', 'test_collections-2213', 'test_collections-2214', 'test_collections-2215', 'test_collections-2216', 'test_collections-2217', 'test_collections-2218', 'test_collections-2219', 'test_collections-222', 'test_collections-2220', 'test_collections-2221', 'test_collections-2222', 'test_collections-2223', 'test_collections-2224', 'test_collections-2225', 'test_collections-2226', 'test_collections-2227', 'test_collections-2228', 'test_collections-2229', 'test_collections-223', 'test_collections-2230', 'test_collections-2231', 'test_collections-2232', 'test_collections-2233', 'test_collections-2234', 'test_collections-2235', 'test_collections-2236', 'test_collections-2237', 'test_collections-2238', 'test_collections-2239', 'test_collections-224', 'test_collections-2240', 'test_collections-2241', 'test_collections-2242', 'test_collections-2243', 'test_collections-2244', 'test_collections-2245', 'test_collections-2246', 'test_collections-2247', 'test_collections-2248', 'test_collections-2249', 'test_collections-225', 'test_collections-2250', 'test_collections-2251', 'test_collections-2252', 'test_collections-2253', 'test_collections-2254', 'test_collections-2255', 'test_collections-2256', 'test_collections-2257', 'test_collections-2258', 'test_collections-2259', 'test_collections-226', 'test_collections-2260', 'test_collections-2261', 'test_collections-2262', 'test_collections-2263', 'test_collections-2264', 'test_collections-2265', 'test_collections-2266', 'test_collections-2267', 'test_collections-2268', 'test_collections-2269', 'test_collections-227', 'test_collections-2270', 'test_collections-2271', 'test_collections-2272', 'test_collections-2273', 'test_collections-2274', 'test_collections-2275', 'test_collections-2276', 'test_collections-2277', 'test_collections-2278', 'test_collections-2279', 'test_collections-228', 'test_collections-2280', 'test_collections-2281', 'test_collections-2282', 'test_collections-2283', 'test_collections-2284', 'test_collections-2285', 'test_collections-2286', 'test_collections-2287', 'test_collections-2288', 'test_collections-2289', 'test_collections-229', 'test_collections-2290', 'test_collections-2291', 'test_collections-2292', 'test_collections-2293', 'test_collections-2294', 'test_collections-2295', 'test_collections-2296', 'test_collections-2297', 'test_collections-2298', 'test_collections-2299', 'test_collections-230', 'test_collections-2300', 'test_collections-2301', 'test_collections-2302', 'test_collections-2303', 'test_collections-2304', 'test_collections-2305', 'test_collections-2306', 'test_collections-2307', 'test_collections-2308', 'test_collections-2309', 'test_collections-231', 'test_collections-2310', 'test_collections-2311', 'test_collections-2312', 'test_collections-2313', 'test_collections-2314']

INSERT INTO default:`9cVUXhUQTH-IzwJSps-8-536000`.`fCIDQyeKdxPZ-8-537000`.`JQ-oRSQL6FZ-8-537000` (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:`9cVUXhUQTH-IzwJSps-8-536000`.`fCIDQyeKdxPZ-8-537000`.`JQ-oRSQL6FZ-8-537000` t WHERE citizen_of='USA' AND job_title='Sales' LIMIT 10 RETURNING *

SELECT * from default:`9cVUXhUQTH-IzwJSps-8-536000`.`fCIDQyeKdxPZ-8-537000`.`JQ-oRSQL6FZ-8-537000` WHERE META().id in ['Hedda Lee', 'Kacila Jones', 'Quella cox', 'Fantine Evans', 'Ebony Clark', 'Salina Wright', 'Sirena Hall', 'Severin Wilson', 'Treva Davis', 'Jerica Lee']

DELETE FROM default:`6_E-8-534000`.`yPSxubDnVGiAgBeAizQ-8-535000`.`fhtwWVeKT-8-536000` WHERE join_mo < 3 OR join_mo > 11 LIMIT 200 RETURNING meta().id

In pierres case his txn had no savepoints yet when it failed