./testrunner -i /tmp/atomicity.ini rerun=False,replicas=0,get-cbcollect-info=True -t N1qlTransaction.basic_ops.BasicOps.test_concurrent_txn,nodes_init=2,num_stmt_txn=8,num_collection=3,num_savepoints=3,override_savepoint=2,num_txn=4,num_commit=4,bucket_spec=single_bucket.def_scope_ten_collections Test Input params: {'cluster_name': 'atomicity', 'num_stmt_txn': '8', 'ini': '/tmp/atomicity.ini', 'get-cbcollect-info': 'True', 'replicas': '0', 'conf_file': 'conf/N1qlTransaction/concurrency.conf', 'spec': 'concurrency', 'rerun': 'False', 'num_nodes': 2, 'logs_folder': '/data/workspace/temp_atomicity_phase1/logs/testrunner-20-Sep-24_20-54-18/test_14', 'num_txn': '4', 'nodes_init': '2', 'num_collection': '3', 'num_savepoints': '3', 'num_commit': '4', 'override_savepoint': '2', 'bucket_spec': 'single_bucket.def_scope_ten_collections', 'case_number': 14} test_concurrent_txn (N1qlTransaction.basic_ops.BasicOps) ... 2020-09-24 22:07:28,177 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup started for test #14 test_concurrent_txn ========= 2020-09-24 22:07:28,612 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup started for test #14 test_concurrent_txn ========= 2020-09-24 22:07:28,772 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2020-09-24 22:07:28,928 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:572] RAM quota was defined less than 100 MB: 2020-09-24 22:07:40,598 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: default, Nodes: 172.23.105.229, 172.23.105.230 initialized 2020-09-24 22:07:40,609 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup finished for test #14 test_concurrent_txn ========= 2020-09-24 22:07:40,628 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= CollectionBase setup started for test #14 test_concurrent_txn ========= 2020-09-24 22:07:42,947 | test | INFO | pool-15-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-----------------+--------------+ | Nodes | Services | Status | +----------------+-----------------+--------------+ | 172.23.105.229 | index, kv, n1ql | Cluster node | | 172.23.105.230 | None | <--- IN --- | +----------------+-----------------+--------------+ 2020-09-24 22:07:42,957 | test | INFO | pool-15-thread-3 | [task:check:320] Rebalance - status: running, progress: 0 2020-09-24 22:07:52,990 | test | INFO | pool-15-thread-3 | [task:check:320] Rebalance - status: none, progress: 100 2020-09-24 22:07:53,003 | test | INFO | pool-15-thread-3 | [task:check:377] Rebalance completed with progress: 100% in 10.0550000668 sec 2020-09-24 22:07:53,147 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1766] Creating required buckets from template 2020-09-24 22:07:53,375 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection Zz creation 2020-09-24 22:07:55,414 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection wLxjBMKxnYBHcBICWveDhIyi creation 2020-09-24 22:07:57,447 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection DEgKKYboAzYDdRafUbqTjBL creation 2020-09-24 22:07:59,480 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection bvOxUnnoVnOsHIz creation 2020-09-24 22:08:01,513 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection Y creation 2020-09-24 22:08:03,548 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection cSppezyFzQzWlvEFQXfePXYAEV creation 2020-09-24 22:08:05,582 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection JNlTMRbFDAdwsCcAkwQoxKBuHWA creation 2020-09-24 22:08:07,615 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection ptwrxWQQ creation 2020-09-24 22:08:09,644 | test | INFO | Thread-2229 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection HUJdPVCtsqs creation 2020-09-24 22:08:11,717 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection AqigtglYoNmwzbcgfiFVtQ creation 2020-09-24 22:08:13,746 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection POxzNggxLtAxNqAuo creation 2020-09-24 22:08:15,775 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection nvDdtRHDfqsJHJqpySwSCUgQmWnIM creation 2020-09-24 22:08:17,802 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection krpnlB creation 2020-09-24 22:08:19,835 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection ZodcTMAyZSYkDCZWurcrTe creation 2020-09-24 22:08:21,868 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection TWmkqfVsBR creation 2020-09-24 22:08:23,904 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection AbIqwRhekPC creation 2020-09-24 22:08:25,934 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection RzMwVCTLUOy creation 2020-09-24 22:08:27,970 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection FBTHuPAwespJKuYXmWgL creation 2020-09-24 22:08:30,003 | test | INFO | Thread-2239 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection yv creation 2020-09-24 22:08:32,078 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection FNYcuaWhLouqKpc creation 2020-09-24 22:08:34,115 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection iosKFsGSgQpqsTBr creation 2020-09-24 22:08:36,147 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection MGtxakXnerRMATHfKbTAomGKkRbo creation 2020-09-24 22:08:38,188 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection p creation 2020-09-24 22:08:40,223 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection DAXmHzzfTuvfzHomYnCZ creation 2020-09-24 22:08:42,256 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection ywTBBq creation 2020-09-24 22:08:44,290 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection qi creation 2020-09-24 22:08:46,322 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection AqJCeDEBWDviXnAxtBU creation 2020-09-24 22:08:48,355 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection KRuPbDXqtdNBekJORugzYKdkynQiWq creation 2020-09-24 22:08:50,394 | test | INFO | Thread-2250 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection pzH creation 2020-09-24 22:08:52,463 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection wQXtZFGndXW creation 2020-09-24 22:08:54,500 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection MxuWEJ creation 2020-09-24 22:08:56,535 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection CSsEPtFWkU creation 2020-09-24 22:08:58,569 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection pW creation 2020-09-24 22:09:00,602 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection myghPMBeVDF creation 2020-09-24 22:09:02,637 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection hsRFYLmUdSoXHHzKjmqetpht creation 2020-09-24 22:09:04,671 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection BSQXQatMMbUcXsckYjky creation 2020-09-24 22:09:06,703 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection MlSxvIGWGbxYyeoqwqeLuFWWHAkfy creation 2020-09-24 22:09:08,737 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection uGAzSJDhMitgGqNuBILoYEMhxyHc creation 2020-09-24 22:09:10,769 | test | INFO | Thread-2261 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection beTLuRCaHAbSFX creation 2020-09-24 22:09:12,842 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection H creation 2020-09-24 22:09:14,875 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection jTKNuLiGyMLgKdvlaL creation 2020-09-24 22:09:16,911 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection tgQouimcoHEylRJBlYgcILGAP creation 2020-09-24 22:09:18,950 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection olqKvbZ creation 2020-09-24 22:09:20,983 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection ueFmhTWtzUFqrkQbNgrBOA creation 2020-09-24 22:09:23,019 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection tUDejuMKHG creation 2020-09-24 22:09:25,053 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection eWaFPfMtAMpgQcRACOx creation 2020-09-24 22:09:27,092 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection x creation 2020-09-24 22:09:29,127 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection cSDMXclvtbebJjQkfqiJZogQn creation 2020-09-24 22:09:31,161 | test | INFO | Thread-2272 | [common_lib:sleep:22] Sleep 2 seconds. Reason: MB-38073: Wait after collection Z creation 2020-09-24 22:09:33,163 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2227] Waiting for all collections to be created 2020-09-24 22:09:33,210 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | pxewcBcl | couchbase | 1 | none | 0 | 0 | 5322571776 | 69649264 | 88428457 | +----------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2020-09-24 22:09:33,210 | test | INFO | MainThread | [collections_base:collection_setup:123] Creating required SDK clients for client_pool 2020-09-24 22:09:41,213 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: MB-38497 2020-09-24 22:11:41,214 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4544] Performing scope/collection specific operations 2020-09-24 22:11:52,157 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+-----------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+-----------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.105.229 | index, kv, n1ql | 56.7088607595 | 4201840640 | 3297996800 | 85983232 / 3758092288 | 20901 / 18639 | 7.0.0-3167-enterprise | | 172.23.105.230 | kv | 55.076142132 | 4201844736 | 3353686016 | 17825792 / 3758092288 | 20908 / 20901 | 7.0.0-3167-enterprise | +----------------+-----------------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2020-09-24 22:11:53,177 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4354] Validating collection stats and item counts 2020-09-24 22:11:58,220 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | pxewcBcl | couchbase | 1 | none | 0 | 100000 | 5322571776 | 213118176 | 207449116 | +----------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2020-09-24 22:11:58,220 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= CollectionBase setup complete for test #14 test_concurrent_txn ========= index query is CREATE PRIMARY INDEX ix1a on default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p USING GSI index query is CREATE PRIMARY INDEX ix1a on default:pxewcBcl.Mzarcx.wQXtZFGndXW USING GSI index query is CREATE PRIMARY INDEX ix1a on default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX USING GSI 2020-09-24 22:12:15,385 | test | INFO | Thread-2397 | [N1qlBase:thread_txn:667] values are ["pxewcBcl.Mzarcx.beTLuRCaHAbSFX:UPDATE: join_mo < 3 OR join_mo > 11:period='winter'", 'SAVEPOINT:a2', "pxewcBcl.Mzarcx.wQXtZFGndXW:DELETE: 'Development' in skills:isdevloper=True", 'SAVEPOINT:a0', "pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p:UPDATE: join_day > 10 AND join_mo > 10:join_month='October'", 'SAVEPOINT:a0', 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a0', 'SAVEPOINT:a0'] True False False 2020-09-24 22:12:15,385 | test | INFO | Thread-2398 | [N1qlBase:thread_txn:667] values are ['SAVEPOINT:a0', "pxewcBcl.Mzarcx.beTLuRCaHAbSFX:DELETE: join_mo=9:join_month='SEPTEMBER'", "pxewcBcl.Mzarcx.beTLuRCaHAbSFX:UPDATE: job_title='Engineer' and 'Testing' in skills:QE=True", 'SAVEPOINT:a2', "pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p:UPDATE: job_title='Support':SE=True", 'SAVEPOINT:a1', 'SAVEPOINT:a1', 'SAVEPOINT:a1', 'SAVEPOINT:a1', 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a0'] True False False 2020-09-24 22:12:15,385 | test | INFO | Thread-2396 | [N1qlBase:thread_txn:667] values are ['SAVEPOINT:a2', 'SAVEPOINT:a0', 'SAVEPOINT:a0', "pxewcBcl.Mzarcx.beTLuRCaHAbSFX:DELETE: job_title='Support':SE=True", 'SAVEPOINT:a2', 'SAVEPOINT:a1', "pxewcBcl.Mzarcx.wQXtZFGndXW:INSERT: job_title='Engineer':Team='Development'", 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a0', "pxewcBcl.Mzarcx.wQXtZFGndXW:INSERT: join_mo =10:join_month='October'", 'SAVEPOINT:a0'] True False False 2020-09-24 22:12:15,385 | test | INFO | Thread-2399 | [N1qlBase:thread_txn:667] values are ['SAVEPOINT:a1', 'SAVEPOINT:a2', 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a0', 'SAVEPOINT:a1', "pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p:UPDATE: join_mo=9:join_month='SEPTEMBER'", 'SAVEPOINT:a2', 'SAVEPOINT:a0', 'SAVEPOINT:a0', "pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p:INSERT: job_title='Support':SE=True", "pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p:DELETE: join_mo < 3 OR join_mo > 11:period='winter'"] True False False 2020-09-24 22:12:15,385 | test | INFO | Thread-2400 | [N1qlBase:thread_txn:667] values are ['SAVEPOINT:a1', "pxewcBcl.Mzarcx.wQXtZFGndXW:INSERT: job_title='Support':SE=True", "pxewcBcl.Mzarcx.beTLuRCaHAbSFX:UPDATE: 'Development' in skills:isdevloper=True", 'SAVEPOINT:a2', 'SAVEPOINT:a1', 'SAVEPOINT:a0', 'SAVEPOINT:a0', 'SAVEPOINT:a1', 'SAVEPOINT:a0', 'SAVEPOINT:a2', "pxewcBcl.Mzarcx.wQXtZFGndXW:INSERT: 'Development' in skills:isdevloper=True", 'SAVEPOINT:a2'] False False False 2020-09-24 22:12:19,095 | test | INFO | Thread-2400 | [N1qlBase:run_cbq_query:199] txn failed 2020-09-24 22:12:19,095 | test | INFO | Thread-2400 | [N1qlBase:run_cbq_query:208] txn failed with error {u'signature': {u'*': u'*'}, u'requestID': u'66c5963c-063f-4389-a159-0a85f698c2d8', u'metrics': {u'resultCount': 0, u'executionTime': u'508.582043ms', u'serviceLoad': 7, u'resultSize': 0, u'errorCount': 1, u'elapsedTime': u'508.646938ms'}, u'results': [], u'errors': [{u'msg': u'Duplicate Key: employee-1019', u'code': 17012}], u'status': u'errors'} 2020-09-24 22:12:19,118 | test | INFO | Thread-2400 | [N1qlBase:run_cbq_query:199] txn failed 2020-09-24 22:12:19,118 | test | INFO | Thread-2400 | [N1qlBase:full_execute_query:552] txn failed with unexpected errors {u'signature': u'json', u'requestID': u'5e76137e-faa1-43bd-ba8d-ee3f387af9e4', u'metrics': {u'resultCount': 0, u'executionTime': u'520.298\xb5s', u'serviceLoad': 7, u'resultSize': 0, u'errorCount': 1, u'elapsedTime': u'594.082\xb5s'}, u'results': [], u'errors': [{u'msg': u'Panic: runtime error: index out of range [-1]', u'code': 5001}], u'status': u'aborted'} 2020-09-24 22:12:19,118 | test | INFO | Thread-2400 | [N1qlBase:thread_txn:671] queries executed in txn are {u'1103ecbd-f510-478d-afc6-c2fcf4e6adb8': ['SAVEPOINT a1', "INSERT INTO default:pxewcBcl.Mzarcx.wQXtZFGndXW (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:pxewcBcl.Mzarcx.wQXtZFGndXW t WHERE job_title='Support' LIMIT 10 RETURNING *", ["UPDATE default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX SET isdevloper=True WHERE 'Development' in skills LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a0', 'SAVEPOINT a0', 'SAVEPOINT a1', 'SAVEPOINT a0', 'SAVEPOINT a2', "INSERT INTO default:pxewcBcl.Mzarcx.wQXtZFGndXW (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:pxewcBcl.Mzarcx.wQXtZFGndXW t WHERE 'Development' in skills LIMIT 10 RETURNING *"]} 2020-09-24 22:12:20,032 | test | INFO | Thread-2399 | [N1qlBase:thread_txn:671] queries executed in txn are {u'cac4553e-4efb-4e86-a4a1-139ea70474c2': ['SAVEPOINT a1', 'SAVEPOINT a2', 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a0', 'SAVEPOINT a1', ["UPDATE default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p SET join_month='SEPTEMBER' WHERE join_mo=9 LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a2', 'SAVEPOINT a0', 'SAVEPOINT a0', "INSERT INTO default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p t WHERE job_title='Support' LIMIT 10 RETURNING *", 'DELETE FROM default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p WHERE join_mo < 3 OR join_mo > 11 LIMIT 200 RETURNING meta().id', 'COMMIT WORK']} 2020-09-24 22:12:20,053 | test | INFO | Thread-2396 | [N1qlBase:run_cbq_query:199] txn failed 2020-09-24 22:12:20,053 | test | INFO | Thread-2396 | [N1qlBase:run_cbq_query:208] txn failed with error {u'signature': u'json', u'requestID': u'553c55c8-7847-42d5-991b-aecd535a4235', u'metrics': {u'resultCount': 0, u'executionTime': u'129.647888ms', u'serviceLoad': 4, u'resultSize': 0, u'errorCount': 1, u'elapsedTime': u'129.716233ms'}, u'results': [], u'errors': [{u'msg': u'Commit Transaction statement error: transaction operation failed | document exists | {"status_code":2,"bucket":"pxewcBcl","scope":"Mzarcx","collection":"beTLuRCaHAbSFX","collection_id":46,"error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":837,"last_dispatched_to":"172.23.105.229:11210","last_dispatched_from":"172.23.105.229:55284","last_connection_id":"4015b7f8b466846e/c4ee6093acac43ce"}', u'code': 17007}], u'status': u'errors'} 2020-09-24 22:12:20,053 | test | INFO | Thread-2396 | [N1qlBase:thread_txn:671] queries executed in txn are {u'9e4860f6-78b2-4f49-adbd-e84c4983c464': ['SAVEPOINT a2', 'SAVEPOINT a0', 'SAVEPOINT a0', "DELETE FROM default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX WHERE job_title='Support' LIMIT 200 RETURNING meta().id", 'SAVEPOINT a2', 'SAVEPOINT a1', "INSERT INTO default:pxewcBcl.Mzarcx.wQXtZFGndXW (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:pxewcBcl.Mzarcx.wQXtZFGndXW t WHERE job_title='Engineer' LIMIT 10 RETURNING *", 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a0', 'INSERT INTO default:pxewcBcl.Mzarcx.wQXtZFGndXW (KEY k1, value t) SELECT DISTINCT t.name AS k1,t FROM default:pxewcBcl.Mzarcx.wQXtZFGndXW t WHERE join_mo =10 LIMIT 10 RETURNING *', 'SAVEPOINT a0', 'COMMIT WORK']} 2020-09-24 22:12:20,088 | test | INFO | Thread-2398 | [N1qlBase:run_cbq_query:199] txn failed 2020-09-24 22:12:20,088 | test | INFO | Thread-2398 | [N1qlBase:run_cbq_query:208] txn failed with error {u'signature': u'json', u'requestID': u'e7750a40-16bd-47c8-84c0-2b791e85b591', u'metrics': {u'resultCount': 0, u'executionTime': u'321.498719ms', u'serviceLoad': 3, u'resultSize': 0, u'errorCount': 1, u'elapsedTime': u'321.573076ms'}, u'results': [], u'errors': [{u'msg': u'Commit Transaction statement error: transaction operation failed | document exists | {"status_code":2,"bucket":"pxewcBcl","scope":"Mzarcx","collection":"beTLuRCaHAbSFX","collection_id":46,"error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":569,"last_dispatched_to":"172.23.105.229:11210","last_dispatched_from":"172.23.105.229:55290","last_connection_id":"4015b7f8b466846e/eb6d62cc87c78f04"}', u'code': 17007}], u'status': u'errors'} 2020-09-24 22:12:20,088 | test | INFO | Thread-2398 | [N1qlBase:thread_txn:671] queries executed in txn are {u'396487f0-6eb0-42c8-b767-b004ebad255b': ['SAVEPOINT a0', 'DELETE FROM default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX WHERE join_mo=9 LIMIT 200 RETURNING meta().id', ["UPDATE default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX SET QE=True WHERE job_title='Engineer' and 'Testing' in skills LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a2', ["UPDATE default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p SET SE=True WHERE job_title='Support' LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a1', 'SAVEPOINT a1', 'SAVEPOINT a1', 'SAVEPOINT a1', 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a0', 'COMMIT WORK']} 2020-09-24 22:12:20,926 | test | INFO | Thread-2397 | [N1qlBase:thread_txn:671] queries executed in txn are {u'bc6c4888-a2cf-4981-b8a1-b88d11f9200e': [["UPDATE default:pxewcBcl.Mzarcx.beTLuRCaHAbSFX SET period='winter' WHERE join_mo < 3 OR join_mo > 11 LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a2', "DELETE FROM default:pxewcBcl.Mzarcx.wQXtZFGndXW WHERE 'Development' in skills LIMIT 200 RETURNING meta().id", 'SAVEPOINT a0', ["UPDATE default:pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p SET join_month='October' WHERE join_day > 10 AND join_mo > 10 LIMIT 100 RETURNING meta().id"], 'SAVEPOINT a0', 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a2', 'SAVEPOINT a1', 'SAVEPOINT a0', 'SAVEPOINT a0', 'COMMIT WORK']} 2020-09-24 22:12:20,927 | test | INFO | MainThread | [N1qlBase:get_stmt_for_threads:712] One of the thread has failed result is [[{'a0:7': {}, 'a2:6': {'pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p': {'DELETE': [], 'INSERT': {}, 'UPDATE': {"join_month='SEPTEMBER'": ['test_collections-0', 'test_collections-1', 'test_collections-10', 'test_collections-100', 'test_collections-1000', 'test_collections-1001', 'test_collections-1002', 'test_collections-1003', 'test_collections-1004', 'test_collections-1005', 'test_collections-1006', 'test_collections-1007', 'test_collections-1008', 'test_collections-1009', 'test_collections-101', 'test_collections-1010', 'test_collections-1011', 'test_collections-1012', 'test_collections-1013', 'test_collections-1014', 'test_collections-1015', 'test_collections-1016', 'test_collections-1017', 'test_collections-1018', 'test_collections-1019', 'test_collections-102', 'test_collections-1020', 'test_collections-1021', 'test_collections-1022', 'test_collections-1023', 'test_collections-103', 'test_collections-104', 'test_collections-105', 'test_collections-106', 'test_collections-107', 'test_collections-108', 'test_collections-109', 'test_collections-11', 'test_collections-110', 'test_collections-111', 'test_collections-112', 'test_collections-113', 'test_collections-114', 'test_collections-115', 'test_collections-116', 'test_collections-117', 'test_collections-118', 'test_collections-119', 'test_collections-12', 'test_collections-120', 'test_collections-121', 'test_collections-122', 'test_collections-123', 'test_collections-124', 'test_collections-125', 'test_collections-126', 'test_collections-127', 'test_collections-128', 'test_collections-129', 'test_collections-13', 'test_collections-130', 'test_collections-131', 'test_collections-132', 'test_collections-133', 'test_collections-134', 'test_collections-135', 'test_collections-136', 'test_collections-137', 'test_collections-138', 'test_collections-139', 'test_collections-14', 'test_collections-140', 'test_collections-1408', 'test_collections-1409', 'test_collections-141', 'test_collections-1410', 'test_collections-1411', 'test_collections-1412', 'test_collections-1413', 'test_collections-1414', 'test_collections-1415', 'test_collections-1416', 'test_collections-1417', 'test_collections-1418', 'test_collections-1419', 'test_collections-142', 'test_collections-1420', 'test_collections-1421', 'test_collections-1422', 'test_collections-1423', 'test_collections-1424', 'test_collections-1425', 'test_collections-1426', 'test_collections-1427', 'test_collections-1428', 'test_collections-1429', 'test_collections-143', 'test_collections-1430', 'test_collections-1431', 'test_collections-1432']}}}, 'a0:8': {}, 'last': {'pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p': {'DELETE': [], 'INSERT': {u'employee-1023': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 15, u'skills': [u'Testing', u'C++'], u'join_yr': 2012, u'name': u'employee-1023', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1023-mail@couchbase.com']}, u'employee-118': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 12, u'skills': [u'Development', u'C++'], u'join_yr': 2011, u'name': u'employee-118', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'118-mail@couchbase.com']}, u'employee-1418': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 2, u'skills': [u'Testing', u'Python'], u'join_yr': 2015, u'name': u'employee-1418', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1418-mail@couchbase.com']}, u'employee-139': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 18, u'skills': [u'Development', u'Testing'], u'join_yr': 2015, u'name': u'employee-139', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'139-mail@couchbase.com']}, u'employee-1428': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 22, u'skills': [u'Python', u'Testing'], u'join_yr': 2015, u'name': u'employee-1428', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1428-mail@couchbase.com']}, u'employee-109': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 23, u'skills': [u'Development', u'C++'], u'join_yr': 2015, u'name': u'employee-109', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'109-mail@couchbase.com']}, u'employee-1414': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 22, u'skills': [u'Java', u'Development'], u'join_yr': 2010, u'name': u'employee-1414', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1414-mail@couchbase.com']}, u'employee-1424': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 15, u'skills': [u'Testing', u'Python'], u'join_yr': 2010, u'name': u'employee-1424', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1424-mail@couchbase.com']}, u'employee-135': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 10, u'skills': [u'Java', u'Python'], u'join_yr': 2010, u'name': u'employee-135', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'135-mail@couchbase.com']}, u'employee-1427': {u'join_month': u'SEPTEMBER', u'test_rate': [1, 2, 3, 4, 5, 6, 7, 8, 9], u'join_mo': 9, u'join_day': 27, u'skills': [u'Java', u'C++'], u'join_yr': 2016, u'name': u'employee-1427', u'job_title': u'Support', u'vms': [[{u'memory': 9, u'os': u'ubuntu', u'name': u'vm_9', u'RAM': 9}, {u'memory': 9, u'os': u'windows', u'name': u'vm_10', u'RAM': 9}]], u'email': [u'1427-mail@couchbase.com']}}, 'UPDATE': {}}}, 'a2:2': {}, 'a1:3': {}, 'a1:5': {}, 'a0': {}, 'a1': {}, 'a2': {}}, ['a1', 'a2', 'a2:2', 'a1:3', 'a0', 'a1:5', 'a2:6', 'a0:7', 'a0:8', 'last']], [{}, []], [{}, []], [{'a2:5': {}, 'a1:6': {}, 'a0:7': {}, 'a0:8': {}, 'last': {}, 'a2:3': {}, 'a0:2': {'pxewcBcl.eakXDzXwrQLPMlmZXHkHctr.p': {'DELETE': [], 'INSERT': {}, 'UPDATE': {"join_month='October'": []}}}, 'a0': {'pxewcBcl.Mzarcx.wQXtZFGndXW': {'DELETE': ['test_collections-1', 'test_collections-10', 'test_collections-1001', 'test_collections-1004', 'test_collections-1009', 'test_collections-101', 'test_collections-1010', 'test_collections-1013', 'test_collections-1016', 'test_collections-1017', 'test_collections-1018', 'test_collections-1019', 'test_collections-102', 'test_collections-1025', 'test_collections-1026', 'test_collections-1028', 'test_collections-1029', 'test_collections-1030', 'test_collections-1031', 'test_collections-1032', 'test_collections-1034', 'test_collections-1039', 'test_collections-1044', 'test_collections-1045', 'test_collections-1047', 'test_collections-1048', 'test_collections-1050', 'test_collections-1051', 'test_collections-1055', 'test_collections-1057', 'test_collections-1059', 'test_collections-1061', 'test_collections-1062', 'test_collections-1063', 'test_collections-1065', 'test_collections-1066', 'test_collections-1068', 'test_collections-1073', 'test_collections-1074', 'test_collections-1075', 'test_collections-1077', 'test_collections-1078', 'test_collections-1080', 'test_collections-1081', 'test_collections-1082', 'test_collections-1084', 'test_collections-1088', 'test_collections-109', 'test_collections-1092', 'test_collections-1099', 'test_collections-11', 'test_collections-110', 'test_collections-1100', 'test_collections-1101', 'test_collections-1102', 'test_collections-1104', 'test_collections-1105', 'test_collections-1107', 'test_collections-1108', 'test_collections-111', 'test_collections-1111', 'test_collections-1116', 'test_collections-1117', 'test_collections-1122', 'test_collections-1123', 'test_collections-1126', 'test_collections-1131', 'test_collections-1134', 'test_collections-1135', 'test_collections-1137', 'test_collections-1139', 'test_collections-1140', 'test_collections-1147', 'test_collections-115', 'test_collections-1152', 'test_collections-1153', 'test_collections-1155', 'test_collections-1156', 'test_collections-116', 'test_collections-1163', 'test_collections-1168', 'test_collections-1172', 'test_collections-1175', 'test_collections-1179', 'test_collections-118', 'test_collections-1180', 'test_collections-1181', 'test_collections-1182', 'test_collections-1189', 'test_collections-1190', 'test_collections-1192', 'test_collections-1197', 'test_collections-1198', 'test_collections-1201', 'test_collections-1202', 'test_collections-1203', 'test_collections-1206', 'test_collections-1211', 'test_collections-1214', 'test_collections-1215', 'test_collections-1216', 'test_collections-1221', 'test_collections-1224', 'test_collections-1225', 'test_collections-1226', 'test_collections-1229', 'test_collections-123', 'test_collections-1235', 'test_collections-1237', 'test_collections-1238', 'test_collections-1241', 'test_collections-1244', 'test_collections-1245', 'test_collections-1248', 'test_collections-1255', 'test_collections-1256', 'test_collections-1259', 'test_collections-126', 'test_collections-1262', 'test_collections-1263', 'test_collections-1264', 'test_collections-1271', 'test_collections-1272', 'test_collections-1274', 'test_collections-1277', 'test_collections-1278', 'test_collections-1279', 'test_collections-1280', 'test_collections-1287', 'test_collections-1288', 'test_collections-1290', 'test_collections-1293', 'test_collections-1294', 'test_collections-1295', 'test_collections-1296', 'test_collections-1301', 'test_collections-1304', 'test_collections-1305', 'test_collections-1306', 'test_collections-1309', 'test_collections-131', 'test_collections-1316', 'test_collections-1319', 'test_collections-132', 'test_collections-1322', 'test_collections-1323', 'test_collections-1327', 'test_collections-1328', 'test_collections-1335', 'test_collections-1336', 'test_collections-1339', 'test_collections-134', 'test_collections-1342', 'test_collections-1343', 'test_collections-1345', 'test_collections-1346', 'test_collections-1349', 'test_collections-1352', 'test_collections-1353', 'test_collections-1354', 'test_collections-1361', 'test_collections-1362', 'test_collections-1364', 'test_collections-1367', 'test_collections-1368', 'test_collections-1370', 'test_collections-1375', 'test_collections-1376', 'test_collections-1377', 'test_collections-1379', 'test_collections-1380', 'test_collections-1382', 'test_collections-1383', 'test_collections-1385', 'test_collections-1386', 'test_collections-1388', 'test_collections-1389', 'test_collections-139', 'test_collections-1393', 'test_collections-1397', 'test_collections-1398', 'test_collections-1399', 'test_collections-1401', 'test_collections-1402', 'test_collections-1404', 'test_collections-1409', 'test_collections-1413', 'test_collections-1414', 'test_collections-1417', 'test_collections-142', 'test_collections-1420', 'test_collections-1425', 'test_collections-1426', 'test_collections-1429', 'test_collections-1432', 'test_collections-1433', 'test_collections-1434', 'test_collections-144', 'test_collections-1444', 'test_collections-1447'], 'INSERT': {}, 'UPDATE': {}}}, 'a1': {}, 'a2': {'pxewcBcl.Mzarcx.beTLuRCaHAbSFX': {'DELETE': [], 'INSERT': {}, 'UPDATE': {"period='winter'": []}}}}, ['a2', 'a0', 'a0:2', 'a2:3', 'a1', 'a2:5', 'a1:6', 'a0:7', 'a0:8', 'last']]] delete key count is 0 2020-09-24 22:12:21,236 | test | INFO | MainThread | [N1qlBase:validate_keys:618] len of expected keys, actual keys, deleted keys 2010, 2010, 0 delete key count is 200 2020-09-24 22:12:21,552 | test | INFO | MainThread | [N1qlBase:validate_keys:618] len of expected keys, actual keys, deleted keys 1800, 2000, 200 delete key count is 0 2020-09-24 22:12:21,864 | test | INFO | MainThread | [N1qlBase:validate_keys:618] len of expected keys, actual keys, deleted keys 2000, 2000, 0 FAIL 2020-09-24 22:12:22,441 | test | INFO | MainThread | [collections_base:tearDown:83] Bucket: pxewcBcl, Resident ratio(DGM): 100% 2020-09-24 22:12:29,292 | test | INFO | MainThread | [basetestcase:check_coredump_exist:694] 172.23.105.229 : SSH Successful 2020-09-24 22:12:29,293 | test | INFO | MainThread | [basetestcase:check_coredump_exist:695] 172.23.105.229 : Looking for crash dump files 2020-09-24 22:12:29,309 | test | INFO | MainThread | [basetestcase:check_coredump_exist:714] 172.23.105.229 : No crash files found 2020-09-24 22:12:29,309 | test | INFO | MainThread | [basetestcase:check_coredump_exist:716] 172.23.105.229 : Looking for CRITICAL messages in log 2020-09-24 22:12:30,611 | test | INFO | MainThread | [basetestcase:check_coredump_exist:694] 172.23.105.230 : SSH Successful 2020-09-24 22:12:30,611 | test | INFO | MainThread | [basetestcase:check_coredump_exist:695] 172.23.105.230 : Looking for crash dump files 2020-09-24 22:12:30,627 | test | INFO | MainThread | [basetestcase:check_coredump_exist:714] 172.23.105.230 : No crash files found 2020-09-24 22:12:30,628 | test | INFO | MainThread | [basetestcase:check_coredump_exist:716] 172.23.105.230 : Looking for CRITICAL messages in log 2020-09-24 22:12:31,336 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:595] Running cbcollect on node 172.23.105.229 2020-09-24 22:12:31,362 | test | INFO | MainThread | [rest_client:_http_request:727] {'date': 'Fri, 25 Sep 2020 05:12:31 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2020-09-24 22:12:31,362 | test | INFO | MainThread | [rest_client:_http_request:728] [] 2020-09-24 22:12:41,371 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:599] 172.23.105.229 - cbcollect status: True 2020-09-24 22:12:41,371 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:602] Polling active_tasks to check cbcollect status 2020-09-24 22:15:31,670 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:619] Copying cbcollect ZIP file to Client 2020-09-24 22:15:32,693 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:628] 172.23.105.229 node cb collect zip coped on client : True 2020-09-24 22:15:32,720 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:595] Running cbcollect on node 172.23.105.230 2020-09-24 22:15:32,743 | test | INFO | MainThread | [rest_client:_http_request:727] {'date': 'Fri, 25 Sep 2020 05:15:32 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2020-09-24 22:15:32,744 | test | INFO | MainThread | [rest_client:_http_request:728] [] 2020-09-24 22:15:42,752 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:599] 172.23.105.230 - cbcollect status: True 2020-09-24 22:15:42,752 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:602] Polling active_tasks to check cbcollect status 2020-09-24 22:17:42,973 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:619] Copying cbcollect ZIP file to Client 2020-09-24 22:17:43,786 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:628] 172.23.105.230 node cb collect zip coped on client : True 2020-09-24 22:17:43,806 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:489] Skipping get_trace !! 2020-09-24 22:17:56,244 | test | INFO | MainThread | [rest_client:monitorRebalance:1432] Rebalance done. Taken 11.0599999428 seconds to complete 2020-09-24 22:17:56,253 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ====================================================================== FAIL: test_concurrent_txn (N1qlTransaction.basic_ops.BasicOps) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/N1qlTransaction/basic_ops.py", line 54, in test_concurrent_txn self.fail("One of the thread failed with unexpected errors") AssertionError: One of the thread failed with unexpected errors ---------------------------------------------------------------------- Ran 1 test in 650.396s FAILED (failures=1) During the test, Remote Connections: 372, Disconnections: 330 SDK Connections: 420, Disconnections: 420