jython -J-cp /Users/ashwin/gitRepos/couchbaselabs/TAF/build/classes/java/main:/Users/ashwin/gitRepos/couchbaselabs/TAF/build/resources/main:/Users/ashwin/gitRepos/couchbaselabs/TAF/jars/java-client-3.1.5-SNAPSHOT.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/Users/ashwin/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i node.ini -c debug.conf -p threads_to_use=10,rerun=False,get-cbcollect-info=False,skip_collections_cleanup=True,skip_cleanup=True -m rest Filename: debug.conf Prefix: epengine.collection_crud_negative.CollectionDurabilityTests Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'num_nodes': 2, 'rerun': 'False', 'skip_cleanup': 'True', 'skip_collections_cleanup': 'True', 'spec': 'debug', 'threads_to_use': '10'} Logs will be stored at /Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-21-Apr-17_00-30-39/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini threads_to_use=10,rerun=False,get-cbcollect-info=False,skip_collections_cleanup=True,skip_cleanup=True -t epengine.collection_crud_negative.CollectionDurabilityTests.test_sync_write_in_progress,nodes_init=2,replicas=1,num_items=100,sdk_timeout=60,doc_ops=create;create,bucket_spec=single_bucket.three_scopes_two_collections' Test Input params: {'cluster_name': 'node', 'ini': 'node.ini', 'get-cbcollect-info': 'False', 'replicas': '1', 'conf_file': 'debug.conf', 'threads_to_use': '10', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 2, 'logs_folder': '/Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-21-Apr-17_00-30-39/test_1', 'skip_cleanup': 'True', 'nodes_init': '2', 'doc_ops': 'create;create', 'num_items': '100', 'skip_collections_cleanup': 'True', 'sdk_timeout': '60', 'bucket_spec': 'single_bucket.three_scopes_two_collections', 'case_number': 1} test_sync_write_in_progress (epengine.collection_crud_negative.CollectionDurabilityTests) ... 2021-04-17 00:30:40,980 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionDurabilityTests setup started for test #1 test_sync_write_in_progress ========= 2021-04-17 00:30:41,296 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_sync_write_in_progress ========= 2021-04-17 00:30:41,739 | test | WARNING | MainThread | [basetestcase:setUp:303] Cluster operation in setup will be skipped 2021-04-17 00:30:41,740 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_sync_write_in_progress ========= 2021-04-17 00:30:41,740 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_sync_write_in_progress ========= 2021-04-17 00:30:44,335 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 10.112.210.101 | kv | 7.0.0-4907-enterprise | 1.42566191446 | Cluster node | | 10.112.210.102 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-04-17 00:30:44,358 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-17 00:30:54,803 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-17 00:30:54,822 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 10.4869999886 sec 2021-04-17 00:30:55,167 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-17 00:30:56,142 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-04-17 00:30:56,311 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | iYsSfE39w--55-218000 | couchbase | 1 | none | 0 | 0 | 1310720000 | 0 | 0 | +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-17 00:30:56,319 | test | INFO | MainThread | [collections_base:collection_setup:162] Creating required SDK clients for client_pool 2021-04-17 00:30:59,927 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-17 00:30:59,944 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-17 00:31:01,627 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 10.112.210.102 | kv | 1.21703853955 | 1930829824 | 1556295680 | 0 / 1107292160 | 0 / 0 | 7.0.0-4907-enterprise | | 10.112.210.101 | kv | 5.01022494888 | 1042685952 | 582742016 | 31387648 / 1107292160 | 0 / 0 | 7.0.0-4907-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-04-17 00:31:09,441 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-17 00:31:12,292 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | iYsSfE39w--55-218000 | couchbase | 1 | none | 0 | 600 | 1310720000 | 59987328 | 13765992 | +----------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-17 00:31:12,293 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_sync_write_in_progress ========= 2021-04-17 00:31:12,295 | test | INFO | MainThread | [collection_crud_negative:_CollectionDurabilityTests__get_d_level_and_error_to_simulate:54] Testing with durability_level=MAJORITY, simulate_error=stop_memcached 2021-04-17 00:31:13,147 | test | INFO | MainThread | [cb_error:create:39] Simulating 'stop_memcached' in 10.112.210.102 2021-04-17 00:31:13,184 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 3 seconds. Reason: Wait for error simulation to take effect 2021-04-17 00:31:16,187 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-17 00:31:16,194 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-17 00:31:16,286 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait for doc ops to reach server 2021-04-17 00:31:27,334 | test | ERROR | MainThread | [basetestcase:log_failure:626] Invalid retry reason for test_collections-101: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x474cf0bc00000007","idempotent":false,"lastChannelId":"474CF0BC00000007/000000005780EDEE","lastDispatchedFrom":"10.112.210.1:49413","lastDispatchedTo":"10.112.210.101:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1147,"requestType":"InsertRequest","retried":0,"service":{"bucket":"iYsSfE39w--55-218000","collection":"QgFROA-KggG1-55-219000","documentId":"test_collections-101","opaque":"0x1384","scope":"XXSx-55-219000","syncDurability":"MAJORITY","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":30,"totalMicros":160}} 2021-04-17 00:31:29,342 | test | ERROR | MainThread | [basetestcase:log_failure:626] Invalid retry reason for test_collections-101: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x474cf0bc00000007","idempotent":false,"lastChannelId":"474CF0BC00000007/000000005780EDEE","lastDispatchedFrom":"10.112.210.1:49413","lastDispatchedTo":"10.112.210.101:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1188,"requestType":"InsertRequest","retried":0,"service":{"bucket":"iYsSfE39w--55-218000","collection":"3PMaynq-55-219000","documentId":"test_collections-101","opaque":"0x13ad","scope":"ZICLaiQfs-55-219000","syncDurability":"MAJORITY","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":38,"totalMicros":147}} 2021-04-17 00:31:31,351 | test | INFO | MainThread | [cb_error:revert:68] Reverting 'stop_memcached' in 10.112.210.102 2021-04-17 00:31:32,026 | test | INFO | pool-2-thread-9 | [table_view:display:72] Ops trend for bucket 'iYsSfE39w--55-218000' +--------+-------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------+--------+ | 30.012 | ...................................................................................................*X | 30.012 | | 0.000 | *******...............**X | 4.600 | +--------+-------------------------------------------------------------------------------------------------------+--------+ FAIL 2021-04-17 00:31:32,338 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: iYsSfE39w--55-218000, Active Resident ratio(DGM): 100% 2021-04-17 00:31:32,338 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: iYsSfE39w--55-218000, Replica Resident ratio(DGM): 100% 2021-04-17 00:31:33,644 | test | WARNING | MainThread | [basetestcase:tearDownEverything:466] CLEANUP WAS SKIPPED ====================================================================== FAIL: test_sync_write_in_progress (epengine.collection_crud_negative.CollectionDurabilityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/collection_crud_negative.py", line 501, in test_sync_write_in_progress self.validate_test_failure() File "pytests/basetestcase.py", line 633, in validate_test_failure self.fail(self.test_failure) AssertionError: Invalid retry reason for test_collections-101: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x474cf0bc00000007","idempotent":false,"lastChannelId":"474CF0BC00000007/000000005780EDEE","lastDispatchedFrom":"10.112.210.1:49413","lastDispatchedTo":"10.112.210.101:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1147,"requestType":"InsertRequest","retried":0,"service":{"bucket":"iYsSfE39w--55-218000","collection":"QgFROA-KggG1-55-219000","documentId":"test_collections-101","opaque":"0x1384","scope":"XXSx-55-219000","syncDurability":"MAJORITY","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":30,"totalMicros":160}} ---------------------------------------------------------------------- Ran 1 test in 53.167s FAILED (failures=1) During the test, Remote Connections: 17, Disconnections: 16 SDK Connections: 7, Disconnections: 7 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! summary so far suite epengine.collection_crud_negative.CollectionDurabilityTests , pass 0 , fail 1 failures so far... epengine.collection_crud_negative.CollectionDurabilityTests.test_sync_write_in_progress testrunner logs, diags and results are available under /Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-21-Apr-17_00-30-39/test_1 (base) MacBook-Pro:TAF$