./testrunner -i node.ini threads_to_use=10,rerun=False,get-cbcollect-info=False,skip_collections_cleanup=True,skip_cleanup=True -t epengine.collection_crud_negative.CollectionDurabilityTests.test_durability_abort,num_items=100,replicas=1,nodes_init=2,override_spec_params=replicas;num_items;durability,durability=MAJORITY,collect_pcaps=True Test Input params: {'cluster_name': 'node', 'ini': 'node.ini', 'get-cbcollect-info': 'False', 'replicas': '1', 'durability': 'MAJORITY', 'conf_file': 'debug.conf', 'threads_to_use': '10', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 2, 'logs_folder': '/Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-20-Aug-20_12-55-05/test_1', 'skip_cleanup': 'True', 'nodes_init': '2', 'num_items': '100', 'skip_collections_cleanup': 'True', 'collect_pcaps': 'True', 'case_number': 1, 'override_spec_params': 'replicas;num_items;durability'} WARNING: Failed to import concurrent.futures module 2020-08-20 12:55:20,088 | infra | ERROR | MainThread | [remote_util:log_command_output:3049] Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast 2020-08-20 12:55:34,736 | infra | ERROR | MainThread | [remote_util:log_command_output:3049] Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast 2020-08-20 12:55:36,328 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup started for test #1 test_durability_abort ========= 2020-08-20 12:55:36,908 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup started for test #1 test_durability_abort ========= 2020-08-20 12:55:37,338 | test | WARNING | MainThread | [basetestcase:setUp:303] Cluster operation in setup will be skipped 2020-08-20 12:55:37,341 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= BaseTestCase setup finished for test #1 test_durability_abort ========= 2020-08-20 12:55:37,342 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= CollectionBase setup started for test #1 test_durability_abort ========= 2020-08-20 12:55:39,161 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 10.112.191.101 | kv | Cluster node | | 10.112.191.102 | None | <--- IN --- | +----------------+----------+--------------+ 2020-08-20 12:55:39,197 | test | INFO | pool-2-thread-1 | [task:check:320] Rebalance - status: running, progress: 0 2020-08-20 12:55:49,683 | test | INFO | pool-2-thread-1 | [task:check:320] Rebalance - status: none, progress: 100 2020-08-20 12:55:49,700 | test | INFO | pool-2-thread-1 | [task:check:377] Rebalance completed with progress: 100% in 10.5409998894 sec 2020-08-20 12:55:49,951 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1715] Creating required buckets from template 2020-08-20 12:55:50,062 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2176] Waiting for all collections to be created 2020-08-20 12:55:50,247 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ | YuZxX8l_FTdbPYoGu41XD9IaWn | couchbase | 1 | none | 0 | 0 | 536870912 | 0 | 0 | +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ 2020-08-20 12:55:50,249 | test | INFO | MainThread | [collections_base:collection_setup:122] Creating required SDK clients for client_pool 2020-08-20 12:55:57,483 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: MB-38497 2020-08-20 12:57:57,490 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4491] Performing scope/collection specific operations 2020-08-20 12:57:58,809 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ | 10.112.191.102 | kv | 8.08080808081 | 1930829824 | 1487970304 | 24576 / 1107292160 | 24 / 0 | 7.0.0-2241-enterprise | | 10.112.191.101 | kv | 7.36842105263 | 1930829824 | 1471295488 | 4186112 / 1107292160 | 0 / 0 | 7.0.0-2241-enterprise | +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ 2020-08-20 12:57:59,380 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4301] Validating collection stats and item counts 2020-08-20 12:58:02,884 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ | YuZxX8l_FTdbPYoGu41XD9IaWn | couchbase | 1 | none | 0 | 24 | 536870912 | 50471704 | 19986048 | +----------------------------+-----------+----------+------------+-----+-------+-----------+----------+-----------+ 2020-08-20 12:58:02,885 | test | INFO | MainThread | [basetestcase:log_setup_status:528] ========= CollectionBase setup complete for test #1 test_durability_abort ========= 2020-08-20 12:58:02,887 | test | INFO | MainThread | [collection_crud_negative:test_durability_abort:221] Loading docs such that all sync_writes will be aborted with durability_level 'MAJORITY' 2020-08-20 12:58:03,269 | test | INFO | MainThread | [cb_error:create:37] Simulating 'stop_memcached' in 10.112.191.101 2020-08-20 12:58:03,311 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4491] Performing scope/collection specific operations 2020-08-20 12:58:05,259 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-100 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:05,259 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:05,259 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:05,262 | test | ERROR | pool-2-thread-10 | [task:call:3398] Failed to load 1 docs from 2 to 3 2020-08-20 12:58:07,232 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 10.112.191.102:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:07,233 | test | ERROR | pool-2-thread-10 | [task:call:3398] Failed to load 2 docs from 0 to 2 2020-08-20 12:58:07,234 | test | ERROR | pool-2-thread-10 | [task:call:3398] Failed to load 1 docs from 100 to 101 2020-08-20 12:58:07,698 | test | INFO | MainThread | [cb_error:revert:60] Reverting 'stop_memcached' in 10.112.191.101 2020-08-20 12:58:12,890 | test | INFO | MainThread | [cb_error:create:37] Simulating 'stop_memcached' in 10.112.191.102 2020-08-20 12:58:12,966 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4491] Performing scope/collection specific operations 2020-08-20 12:58:14,857 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-101 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:14,857 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:14,857 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in YuZxX8l_FTdbPYoGu41XD9IaWn:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2020-08-20 12:58:14,862 | test | ERROR | pool-2-thread-3 | [task:call:3398] Failed to load 1 docs from 3 to 4 2020-08-20 12:58:14,864 | test | ERROR | pool-2-thread-3 | [task:call:3398] Failed to load 1 docs from 2 to 3 2020-08-20 12:58:14,865 | test | ERROR | pool-2-thread-3 | [task:call:3398] Failed to load 1 docs from 101 to 102 2020-08-20 12:58:15,223 | test | INFO | MainThread | [cb_error:revert:60] Reverting 'stop_memcached' in 10.112.191.102 2020-08-20 12:58:20,371 | test | ERROR | MainThread | [durability_helper:verify_vbucket_details_stats:287] Mismatch in sync_write_committed_count stats. 1 != 0 2020-08-20 12:58:20,372 | test | ERROR | MainThread | [durability_helper:verify_vbucket_details_stats:287] Mismatch in ops_delete stats. 1 != 0 2020-08-20 12:58:20,372 | test | ERROR | MainThread | [basetestcase:log_failure:643] Cbstat vbucket-details verification failed after aborts 2020-08-20 12:58:20,710 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: YuZxX8l_FTdbPYoGu41XD9IaWn, Resident ratio(DGM): 100% 2020-08-20 12:58:22,855 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:423] 10.112.191.101 node pcap zip coped on client : True 2020-08-20 12:58:24,576 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:423] 10.112.191.102 node pcap zip coped on client : True 2020-08-20 12:58:24,786 | test | INFO | MainThread | [basetestcase:check_coredump_exist:694] 10.112.191.101 : SSH Successful 2020-08-20 12:58:24,788 | test | INFO | MainThread | [basetestcase:check_coredump_exist:695] 10.112.191.101 : Looking for crash dump files 2020-08-20 12:58:24,799 | test | INFO | MainThread | [basetestcase:check_coredump_exist:714] 10.112.191.101 : No crash files found 2020-08-20 12:58:24,801 | test | INFO | MainThread | [basetestcase:check_coredump_exist:716] 10.112.191.101 : Looking for CRITICAL messages in log 2020-08-20 12:58:25,022 | test | INFO | MainThread | [basetestcase:check_coredump_exist:694] 10.112.191.102 : SSH Successful 2020-08-20 12:58:25,022 | test | INFO | MainThread | [basetestcase:check_coredump_exist:695] 10.112.191.102 : Looking for crash dump files 2020-08-20 12:58:25,032 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:707] Node 10.112.191.102 - Core dump seen: 1 10.112.191.102 : Stack Trace of first crash: 263f6d1a-0dd0-4091-d2a8b8ad-dba7f88e.dmp 2020-08-20 12:58:25,065 | test | INFO | MainThread | [basetestcase:check_coredump_exist:716] 10.112.191.102 : Looking for CRITICAL messages in log 2020-08-20 12:58:25,157 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:595] Running cbcollect on node 10.112.191.101 2020-08-20 12:58:25,187 | test | INFO | MainThread | [rest_client:_http_request:727] {'date': 'Thu, 20 Aug 2020 07:28:24 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2020-08-20 12:58:25,188 | test | INFO | MainThread | [rest_client:_http_request:728] [] 2020-08-20 12:58:35,203 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:599] 10.112.191.101 - cbcollect status: True 2020-08-20 12:58:35,203 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:602] Polling active_tasks to check cbcollect status 2020-08-20 13:00:15,456 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:619] Copying cbcollect ZIP file to Client 2020-08-20 13:00:15,711 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:628] 10.112.191.101 node cb collect zip coped on client : True 2020-08-20 13:00:15,719 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:595] Running cbcollect on node 10.112.191.102 2020-08-20 13:00:15,743 | test | INFO | MainThread | [rest_client:_http_request:727] {'date': 'Thu, 20 Aug 2020 07:30:15 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2020-08-20 13:00:15,743 | test | INFO | MainThread | [rest_client:_http_request:728] [] 2020-08-20 13:00:25,752 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:599] 10.112.191.102 - cbcollect status: True 2020-08-20 13:00:25,755 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:602] Polling active_tasks to check cbcollect status 2020-08-20 13:01:56,017 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:619] Copying cbcollect ZIP file to Client 2020-08-20 13:01:56,539 | test | INFO | MainThread | [basetestcase:fetch_cb_collect_logs:628] 10.112.191.102 node cb collect zip coped on client : True 2020-08-20 13:01:56,568 | test | WARNING | MainThread | [basetestcase:tearDownEverything:468] CLEANUP WAS SKIPPED During the test, Remote Connections: 26, Disconnections: 23 SDK Connections: 10, Disconnections: 10 summary so far suite epengine.collection_crud_negative.CollectionDurabilityTests , pass 0 , fail 1 failures so far... epengine.collection_crud_negative.CollectionDurabilityTests.test_durability_abort testrunner logs, diags and results are available under /Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-20-Aug-20_12-55-05/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 6m 58s 2 actionable tasks: 1 executed, 1 up-to-date test_durability_abort (epengine.collection_crud_negative.CollectionDurabilityTests) ... FAIL ERROR ====================================================================== ERROR: test_durability_abort (epengine.collection_crud_negative.CollectionDurabilityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/collection_crud_negative.py", line 31, in tearDown super(CollectionDurabilityTests, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 88, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 444, in tearDown self.assertFalse(result, msg=core_msg + stream_msg) AssertionError: Node 10.112.191.102 - Core dump seen: 1 ====================================================================== FAIL: test_durability_abort (epengine.collection_crud_negative.CollectionDurabilityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/collection_crud_negative.py", line 270, in test_durability_abort self.validate_test_failure() File "pytests/basetestcase.py", line 650, in validate_test_failure self.fail(self.test_failure) AssertionError: Cbstat vbucket-details verification failed after aborts ---------------------------------------------------------------------- Ran 1 test in 410.542s FAILED (failures=1, errors=1)