Logs will be stored at /data/workspace/centos-p0-analytics-vset00-00-analytics_collection_must_pass_7.0_P0_set3/logs/testrunner-21-Apr-18_00-30-20/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5851.ini GROUP=P0SET3,threads_to_use=10,upgrade_version=7.0.0-4960 -t cbas.cbas_collection_functional_test.CBASDatasetsAndCollections.test_docs_deleted_in_dataset_once_MaxTTL_reached,num_collections=5,upgrade_version=7.0.0-4960,ingestion_timeout=1000,run_query=True,GROUP=P0SET3,services_init=kv:n1ql-kv-cbas-cbas-cbas-cbas,set_cbas_memory_from_available_free_memory=True,bucket_size=auto,bucketTTL=300,threads_to_use=10,num_scopes=5,nodes_init=6,num_items=250,setup_infra=False,override_spec_params=num_items;num_collections;bucket_size;num_scopes;num_buckets,parallel_load_percent=300' Test Input params: {u'num_collections': u'5', 'conf_file': 'cbas/py-cbas-collections.conf', u'upgrade_version': '7.0.0-4960', 'num_nodes': 6, 'spec': 'py-cbas-collections', u'ingestion_timeout': u'1000', 'rerun': False, u'run_query': u'True', u'GROUP': 'P0SET3', u'services_init': u'kv:n1ql-kv-cbas-cbas-cbas-cbas', 'case_number': 11, u'set_cbas_memory_from_available_free_memory': u'True', u'bucket_size': u'auto', 'cluster_name': 'testexec.5851', u'bucketTTL': u'300', 'ini': '/tmp/testexec.5851.ini', u'threads_to_use': '10', 'logs_folder': '/data/workspace/centos-p0-analytics-vset00-00-analytics_collection_must_pass_7.0_P0_set3/logs/testrunner-21-Apr-18_00-30-20/test_11', u'num_scopes': u'5', u'nodes_init': u'6', u'num_items': u'250', u'setup_infra': u'False', u'override_spec_params': u'num_items;num_collections;bucket_size;num_scopes;num_buckets', u'parallel_load_percent': u'300'} test_docs_deleted_in_dataset_once_MaxTTL_reached (cbas.cbas_collection_functional_test.CBASDatasetsAndCollections) ... 2021-04-18 01:47:21,059 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CBASDatasetsAndCollections setup started for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached ========= 2021-04-18 01:47:21,543 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached ========= 2021-04-18 01:47:22,421 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.123.164:8091 during connect (<_realsocket at 0x7122e type=client open_count=1 channel=[id: 0xc76fcf11, 0.0.0.0/0.0.0.0:38914] timeout=300.0>) 2021-04-18 01:47:25,944 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.164:8091 during connect (<_realsocket at 0x7122f type=client open_count=1 channel=[id: 0x8d02ef9b, 0.0.0.0/0.0.0.0:55127] timeout=300.0>) 2021-04-18 01:47:27,973 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.164:8091 during connect (<_realsocket at 0x71230 type=client open_count=1 channel=[id: 0x084e1c31, 0.0.0.0/0.0.0.0:36317] timeout=300.0>) 2021-04-18 01:47:30,003 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.164:8091 during connect (<_realsocket at 0x71231 type=client open_count=1 channel=[id: 0xa12d87bd, 0.0.0.0/0.0.0.0:45810] timeout=300.0>) 2021-04-18 01:47:32,033 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.163:8091 during connect (<_realsocket at 0x71232 type=client open_count=1 channel=[id: 0x8d49652c, 0.0.0.0/0.0.0.0:59537] timeout=300.0>) 2021-04-18 01:47:37,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.163:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.163:8091 during connect (<_realsocket at 0x71233 type=client open_count=1 channel=[id: 0xe1f74793, 0.0.0.0/0.0.0.0:33924] timeout=300.0>) 2021-04-18 01:47:39,543 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.163:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.163:8091 during connect (<_realsocket at 0x71234 type=client open_count=1 channel=[id: 0xe21122a1, 0.0.0.0/0.0.0.0:50199] timeout=300.0>) 2021-04-18 01:47:41,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.163:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.163:8091 during connect (<_realsocket at 0x71235 type=client open_count=1 channel=[id: 0x9a5f1560, 0.0.0.0/0.0.0.0:53515] timeout=300.0>) 2021-04-18 01:47:43,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.163:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.82:8091 during connect (<_realsocket at 0x71236 type=client open_count=1 channel=[id: 0xe5ee5f35, 0.0.0.0/0.0.0.0:47115] timeout=300.0>) 2021-04-18 01:47:49,160 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.82:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.82:8091 during connect (<_realsocket at 0x71237 type=client open_count=1 channel=[id: 0xf1e0a87d, 0.0.0.0/0.0.0.0:41344] timeout=300.0>) 2021-04-18 01:47:51,187 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.82:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.82:8091 during connect (<_realsocket at 0x71238 type=client open_count=1 channel=[id: 0x1b713ce8, 0.0.0.0/0.0.0.0:60690] timeout=300.0>) 2021-04-18 01:47:53,217 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.82:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.82:8091 during connect (<_realsocket at 0x71239 type=client open_count=1 channel=[id: 0x6903c4eb, 0.0.0.0/0.0.0.0:56443] timeout=300.0>) 2021-04-18 01:47:55,246 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.82:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.83:8091 during connect (<_realsocket at 0x7123a type=client open_count=1 channel=[id: 0xdb1bd65d, 0.0.0.0/0.0.0.0:43740] timeout=300.0>) 2021-04-18 01:48:01,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.83:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.83:8091 during connect (<_realsocket at 0x7123b type=client open_count=1 channel=[id: 0xb802fc1a, 0.0.0.0/0.0.0.0:47164] timeout=300.0>) 2021-04-18 01:48:03,056 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.83:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.83:8091 during connect (<_realsocket at 0x7123c type=client open_count=1 channel=[id: 0xae0afb72, 0.0.0.0/0.0.0.0:42389] timeout=300.0>) 2021-04-18 01:48:05,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.83:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.123.83:8091 during connect (<_realsocket at 0x7123d type=client open_count=1 channel=[id: 0x8a8ab840, 0.0.0.0/0.0.0.0:42849] timeout=300.0>) 2021-04-18 01:48:07,112 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.123.83:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.229:8091 during connect (<_realsocket at 0x7123e type=client open_count=1 channel=[id: 0xb31b1e89, 0.0.0.0/0.0.0.0:55486] timeout=300.0>) 2021-04-18 01:48:12,653 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.229:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.229:8091 during connect (<_realsocket at 0x7123f type=client open_count=1 channel=[id: 0x11c4cf4f, 0.0.0.0/0.0.0.0:54093] timeout=300.0>) 2021-04-18 01:48:14,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.229:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.229:8091 during connect (<_realsocket at 0x71240 type=client open_count=1 channel=[id: 0xf54ed375, 0.0.0.0/0.0.0.0:52668] timeout=300.0>) 2021-04-18 01:48:16,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.229:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.229:8091 during connect (<_realsocket at 0x71241 type=client open_count=1 channel=[id: 0x9747606d, 0.0.0.0/0.0.0.0:35481] timeout=300.0>) 2021-04-18 01:48:18,746 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.229:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.35:8091 during connect (<_realsocket at 0x71242 type=client open_count=1 channel=[id: 0x28eb53e0, 0.0.0.0/0.0.0.0:46445] timeout=300.0>) 2021-04-18 01:48:24,232 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.35:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.35:8091 during connect (<_realsocket at 0x71243 type=client open_count=1 channel=[id: 0x6b0deeb4, 0.0.0.0/0.0.0.0:46622] timeout=300.0>) 2021-04-18 01:48:26,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.35:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.35:8091 during connect (<_realsocket at 0x71244 type=client open_count=1 channel=[id: 0x9a18c346, 0.0.0.0/0.0.0.0:49323] timeout=300.0>) 2021-04-18 01:48:28,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.35:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.35:8091 during connect (<_realsocket at 0x71245 type=client open_count=1 channel=[id: 0xc49b03d0, 0.0.0.0/0.0.0.0:56648] timeout=300.0>) 2021-04-18 01:48:30,313 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.35:8091/nodes/self error [Errno 111] Connection refused 2021-04-18 01:48:46,802 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:591] RAM quota was defined less than 100 MB: 2021-04-18 01:48:59,933 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.164, 172.23.123.163, 172.23.123.82, 172.23.123.83, 172.23.121.229, 172.23.105.35 initialized 2021-04-18 01:48:59,973 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached ========= 2021-04-18 01:49:00,036 | test | INFO | MainThread | [cbas_base:setUp:33] Starting Test: test_docs_deleted_in_dataset_once_MaxTTL_reached 2021-04-18 01:49:00,644 | test | INFO | MainThread | [cbas_base:set_memory_for_services:834] Setting 3205 memory quota for cbas 2021-04-18 01:49:14,581 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2059998512 seconds to complete 2021-04-18 01:49:14,601 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-04-18 01:49:20,309 | test | INFO | MainThread | [cbas_base:cbas_logger:350] **********No buckets to disconnect********** 2021-04-18 01:49:20,392 | test | INFO | MainThread | [cbas_base:cbas_logger:350] **********No datasets to drop********** 2021-04-18 01:49:20,446 | test | INFO | MainThread | [cbas_base:cbas_logger:350] **********No buckets to drop********** 2021-04-18 01:49:20,447 | test | INFO | MainThread | [cbas_base:cleanup_cbas:398] Drop Dataverse other than Default and Metadata 2021-04-18 01:49:20,510 | test | INFO | MainThread | [cbas_base:cbas_logger:350] **********No dataverse to drop********** 2021-04-18 01:49:31,088 | test | INFO | pool-13-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-----------+-----------------------+---------------+--------------+ | 172.23.123.164 | kv, n1ql | 7.0.0-4960-enterprise | 1.05633802817 | Cluster node | | 172.23.123.82 | cbas | 7.0.0-4960-enterprise | 4.12994208008 | Cluster node | | 172.23.123.163 | [u'kv'] | | | <--- IN --- | | 172.23.123.83 | [u'cbas'] | | | <--- IN --- | | 172.23.121.229 | [u'cbas'] | | | <--- IN --- | | 172.23.105.35 | [u'cbas'] | | | <--- IN --- | +----------------+-----------+-----------------------+---------------+--------------+ 2021-04-18 01:49:31,121 | test | INFO | pool-13-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-18 01:49:36,197 | test | INFO | pool-13-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-18 01:49:46,316 | test | INFO | pool-13-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-18 01:49:46,357 | test | INFO | pool-13-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 15.2690000534 sec 2021-04-18 01:49:46,358 | test | INFO | MainThread | [cbas_base:setUp:336] === CBAS_BASE setup was finished for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached === 2021-04-18 01:49:46,358 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CBASDatasetsAndCollections setUp Finished for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached ========= 2021-04-18 01:49:46,359 | test | INFO | MainThread | [cbas_collection_functional_test:test_docs_deleted_in_dataset_once_MaxTTL_reached:1370] Test started 2021-04-18 01:49:46,767 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-18 01:49:48,890 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-04-18 01:49:49,118 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 0 | 2929721344 | 0 | 0 | | default | couchbase | 1 | none | 300 | 0 | 0 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-18 01:49:49,118 | test | INFO | MainThread | [cbas_base:collectionSetUp:700] Creating required SDK clients for client_pool 2021-04-18 01:49:59,219 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.123.163 | kv | 24.5340822058 | 4201672704 | 3372675072 | 0 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | | 172.23.121.229 | cbas | 4.03429147756 | 4201684992 | 3100147712 | 0 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | | 172.23.123.164 | kv, n1ql | 26.7584097859 | 4201672704 | 3188133888 | 0 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | | 172.23.105.35 | cbas | 4.07342217752 | 4201844736 | 3126292480 | 0 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | | 172.23.123.82 | cbas | 3.85584677419 | 4201672704 | 3041091584 | 110915584 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | | 172.23.123.83 | cbas | 4.10682791635 | 4201676800 | 3099967488 | 0 / 3758092288 | 0 / 0 | 7.0.0-4960-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-04-18 01:49:59,221 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-18 01:49:59,234 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-18 01:50:06,609 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-18 01:50:19,630 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-18 01:50:19,641 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-18 01:50:19,654 | test | INFO | MainThread | [cbas_utils_v2:create_datasets_on_all_collections:2134] Creating Datasets on all KV collections 2021-04-18 01:50:19,750 | test | INFO | pool-13-thread-10 | [common_lib:sleep:22] Sleep 12 seconds. Reason: Wait for atleast one dataset to be created 2021-04-18 01:50:34,191 | test | INFO | pool-13-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 383.100 | .......**.......*******.....................................................................................X | 4455.200 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-04-18 01:50:34,430 | test | INFO | pool-13-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 932.900 | ...........*X | 932.900 | | 251.500 | ..*******.................................................................................................X | 7323.500 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-04-18 01:52:29,507 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-18 01:52:42,440 | test | INFO | MainThread | [cbas_utils_v2:wait_for_ingestion_all_datasets:1650] Waiting for data to be ingested into datasets 2021-04-18 01:52:45,559 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 330 seconds. Reason: waiting for maxTTL to complete 2021-04-18 01:58:15,561 | test | INFO | MainThread | [cbas_collection_functional_test:test_docs_deleted_in_dataset_once_MaxTTL_reached:1419] Validating item count 2021-04-18 01:58:18,993 | test | INFO | MainThread | [cbas_collection_functional_test:test_docs_deleted_in_dataset_once_MaxTTL_reached:1459] Test finished 2021-04-18 01:58:18,994 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CBASDatasetsAndCollections tearDown Started for test #11 test_docs_deleted_in_dataset_once_MaxTTL_reached ========= 2021-04-18 01:58:22,168 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:724] 172.23.123.163: 1 core dump seen 2021-04-18 01:58:22,644 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:725] 172.23.123.163: Stack Trace of first crash - 8f436485-1302-44c1-dca68dba-71ced933.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f9a31f52337 in raise () from /lib64/libc.so.6 #0 0x00007f9a31f52337 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f9a31f53a28 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f9a32ab963c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x00000000009131ef in backtrace_terminate_handler() () No symbol table info available. #4 0x00007f9a32ac48f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007f9a32ac4961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007f9a32ac4bf4 in __cxxabiv1::__cxa_throw (obj=, tinfo=0xd61bf0 , dest=0x439fc0 <_ZNSt11logic_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 globals = header = 0x7f9a04000eb0 #7 0x000000000049cbbd in CheckpointManager::removeCursor_UNLOCKED(CheckpointCursor*) [clone .cold] () No symbol table info available. #8 0x000000000076b217 in CheckpointManager::removeCursor(CheckpointCursor*) () No symbol table info available. #9 0x000000000084bec9 in ActiveStream::removeCheckpointCursor() () No symbol table info available. #10 0x000000000084fffc in ActiveStream::endStream(cb::mcbp::DcpStreamEndStatus) () No symbol table info available. #11 0x0000000000850ed5 in ActiveStream::setDead(cb::mcbp::DcpStreamEndStatus) () No symbol table info available. #12 0x000000000078bf5e in DcpProducer::closeAllStreams() () No symbol table info available. #13 0x000000000077fabd in DcpConnMap::disconnect(void const*) () No symbol table info available. #14 0x0000000000621eb9 in EventuallyPersistentEngine::disconnect(gsl::not_null) () No symbol table info available. #15 0x0000000000576862 in Connection::propagateDisconnect() const () No symbol table info available. #16 0x00000000005768f6 in Connection::close() () No symbol table info available. #17 0x00000000005806d7 in Connection::executeCommandsCallback() () No symbol table info available. #18 0x00000000005810d1 in Connection::rw_callback(bufferevent*, void*) () No symbol table info available. #19 0x00007f9a33bc6d8e in bufferevent_run_deferred_callbacks_unlocked () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 No symbol table info available. #20 0x00007f9a33bcfd01 in event_process_active_single_queue.isra.31 () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 No symbol table info available. #21 0x00007f9a33bd054f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 No symbol table info available. #22 0x000000000055ac5b in worker_libevent(void*) () No symbol table info available. #23 0x00007f9a34b22be9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 No symbol table info available. #24 0x00007f9a322f1e65 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #25 0x00007f9a3201a88d in clone () from /lib64/libc.so.6 No symbol table info available. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 32 (Thread 0x7f589f5ff700 (LWP 19507)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a202c5a7 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f589e87e700 (LWP 19508)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000091f795 in spdlog::details::thread_pool::process_next_msg_() () #2 0x000000000091fdb0 in spdlog::details::thread_pool::worker_loop_() () #3 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #4 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f589e07d700 (LWP 19509)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000091ef4b in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv () #2 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #3 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f589d87c700 (LWP 19510)): #0 0x00007f58a0229bed in poll () from /lib64/libc.so.6 #1 0x000000000055a6d1 in check_stdin_thread(void*) () #2 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f589d07b700 (LWP 19511)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000089dd2b in worker_thread () #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f589c87a700 (LWP 19512)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000089dd2b in worker_thread () #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f5897fff700 (LWP 19513)): #0 0x00007f58a0229bed in poll () from /lib64/libc.so.6 #1 0x0000000000894c98 in master_thread () #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f58977fe700 (LWP 19514)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000058b3c6 in ExternalAuthManagerThread::run() () #2 0x00007f58a2d4d143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #4 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f5896ffd700 (LWP 19515)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005f96c7 in AuditImpl::consume_events() () #2 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f58967fc700 (LWP 19516)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055ac5b in worker_libevent(void*) () #4 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f5895ffb700 (LWP 19517)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055ac5b in worker_libevent(void*) () #4 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f58957fa700 (LWP 19518)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055ac5b in worker_libevent(void*) () #4 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f5894ff9700 (LWP 19519)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055ac5b in worker_libevent(void*) () #4 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f588ffff700 (LWP 19520)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a0d028bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059e2f1 in Executor::run() () #4 0x00007f58a2d4d143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f588f7fe700 (LWP 19521)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a0d028bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059e2f1 in Executor::run() () #4 0x00007f58a2d4d143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f588effd700 (LWP 19522)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a0d028bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059e2f1 in Executor::run() () #4 0x00007f58a2d4d143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f588e7fc700 (LWP 19523)): #0 0x00007f58a050f9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a0d028bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059e2f1 in Executor::run() () #4 0x00007f58a2d4d143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007f58a2d3cbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f5887dff700 (LWP 19526)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a202be51 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f58875fe700 (LWP 19527)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f5886dfd700 (LWP 19528)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f58865fc700 (LWP 19529)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f5885dfb700 (LWP 19530)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f58855fa700 (LWP 19531)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f5884df9700 (LWP 19532)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f585e7da700 (LWP 19533)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f585dfd9700 (LWP 19534)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965b97 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f585d7d8700 (LWP 19535)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965a5d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f585cfd7700 (LWP 19536)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965b97 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f583bfff700 (LWP 19537)): #0 0x00007f58a022eba9 in syscall () from /lib64/libc.so.6 #1 0x000000000095872d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095fb1a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000965b97 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009660c6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000969820 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095dcb7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x000000000066a02d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f583affd700 (LWP 19539)): #0 0x00007f58a050fda2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f58a202be51 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f58399ff700 (LWP 19540)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000097e3c7 in folly::EventBase::loopBody(int, bool) () #4 0x000000000097e896 in folly::EventBase::loop() () #5 0x0000000000980256 in folly::EventBase::loopForever() () #6 0x000000000096d879 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000979009 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x00007f58a0d07d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #9 0x00007f58a050be65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f58a023488d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f58a41e3e80 (LWP 19506)): #0 0x00007f58a0234e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f58a1df33c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f58a1dea376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000050b1b6 in memcached_main(int, char**) () #4 0x00007f58a0158505 in __libc_start_main () from /lib64/libc.so.6 #5 0x0000000000501341 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 19506 2021-04-18 01:58:25,934 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1085] Running cbcollect on node ns_1@172.23.105.35,ns_1@172.23.121.229,ns_1@172.23.123.163,ns_1@172.23.123.164,ns_1@172.23.123.82,ns_1@172.23.123.83 2021-04-18 01:58:35,990 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1088] ns_1@172.23.105.35,ns_1@172.23.121.229,ns_1@172.23.123.163,ns_1@172.23.123.164,ns_1@172.23.123.82,ns_1@172.23.123.83 - cbcollect status: True 2021-04-18 01:58:35,993 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1093] Polling active_tasks to check cbcollect status 2021-04-18 02:03:17,503 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.105.35: Copying cbcollect ZIP file to Client 2021-04-18 02:03:20,546 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.105.35 node cb collect zip copied on client : True 2021-04-18 02:03:20,546 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.121.229: Copying cbcollect ZIP file to Client 2021-04-18 02:03:23,588 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.121.229 node cb collect zip copied on client : True 2021-04-18 02:03:23,589 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.123.163: Copying cbcollect ZIP file to Client 2021-04-18 02:03:25,378 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.123.163 node cb collect zip copied on client : True 2021-04-18 02:03:25,380 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.123.164: Copying cbcollect ZIP file to Client 2021-04-18 02:03:27,279 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.123.164 node cb collect zip copied on client : True 2021-04-18 02:03:27,279 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.123.82: Copying cbcollect ZIP file to Client 2021-04-18 02:03:30,691 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.123.82 node cb collect zip copied on client : True 2021-04-18 02:03:30,693 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.123.83: Copying cbcollect ZIP file to Client 2021-04-18 02:03:33,740 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.123.83 node cb collect zip copied on client : True 2021-04-18 02:03:49,112 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1889998913 seconds to complete 2021-04-18 02:03:49,135 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_docs_deleted_in_dataset_once_MaxTTL_reached (cbas.cbas_collection_functional_test.CBASDatasetsAndCollections) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/cbas/cbas_collection_functional_test.py", line 273, in tearDown super(CBASDatasetsAndCollections, self).tearDown() File "pytests/cbas/cbas_base.py", line 346, in tearDown super(CBASBaseTest, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 996.697s