Logs will be stored at /data/workspace/centos-p0-couchstore-vset00-00-crash_recovery_graceful_with_durability_7.0_P0/logs/testrunner-21-Apr-21_21-12-16/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.72417.ini bucket_storage=couchstore,rerun=false,GROUP=P0;graceful_durability,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,nodes_init=4,graceful=True,collect_pcaps=True,upgrade_version=7.0.0-4987 -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery,num_items=5000000,doc_size=1024,graceful=True,sdk_timeout=60,doc_ops=update:delete:expiry,replicas=1,durability=PERSIST_TO_MAJORITY,GROUP=P0;graceful_durability' Test Input params: {'doc_size': '256', 'conf_file': 'conf/magma/crash_recovery.conf', 'upgrade_version': '7.0.0-4987', 'graceful': 'True', 'spec': 'crash_recovery', 'rerun': 'false', 'num_nodes': 4, 'GROUP': 'P0;graceful_durability', 'doc_ops': 'update:delete:expiry', 'sdk_timeout': '60', 'case_number': 4, 'cluster_name': 'testexec.72417', 'ini': '/tmp/testexec.72417.ini', 'replicas': '1', 'durability': 'PERSIST_TO_MAJORITY', 'bucket_storage': 'couchstore', 'bucket_eviction_policy': 'fullEviction', 'logs_folder': '/data/workspace/centos-p0-couchstore-vset00-00-crash_recovery_graceful_with_durability_7.0_P0/logs/testrunner-21-Apr-21_21-12-16/test_4', 'nodes_init': '4', 'num_items': '5000000', 'collect_pcaps': 'True', 'randomize_value': 'true'} test_crash_during_recovery (magma.magma_crash_recovery.MagmaCrashTests) ... 2021-04-21 23:31:29,648 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= MagmaCrashTests setup started for test #4 test_crash_during_recovery ========= 2021-04-21 23:31:30,118 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= BaseTestCase setup started for test #4 test_crash_during_recovery ========= 2021-04-21 23:31:30,322 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.106.60:8091 during connect (<_realsocket at 0x2702bd type=client open_count=1 channel=[id: 0x12a50e1e, 0.0.0.0/0.0.0.0:39547] timeout=300.0>) 2021-04-21 23:31:34,339 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.60:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.60:8091 during connect (<_realsocket at 0x2702be type=client open_count=1 channel=[id: 0x200cc9d7, 0.0.0.0/0.0.0.0:51095] timeout=300.0>) 2021-04-21 23:31:36,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.60:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.60:8091 during connect (<_realsocket at 0x2702bf type=client open_count=1 channel=[id: 0x58e3fc2e, 0.0.0.0/0.0.0.0:58025] timeout=300.0>) 2021-04-21 23:31:38,381 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.60:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.60:8091 during connect (<_realsocket at 0x2702c0 type=client open_count=1 channel=[id: 0x7cb99150, 0.0.0.0/0.0.0.0:52024] timeout=300.0>) 2021-04-21 23:31:40,401 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.60:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.53:8091 during connect (<_realsocket at 0x2702c1 type=client open_count=1 channel=[id: 0xb121c344, 0.0.0.0/0.0.0.0:55099] timeout=300.0>) 2021-04-21 23:31:46,198 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.53:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.53:8091 during connect (<_realsocket at 0x2702c2 type=client open_count=1 channel=[id: 0x82b70718, 0.0.0.0/0.0.0.0:49297] timeout=300.0>) 2021-04-21 23:31:48,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.53:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.53:8091 during connect (<_realsocket at 0x2702c3 type=client open_count=1 channel=[id: 0xe3dfdccb, 0.0.0.0/0.0.0.0:44087] timeout=300.0>) 2021-04-21 23:31:50,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.53:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.53:8091 during connect (<_realsocket at 0x2702c4 type=client open_count=1 channel=[id: 0x6357f193, 0.0.0.0/0.0.0.0:47811] timeout=300.0>) 2021-04-21 23:31:52,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.53:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.56:8091 during connect (<_realsocket at 0x2702c5 type=client open_count=1 channel=[id: 0xc8731aa7, 0.0.0.0/0.0.0.0:46829] timeout=300.0>) 2021-04-21 23:31:57,596 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.56:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.56:8091 during connect (<_realsocket at 0x2702c6 type=client open_count=1 channel=[id: 0x4db8a18f, 0.0.0.0/0.0.0.0:41329] timeout=300.0>) 2021-04-21 23:31:59,803 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.56:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.56:8091 during connect (<_realsocket at 0x2702c7 type=client open_count=1 channel=[id: 0xb9a82a30, 0.0.0.0/0.0.0.0:41205] timeout=300.0>) 2021-04-21 23:32:02,214 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.56:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.62:8091 during connect (<_realsocket at 0x2702c8 type=client open_count=1 channel=[id: 0xc07eb927, 0.0.0.0/0.0.0.0:40179] timeout=300.0>) 2021-04-21 23:32:08,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.62:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.62:8091 during connect (<_realsocket at 0x2702c9 type=client open_count=1 channel=[id: 0x6c465330, 0.0.0.0/0.0.0.0:44071] timeout=300.0>) 2021-04-21 23:32:10,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.62:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.62:8091 during connect (<_realsocket at 0x2702ca type=client open_count=1 channel=[id: 0x7521b2c4, 0.0.0.0/0.0.0.0:52586] timeout=300.0>) 2021-04-21 23:32:13,006 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.62:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.62:8091 during connect (<_realsocket at 0x2702cb type=client open_count=1 channel=[id: 0x2e3670da, 0.0.0.0/0.0.0.0:46781] timeout=300.0>) 2021-04-21 23:32:15,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.62:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.62:8091 during connect (<_realsocket at 0x2702cc type=client open_count=1 channel=[id: 0x569ea1e9, 0.0.0.0/0.0.0.0:36843] timeout=300.0>) 2021-04-21 23:32:17,048 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.62:8091/nodes/self error [Errno 111] Connection refused 2021-04-21 23:32:32,806 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:593] RAM quota was defined less than 100 MB: 2021-04-21 23:32:44,957 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.106.60, 172.23.106.53, 172.23.106.56, 172.23.106.62 initialized 2021-04-21 23:32:44,984 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= BaseTestCase setup finished for test #4 test_crash_during_recovery ========= 2021-04-21 23:32:53,914 | test | INFO | pool-6-thread-5 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.106.60 | kv | 7.0.0-4987-enterprise | 7.58437618506 | Cluster node | | 172.23.106.53 | ['kv'] | | | <--- IN --- | | 172.23.106.56 | ['kv'] | | | <--- IN --- | | 172.23.106.62 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-04-21 23:32:53,927 | test | INFO | pool-6-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-21 23:33:03,967 | test | INFO | pool-6-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-21 23:33:04,115 | test | INFO | pool-6-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 10.2029998302 sec 2021-04-21 23:33:09,028 | test | INFO | MainThread | [magma_base:setUp:95] Scopes list is ['_default'] 2021-04-21 23:33:11,278 | test | INFO | MainThread | [magma_base:get_disk_usage:708] Disk usage stats for bucekt default is below 2021-04-21 23:33:11,278 | test | INFO | MainThread | [magma_base:get_disk_usage:709] Total Disk usage for kvstore is 0MB 2021-04-21 23:33:11,555 | test | INFO | MainThread | [magma_base:get_bucket_dgm:660] Active Resident Threshold of default is 100 2021-04-21 23:33:11,555 | test | INFO | MainThread | [magma_base:setUp:152] disk usage before loading 0 2021-04-21 23:41:49,865 | test | INFO | pool-6-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.....**..........*................**................***..............*........**...*.......***.......*......**.......*.**X | 19279.700 | | 18707.600 | ................................................................................................*X | 18707.600 | | 17888.300 | ...........................................................................................X | 17888.300 | | 17126.600 | .......................................................................................*X | 17126.600 | | 16398.700 | ....................................................................................*X | 16398.700 | | 15475.400 | ...............................................................................*X | 15475.400 | | 14899.108 | ............................................................................X | 14899.108 | | 13991.308 | .......................................................................*X | 13991.308 | | 13692.308 | ......................................................................***..*...*....**.*X | 15632.440 | | 14332.940 | .........................................................................*X | 14332.940 | | 12822.600 | .................................................................X | 12822.600 | | 11785.500 | ............................................................*X | 11785.500 | | 10839.500 | .......................................................***....*.*X | 11777.900 | | 11144.100 | ........................................................X | 11144.100 | | 10885.500 | .......................................................**X | 10885.500 | | 9759.900 | .................................................***.....*....***....X | 12215.600 | | 12046.800 | .............................................................**X | 12046.800 | | 8789.800 | ............................................**X | 8789.800 | | 7741.000 | .......................................X | 7741.000 | | 7276.700 | ....................................***............**X | 9454.500 | | 9267.200 | ...............................................X | 9267.200 | | 9054.700 | .............................................*X | 9054.700 | | 8211.586 | .........................................X | 8211.586 | | 7601.386 | ......................................***.....**..*.....*.**X | 10123.403 | | 9624.503 | ................................................**...X | 10047.403 | | 9953.400 | ..................................................**.**....*X | 10854.600 | | 10565.400 | .....................................................X | 10565.400 | | 10374.800 | ....................................................**.*X | 10576.200 | | 9544.700 | ................................................*X | 9544.700 | | 9157.500 | ..............................................X | 9157.500 | | 8944.900 | .............................................*X | 8944.900 | | 8295.800 | ..........................................**X | 8295.800 | | 8211.400 | .........................................*.X | 8353.971 | | 8332.171 | ..........................................****....**.*...**......*X | 11108.724 | | 10657.124 | ......................................................***X | 10718.200 | | 10481.800 | .....................................................X | 10481.800 | | 9003.000 | .............................................**X | 9003.000 | | 8911.500 | .............................................X | 8911.500 | | 8215.900 | .........................................*X | 8215.900 | | 7319.000 | ....................................*.**X | 7423.900 | | 6687.200 | .................................X | 6687.200 | | 6381.200 | ................................*X | 6381.200 | | 6369.400 | ................................X | 6369.400 | | 6116.500 | ..............................***..*...**....*..***...*..*X | 9143.700 | | 8881.600 | .............................................*....**X | 9769.200 | | 9646.000 | .................................................*....**........*...***....*X | 13393.800 | | 13052.800 | ..................................................................X | 13052.800 | | 11444.300 | ..........................................................X | 11444.300 | | 9911.000 | ..................................................***X | 9911.000 | | 8503.900 | ...........................................X | 8503.900 | | 7407.000 | .....................................X | 7407.000 | | 7154.100 | ....................................X | 7154.100 | | 7117.800 | ...................................**X | 7117.800 | | 6691.400 | .................................*X | 6691.400 | | 6355.500 | ...............................X | 6355.500 | | 5864.800 | .............................*X | 5864.800 | | 5369.200 | ..........................**.****.*.*X | 5921.200 | | 5880.800 | .............................**..***....**.........**........**........*....**X | 12570.700 | | 11333.025 | .........................................................X | 11333.025 | | 10349.925 | ....................................................*X | 10349.925 | | 10023.925 | ..................................................*.***...........*......*X | 13332.100 | | 13005.600 | ..................................................................X | 13005.600 | | 12144.500 | .............................................................**X | 12144.500 | | 10009.916 | ..................................................X | 10009.916 | | 8882.216 | .............................................***....***.....*..*X | 11161.824 | | 11123.624 | ........................................................X | 11123.624 | | 10450.924 | .....................................................****...**..*..**X | 11880.651 | | 11641.800 | ...........................................................X | 11641.800 | | 11283.000 | .........................................................**...*.....***.......*....*X | 14922.553 | | 14250.253 | ........................................................................X | 14250.253 | | 13277.953 | ...................................................................**X | 13277.953 | | 12727.127 | .................................................................***.....*....**X | 14631.227 | | 12882.000 | .................................................................X | 12882.000 | | 11253.500 | .........................................................*X | 11253.500 | | 9094.100 | ..............................................X | 9094.100 | | 7059.400 | ...................................***.X | 7258.200 | | 6943.900 | ...................................**X | 7024.000 | | 6835.300 | ..................................**X | 6835.300 | | 5949.400 | .............................X | 5949.400 | | 5404.200 | ...........................*X | 5404.200 | | 4935.700 | ........................X | 4935.700 | | 4731.900 | .......................***.**..*...*....****...*...*.*.**X | 8283.600 | | 8231.300 | .........................................*X | 8231.300 | | 8194.500 | .........................................*.*.***...**....*..X | 10326.600 | | 10016.100 | ..................................................**X | 10016.100 | | 9258.000 | ...............................................*X | 9258.000 | | 8149.700 | .........................................X | 8149.700 | | 6840.600 | ..................................*X | 6840.600 | | 5955.100 | .............................*X | 5955.100 | | 5614.600 | ............................*X | 5614.600 | | 5585.900 | ...........................**..*..******X | 6319.300 | | 6305.800 | ...............................***.X | 6418.000 | | 6386.100 | ................................*X | 6386.100 | | 6349.600 | ...............................X | 6349.600 | | 6240.500 | ...............................***..*...**...*.**X | 8085.400 | | 7597.400 | ......................................X | 7597.400 | | 7192.400 | ....................................*X | 7192.400 | | 6977.800 | ...................................*X | 6977.800 | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-21 23:41:57,625 | test | INFO | MainThread | [magma_base:get_disk_usage:708] Disk usage stats for bucekt default is below 2021-04-21 23:41:57,625 | test | INFO | MainThread | [magma_base:get_disk_usage:709] Total Disk usage for kvstore is 0MB 2021-04-21 23:41:58,005 | test | INFO | MainThread | [magma_base:get_bucket_dgm:660] Active Resident Threshold of default is 100 2021-04-21 23:41:58,005 | test | INFO | MainThread | [magma_base:initial_load:237] For bucket default disk usage after initial creation is 0MB 2021-04-21 23:41:58,005 | test | INFO | MainThread | [magma_base:setUp:199] ==========Finished magma base setup======== 2021-04-21 23:41:59,726 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 55 seconds. Reason: Iteration:1 waiting for 55 sec to kill memcached on all nodes 2021-04-21 23:42:54,839 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:42:58,022 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:01,065 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:01,089 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:04,134 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:04,157 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:07,230 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:10,319 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:13,413 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:16,476 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:16,513 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:19,561 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:19,585 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:22,658 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:25,742 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:28,881 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:31,937 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:31,986 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:35,029 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:35,051 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:38,118 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:41,204 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:44,321 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:47,403 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:47,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:50,473 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:43:50,494 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:43:53,605 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:09,259 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 31 seconds. Reason: Iteration:2 waiting for 31 sec to kill memcached on all nodes 2021-04-21 23:44:40,342 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:43,536 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:46,581 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:44:46,605 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:49,648 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:44:49,671 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:52,740 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:55,831 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:44:59,477 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:02,533 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:02,569 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:05,618 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:05,645 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:08,720 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:11,802 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:14,961 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:18,006 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:18,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:21,091 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:21,115 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:24,917 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:28,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:31,157 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:34,217 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:34,244 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:37,288 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:45:37,311 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:40,492 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:45:56,806 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 30 seconds. Reason: Iteration:3 waiting for 30 sec to kill memcached on all nodes 2021-04-21 23:46:26,914 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:30,065 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:33,111 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:46:33,138 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:36,186 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:46:36,211 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:39,361 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:42,450 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:45,536 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:48,671 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:46:48,698 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:51,746 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:46:51,772 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:54,849 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:46:57,933 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:01,122 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:04,285 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:47:04,334 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:07,378 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:47:07,401 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:10,476 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:13,562 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:16,686 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:19,740 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:47:19,778 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:22,819 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:47:22,844 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:25,917 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:47:40,864 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 47 seconds. Reason: Iteration:4 waiting for 47 sec to kill memcached on all nodes 2021-04-21 23:48:27,951 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:31,117 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:34,161 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:48:34,184 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:37,229 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:48:37,253 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:40,332 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:43,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:46,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:49,654 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:48:49,681 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:52,732 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:48:52,757 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:55,869 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:48:58,953 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:02,115 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:05,170 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:49:05,203 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:08,250 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:49:08,273 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:11,342 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:14,427 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:17,499 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:20,562 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:49:20,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:23,632 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:49:23,655 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:26,743 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:49:42,096 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 55 seconds. Reason: Iteration:5 waiting for 55 sec to kill memcached on all nodes 2021-04-21 23:50:37,194 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:40,335 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:43,382 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:50:43,407 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:46,451 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:50:46,474 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:49,543 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:52,634 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:56,069 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:50:59,138 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:50:59,167 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:02,214 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:51:02,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:05,319 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:08,404 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:11,565 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:14,615 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:51:14,654 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:17,696 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:51:17,719 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:20,903 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:23,992 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:27,135 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:30,196 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:51:30,233 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:33,276 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:51:33,301 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:36,369 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:51:51,634 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 38 seconds. Reason: Iteration:6 waiting for 38 sec to kill memcached on all nodes 2021-04-21 23:52:29,720 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:32,809 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:35,854 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:52:35,875 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:38,921 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:52:38,943 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:42,012 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:45,107 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:48,223 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:51,328 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:52:51,357 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:54,414 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:52:54,440 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:52:57,766 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:00,848 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:03,980 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:07,071 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:53:07,122 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:10,171 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:53:10,201 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:13,641 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:16,724 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:19,826 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:22,882 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:53:22,930 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:25,976 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:53:26,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:29,094 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:53:45,170 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 60 seconds. Reason: Iteration:7 waiting for 60 sec to kill memcached on all nodes 2021-04-21 23:54:45,265 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:54:48,459 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:54:51,505 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:54:51,526 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:54:54,571 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:54:54,595 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:54:57,671 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:00,763 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:03,923 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:06,982 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:07,017 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:10,069 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:10,094 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:13,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:16,319 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:19,440 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:22,510 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:22,542 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:25,586 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:25,611 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:28,684 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:31,769 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:34,884 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:37,934 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:37,960 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:41,007 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:55:41,032 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:44,098 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:55:58,698 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 40 seconds. Reason: Iteration:8 waiting for 40 sec to kill memcached on all nodes 2021-04-21 23:56:38,779 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:41,897 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:44,947 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:56:44,973 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:48,020 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:56:48,046 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:51,121 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:54,217 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:56:57,423 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:00,492 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:00,523 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:03,591 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:03,634 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:06,720 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:09,799 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:12,950 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:16,099 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:16,157 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:19,292 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:19,316 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:22,505 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:25,591 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:28,660 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:31,704 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:31,726 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:34,769 | infra | ERROR | Thread-4 | [remote_util:log_command_output:3065] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2021-04-21 23:57:34,792 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:37,865 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 3 seconds. Reason: Sleep before killing memcached on same node again. 2021-04-21 23:57:41,335 | test | CRITICAL | Thread-4 | [basetestcase:check_coredump_exist:726] 172.23.106.60: 1 core dump seen 2021-04-21 23:57:42,911 | test | CRITICAL | Thread-4 | [basetestcase:check_coredump_exist:727] 172.23.106.60: Stack Trace of first crash - a7fe9847-4d28-460d-d38b4cb3-233444d1.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00000000007e2546 in PersistenceCallback::operator()(EPTransactionContext&, SingleThreadedRCPtr >, KVStore::FlushStateMutation) () #0 0x00000000007e2546 in PersistenceCallback::operator()(EPTransactionContext&, SingleThreadedRCPtr >, KVStore::FlushStateMutation) () No symbol table info available. #1 0x00000000007e2935 in EPTransactionContext::setCallback(SingleThreadedRCPtr > const&, KVStore::FlushStateMutation) () No symbol table info available. #2 0x00000000007f3351 in CouchKVStore::commitCallback(std::deque >&, kvstats_ctx&, couchstore_error_t) () No symbol table info available. #3 0x0000000000808052 in CouchKVStore::commit(VB::Commit&) () No symbol table info available. #4 0x00000000007c0abe in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () No symbol table info available. #5 0x00000000007c5f2c in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () No symbol table info available. #6 0x00000000007c621f in EPBucket::flushVBucket(Vbid) () No symbol table info available. #7 0x000000000066d850 in Flusher::flushVB() () No symbol table info available. #8 0x000000000066e319 in Flusher::step(GlobalTask*) () No symbol table info available. #9 0x0000000000670557 in GlobalTask::execute() () No symbol table info available. #10 0x000000000066b6a0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () No symbol table info available. #11 0x0000000000975560 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () No symbol table info available. #12 0x000000000095d23a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () No symbol table info available. #13 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () No symbol table info available. #14 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () No symbol table info available. #15 0x00007fc465e78d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 No locals. #16 0x00007fc46567cea5 in start_thread (arg=0x7fc3f1ffb700) at pthread_create.c:307 __res = pd = 0x7fc3f1ffb700 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140479555417856, -519509859033764609, 0, 8392704, 0, 140479555417856, 526278440538254591, 523154533403110655}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = #17 0x00007fc4653a58dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done. done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.17.so.debug...done. done. Loaded symbols for /lib64/librt.so.1 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 41 (Thread 0x7fd434fff700 (LWP 114077)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fd437a225a7 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd434fff700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (Thread 0x7fd43427e700 (LWP 114078)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000091eca5 in spdlog::details::thread_pool::process_next_msg_() () #2 0x000000000091f2c0 in spdlog::details::thread_pool::worker_loop_() () #3 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #4 0x00007fd435f01ea5 in start_thread (arg=0x7fd43427e700) at pthread_create.c:307 #5 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (Thread 0x7fd433a7d700 (LWP 114079)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000091e45b in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv () #2 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd433a7d700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (Thread 0x7fd43327c700 (LWP 114080)): #0 0x00007fd435c1fc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000055a2d1 in check_stdin_thread(void*) () #2 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd43327c700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (Thread 0x7fd432a7b700 (LWP 114081)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000089d1bb in worker_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd432a7b700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (Thread 0x7fd43227a700 (LWP 114082)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000089d1bb in worker_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd43227a700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (Thread 0x7fd431a79700 (LWP 114083)): #0 0x00007fd435c1fc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000894128 in master_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd431a79700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7fd431278700 (LWP 114084)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000058afc6 in ExternalAuthManagerThread::run() () #2 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #4 0x00007fd435f01ea5 in start_thread (arg=0x7fd431278700) at pthread_create.c:307 #5 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7fd42bfff700 (LWP 114085)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000005f92c7 in AuditImpl::consume_events() () #2 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd42bfff700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7fd42b7fe700 (LWP 114086)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42b7fe700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7fd42affd700 (LWP 114087)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42affd700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7fd42a7fc700 (LWP 114088)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42a7fc700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7fd429ffb700 (LWP 114089)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd429ffb700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7fd4297fa700 (LWP 114090)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd4297fa700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7fd428ff9700 (LWP 114091)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd428ff9700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7fd423fff700 (LWP 114092)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd423fff700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 0x7fd4237fe700 (LWP 114093)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4237fe700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7fd422ffd700 (LWP 114094)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd422ffd700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7fd4227fc700 (LWP 114095)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4227fc700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7fd421ffb700 (LWP 114096)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd421ffb700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7fd4217fa700 (LWP 114097)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4217fa700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7fd420ff9700 (LWP 114098)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd420ff9700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7fd4207f8700 (LWP 114099)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4207f8700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7fd41e3f6700 (LWP 114101)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fd437a21e51 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd41e3f6700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7fd41dbf5700 (LWP 114102)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41dbf5700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7fd41d3f4700 (LWP 114103)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41d3f4700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7fd41cbf3700 (LWP 114104)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41cbf3700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7fd3db3ff700 (LWP 114105)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3db3ff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7fd3d93d9700 (LWP 114106)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3d93d9700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7fd3d8bd8700 (LWP 114107)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3d8bd8700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7fd3c7fff700 (LWP 114108)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c7fff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7fd3c77fe700 (LWP 114109)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c77fe700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7fd3c6ffd700 (LWP 114110)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c6ffd700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fd3c67fc700 (LWP 114111)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c67fc700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fd3c5ffb700 (LWP 114112)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c5ffb700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7fd3c57fa700 (LWP 114113)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c57fa700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7fd3c4ff9700 (LWP 114114)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c4ff9700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7fd3a3fff700 (LWP 114115)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a3fff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7fd3a37fe700 (LWP 114116)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a37fe700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7fd3a2ffd700 (LWP 114118)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000097d8d7 in folly::EventBase::loopBody(int, bool) () #4 0x000000000097dda6 in folly::EventBase::loop() () #5 0x000000000097f766 in folly::EventBase::loopForever() () #6 0x000000000096cd89 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #9 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a2ffd700) at pthread_create.c:307 #10 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7fd4399d5e80 (LWP 114076)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000050adb6 in memcached_main(int, char**) () #4 0x00007fd435b4e555 in __libc_start_main (main=0x4e3fa0
, argc=3, argv=0x7ffc8ba27728, init=, fini=, rtld_fini=, stack_end=0x7ffc8ba27718) at ../csu/libc-start.c:266 #5 0x0000000000500f41 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 114076 [Inferior 1 (process 114076) detached] Exception in thread Thread-4:Traceback (most recent call last): File "/opt/jython/Lib/threading.py", line 222, in _Thread__bootstrap self.run() File "/opt/jython/Lib/threading.py", line 213, in run self._target(*self._args, **self._kwargs) File "pytests/magma/magma_base.py", line 893, in crash self.assertFalse(result, "CRASH | CRITICAL | WARN messages " File "/opt/jython/Lib/unittest/case.py", line 419, in assertFalse raise self.failureException(msg) ERROR AssertionError: CRASH | CRITICAL | WARN messages found in cb_logs 2021-04-21 23:57:45,621 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | 172.23.106.62 | kv | 3.2678910703 | 12294119424 | 10539868160 | 0 / 0 | 1029915 / 1077093 | 7.0.0-4987-enterprise | | 172.23.106.60 | kv | 4.24050632911 | 12294115328 | 10465202176 | 0 / 0 | 1048650 / 1114099 | 7.0.0-4987-enterprise | | 172.23.106.53 | kv | 18.7523677232 | 12294115328 | 10599714816 | 0 / 0 | 1029638 / 1040789 | 7.0.0-4987-enterprise | | 172.23.106.56 | kv | 2.86218635733 | 12294119424 | 11356778496 | 0 / 0 | 0 / 0 | 7.0.0-4987-enterprise | +---------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ 2021-04-21 23:57:45,994 | test | INFO | MainThread | [magma_base:tearDown:284] ## Active Resident Threshold of default is 100 ## 2021-04-21 23:59:34,490 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:427] 172.23.106.60 node pcap zip copied on client : True 2021-04-22 00:01:40,713 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:427] 172.23.106.53 node pcap zip copied on client : True 2021-04-22 00:04:03,595 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:427] 172.23.106.56 node pcap zip copied on client : True 2021-04-22 00:06:27,477 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:427] 172.23.106.62 node pcap zip copied on client : True 2021-04-22 00:06:29,032 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:726] 172.23.106.60: 1 core dump seen 2021-04-22 00:06:29,467 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:727] 172.23.106.60: Stack Trace of first crash - a7fe9847-4d28-460d-d38b4cb3-233444d1.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00000000007e2546 in PersistenceCallback::operator()(EPTransactionContext&, SingleThreadedRCPtr >, KVStore::FlushStateMutation) () #0 0x00000000007e2546 in PersistenceCallback::operator()(EPTransactionContext&, SingleThreadedRCPtr >, KVStore::FlushStateMutation) () No symbol table info available. #1 0x00000000007e2935 in EPTransactionContext::setCallback(SingleThreadedRCPtr > const&, KVStore::FlushStateMutation) () No symbol table info available. #2 0x00000000007f3351 in CouchKVStore::commitCallback(std::deque >&, kvstats_ctx&, couchstore_error_t) () No symbol table info available. #3 0x0000000000808052 in CouchKVStore::commit(VB::Commit&) () No symbol table info available. #4 0x00000000007c0abe in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () No symbol table info available. #5 0x00000000007c5f2c in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () No symbol table info available. #6 0x00000000007c621f in EPBucket::flushVBucket(Vbid) () No symbol table info available. #7 0x000000000066d850 in Flusher::flushVB() () No symbol table info available. #8 0x000000000066e319 in Flusher::step(GlobalTask*) () No symbol table info available. #9 0x0000000000670557 in GlobalTask::execute() () No symbol table info available. #10 0x000000000066b6a0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () No symbol table info available. #11 0x0000000000975560 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () No symbol table info available. #12 0x000000000095d23a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () No symbol table info available. #13 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () No symbol table info available. #14 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () No symbol table info available. #15 0x00007fc465e78d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 No locals. #16 0x00007fc46567cea5 in start_thread (arg=0x7fc3f1ffb700) at pthread_create.c:307 __res = pd = 0x7fc3f1ffb700 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140479555417856, -519509859033764609, 0, 8392704, 0, 140479555417856, 526278440538254591, 523154533403110655}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = #17 0x00007fc4653a58dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done. done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.17.so.debug...done. done. Loaded symbols for /lib64/librt.so.1 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 41 (Thread 0x7fd434fff700 (LWP 114077)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fd437a225a7 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd434fff700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (Thread 0x7fd43427e700 (LWP 114078)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000091eca5 in spdlog::details::thread_pool::process_next_msg_() () #2 0x000000000091f2c0 in spdlog::details::thread_pool::worker_loop_() () #3 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #4 0x00007fd435f01ea5 in start_thread (arg=0x7fd43427e700) at pthread_create.c:307 #5 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (Thread 0x7fd433a7d700 (LWP 114079)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000091e45b in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv () #2 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd433a7d700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (Thread 0x7fd43327c700 (LWP 114080)): #0 0x00007fd435c1fc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000055a2d1 in check_stdin_thread(void*) () #2 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd43327c700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (Thread 0x7fd432a7b700 (LWP 114081)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000089d1bb in worker_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd432a7b700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (Thread 0x7fd43227a700 (LWP 114082)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000089d1bb in worker_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd43227a700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (Thread 0x7fd431a79700 (LWP 114083)): #0 0x00007fd435c1fc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000894128 in master_thread () #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd431a79700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7fd431278700 (LWP 114084)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000058afc6 in ExternalAuthManagerThread::run() () #2 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #4 0x00007fd435f01ea5 in start_thread (arg=0x7fd431278700) at pthread_create.c:307 #5 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7fd42bfff700 (LWP 114085)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000005f92c7 in AuditImpl::consume_events() () #2 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #3 0x00007fd435f01ea5 in start_thread (arg=0x7fd42bfff700) at pthread_create.c:307 #4 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7fd42b7fe700 (LWP 114086)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42b7fe700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7fd42affd700 (LWP 114087)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42affd700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7fd42a7fc700 (LWP 114088)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd42a7fc700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7fd429ffb700 (LWP 114089)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd429ffb700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7fd4297fa700 (LWP 114090)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd4297fa700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7fd428ff9700 (LWP 114091)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd428ff9700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7fd423fff700 (LWP 114092)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000055a85b in worker_libevent(void*) () #4 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd435f01ea5 in start_thread (arg=0x7fd423fff700) at pthread_create.c:307 #6 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 0x7fd4237fe700 (LWP 114093)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4237fe700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7fd422ffd700 (LWP 114094)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd422ffd700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7fd4227fc700 (LWP 114095)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4227fc700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7fd421ffb700 (LWP 114096)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd421ffb700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7fd4217fa700 (LWP 114097)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4217fa700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7fd420ff9700 (LWP 114098)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd420ff9700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7fd4207f8700 (LWP 114099)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd4366f88bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000059def1 in Executor::run() () #4 0x00007fd43853e143 in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #5 0x00007fd43852dbe9 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #6 0x00007fd435f01ea5 in start_thread (arg=0x7fd4207f8700) at pthread_create.c:307 #7 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7fd41e3f6700 (LWP 114101)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fd437a21fa2 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007fd435f01ea5 in start_thread (arg=0x7fd41e3f6700) at pthread_create.c:307 #3 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7fd41dbf5700 (LWP 114102)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41dbf5700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7fd41d3f4700 (LWP 114103)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41d3f4700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7fd41cbf3700 (LWP 114104)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd41cbf3700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7fd3db3ff700 (LWP 114105)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3db3ff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7fd3d93d9700 (LWP 114106)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3d93d9700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7fd3d8bd8700 (LWP 114107)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3d8bd8700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7fd3c7fff700 (LWP 114108)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c7fff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7fd3c77fe700 (LWP 114109)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c77fe700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7fd3c6ffd700 (LWP 114110)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c6ffd700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fd3c67fc700 (LWP 114111)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c67fc700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fd3c5ffb700 (LWP 114112)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c5ffb700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7fd3c57fa700 (LWP 114113)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c57fa700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7fd3c4ff9700 (LWP 114114)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000964f6d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3c4ff9700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7fd3a3fff700 (LWP 114115)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a3fff700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7fd3a37fe700 (LWP 114116)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000957c3d in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x000000000095f02a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x00000000009650a7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x00000000009655d6 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000968d30 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x000000000095d1c7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000669c2d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a37fe700) at pthread_create.c:307 #11 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7fd3a2ffd700 (LWP 114118)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000097d8d7 in folly::EventBase::loopBody(int, bool) () #4 0x000000000097dda6 in folly::EventBase::loop() () #5 0x000000000097f766 in folly::EventBase::loopForever() () #6 0x000000000096cd89 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000978519 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x00007fd4366fdd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #9 0x00007fd435f01ea5 in start_thread (arg=0x7fd3a2ffd700) at pthread_create.c:307 #10 0x00007fd435c2a8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7fd4399d5e80 (LWP 114076)): #0 0x00007fd435c2aeb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd4377e93c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007fd4377e0376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000050adb6 in memcached_main(int, char**) () #4 0x00007fd435b4e555 in __libc_start_main (main=0x4e3fa0
, argc=3, argv=0x7ffc8ba27728, init=, fini=, rtld_fini=, stack_end=0x7ffc8ba27718) at ../csu/libc-start.c:266 #5 0x0000000000500f41 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 114076 [Inferior 1 (process 114076) detached] 2021-04-22 00:06:31,839 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1085] Running cbcollect on node ns_1@172.23.106.53,ns_1@172.23.106.56,ns_1@172.23.106.60,ns_1@172.23.106.62 2021-04-22 00:06:41,884 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1088] ns_1@172.23.106.53,ns_1@172.23.106.56,ns_1@172.23.106.60,ns_1@172.23.106.62 - cbcollect status: True 2021-04-22 00:06:41,885 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1093] Polling active_tasks to check cbcollect status 2021-04-22 00:11:22,365 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.106.53: Copying cbcollect ZIP file to Client 2021-04-22 00:11:24,381 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.106.53 node cb collect zip copied on client : True 2021-04-22 00:11:24,382 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.106.56: Copying cbcollect ZIP file to Client 2021-04-22 00:11:25,513 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.106.56 node cb collect zip copied on client : True 2021-04-22 00:11:25,513 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.106.60: Copying cbcollect ZIP file to Client 2021-04-22 00:11:27,344 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.106.60 node cb collect zip copied on client : True 2021-04-22 00:11:27,344 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.106.62: Copying cbcollect ZIP file to Client 2021-04-22 00:11:28,414 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1141] 172.23.106.62 node cb collect zip copied on client : True 2021-04-22 00:11:28,426 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:489] Skipping get_trace !! 2021-04-22 00:11:35,003 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 3.18600010872 seconds to complete 2021-04-22 00:11:35,052 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 3.18600010872 seconds. Reason: Wait after rebalance complete During the test, Remote Connections: 125, Disconnections: 121 SDK Connections: 84, Disconnections: 63 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! summary so far suite magma.magma_crash_recovery.MagmaCrashTests , pass 2 , fail 2 failures so far... magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-couchstore-vset00-00-crash_recovery_graceful_with_durability_7.0_P0/logs/testrunner-21-Apr-21_21-12-16/test_4 ERROR ====================================================================== ERROR: test_crash_during_recovery (magma.magma_crash_recovery.MagmaCrashTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/magma/magma_crash_recovery.py", line 137, in test_crash_during_recovery self.task_manager.get_task_result(task) File "lib/Jython_tasks/task_manager.py", line 28, in get_task_result result = future.get() CancellationException: java.util.concurrent.CancellationException ====================================================================== ERROR: test_crash_during_recovery (magma.magma_crash_recovery.MagmaCrashTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/magma/magma_crash_recovery.py", line 33, in tearDown super(MagmaCrashTests, self).tearDown() File "pytests/magma/magma_base.py", line 290, in tearDown super(MagmaBaseTest, self).tearDown() File "pytests/basetestcase.py", line 449, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 2420.424s