2021-09-30 14:06:33,346 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1846] 172.23.105.105 - Node not part of cluster inactiveFailed 2021-09-30 14:06:33,381 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1846] 172.23.105.105 - Node not part of cluster inactiveFailed 2021-09-30 14:06:39,986 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4849] Performing scope/collection specific operations 2021-09-30 14:06:40,197 | test | INFO | MainProcess | pool-6-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.118 | kv | 7.1.0-1386-enterprise | 29.4042607475 | Cluster node | | 172.23.105.105 | kv | 7.1.0-1386-enterprise | 0.526711813394 | --- OUT ---> | | 172.23.105.112 | kv | 7.1.0-1386-enterprise | 25.1047087194 | Cluster node | | 172.23.105.109 | kv | 7.1.0-1386-enterprise | 26.2744103475 | Cluster node | | 172.23.104.76 | kv | 7.1.0-1386-enterprise | 29.2431192661 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-09-30 14:06:40,214 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 0.0 2021-09-30 14:06:49,454 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4939] Done Performing scope/collection specific operations 2021-09-30 14:06:50,267 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 0.43 2021-09-30 14:07:00,316 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 0.54 2021-09-30 14:07:10,355 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 0.93 2021-09-30 14:07:20,398 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.14 2021-09-30 14:07:30,446 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.31 2021-09-30 14:07:40,486 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.58 2021-09-30 14:07:50,529 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.58 2021-09-30 14:08:00,572 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:08:10,619 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:08:20,657 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:08:30,696 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:08:40,734 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:08:50,769 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:00,812 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:10,849 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:20,885 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:30,934 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:40,970 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:09:51,010 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:10:01,048 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.63 2021-09-30 14:10:11,092 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.68 2021-09-30 14:10:21,128 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:10:31,167 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:10:41,207 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:10:51,250 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:11:01,286 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:11:11,326 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.74 2021-09-30 14:11:21,367 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.85 2021-09-30 14:11:31,410 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.85 2021-09-30 14:11:41,447 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.85 2021-09-30 14:11:51,486 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.85 2021-09-30 14:12:01,529 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 1.96 2021-09-30 14:12:05,055 | test | ERROR | MainProcess | pool-6-thread-16 | [task:execute_tasks:3895] Failed to load 1654 docs from 25538 to 35545 of thread_name LoadDocs_MutateDocsFromSpecTask_1633036012.69_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-59-746000_Vh%ahcumKgxnIxGrY0lTorP_oY3ZL0%ossxoQHUWbtnSLX%QE82_FydeywMYhQq4u3wYtxNf1xipZAfL0b0BF%dtWJIKPFpdl5ki4UzEDCwW%Yy4_i2U8dzhh08Cv7LI%9IZFmqFfB-AXHzKgpRL58o%k_e-59-746000_ttl=0_read_MAJORITY_25538_35545_1633036012.85 2021-09-30 14:12:05,056 | test | ERROR | MainProcess | pool-6-thread-16 | [task:execute_tasks:3895] Failed to load 591 docs from 25538 to 35545 of thread_name LoadDocs_MutateDocsFromSpecTask_1633036012.69_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-59-759000_L%Cd0hgipGn1%_dkX-J%b8Z4X0HOCxZv7yBocDkZkcK0c40FX%YmTPmAiFuKi450gOk1NuRB8p7iLb3RSY82WABNrHsZHZP7-59-759000_ttl=0_read_MAJORITY_25538_35545_1633036012.85 2021-09-30 14:12:11,572 | test | INFO | MainProcess | pool-6-thread-30 | [task:check:357] Rebalance - status: running, progress: 2.23 2021-09-30 14:12:21,604 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:_rebalance_status_and_progress:1638] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a1caf36a1278db2a4b8f496a59b06e1a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=6f3240e24c0478b0b5759c34e095f59a', u'status': u'notRunning'} - rebalance failed 2021-09-30 14:12:21,631 | test | INFO | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2784] Latest logs from UI on 172.23.104.76: 2021-09-30 14:12:21,631 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.104.76', u'tstamp': 1633036337232L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:12:17.232Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.24599.22>,\n {{wait_seqno_persisted_failed,"bucket1",747,\n 795,\n [{\'ns_1@172.23.105.118\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-bucket1\',\n \'ns_1@172.23.105.118\'},\n {if_rebalance,<0.14229.22>,\n {wait_seqno_persisted,747,795}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,506}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 5fbc0db5c38c27dcfe1e079104d55553'} 2021-09-30 14:12:21,631 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.105.118', u'tstamp': 1633036337216L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:12:17.216Z', u'text': u"Service 'memcached' exited with status 139. Restarting. Messages:\n #14 /opt/couchbase/bin/memcached(_ZN5magma5Magma14CompactKVStoreEtRKNS_5SliceES3_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS6_EEtEE+0x6d) [0x400000+0x4de0dd]\n #15 /opt/couchbase/bin/memcached() [0x400000+0x45629e]\n #16 /opt/couchbase/bin/memcached() [0x400000+0x44c5ea]\n #17 /opt/couchbase/bin/memcached() [0x400000+0x44da06]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x3e9ada]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x3eb0f1]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x3089c6]\n #21 /opt/couchbase/bin/memcached() [0x400000+0x61cf32]\n #22 /opt/couchbase/bin/memcached() [0x400000+0x61a055]\n #23 /opt/couchbase/bin/memcached() [0x400000+0x7649a0]\n #24 /opt/couchbase/bin/memcached() [0x400000+0x74c75a]\n #25 /opt/couchbase/bin/memcached() [0x400000+0x767959]\n #26 /opt/couchbase/bin/memcached() [0x400000+0x619ce4]\n #27 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb0467f7000+0xcdd40]\n #28 /lib64/libpthread.so.0() [0x7fb0486e1000+0x7ea5]\n #29 /lib64/libc.so.6(clone+0x6d) [0x7fb045f0f000+0xfe8dd]"} 2021-09-30 14:12:21,631 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.104.76', u'tstamp': 1633036337191L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:12:17.191Z', u'text': u'Worker <0.24993.22> (for action {move,{747,\n [\'ns_1@172.23.105.112\',\n \'ns_1@172.23.105.109\',undefined],\n [\'ns_1@172.23.105.109\',\n \'ns_1@172.23.105.112\',\n \'ns_1@172.23.105.118\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.24599.22>,\n {{wait_seqno_persisted_failed,\n "bucket1",\n 747,795,\n [{\'ns_1@172.23.105.118\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-bucket1\',\n \'ns_1@172.23.105.118\'},\n {if_rebalance,\n <0.14229.22>,\n {wait_seqno_persisted,\n 747,\n 795}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 506}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-09-30 14:12:21,631 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.118', u'tstamp': 1633036337187L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:12:17.187Z', u'text': u"Control connection to memcached on 'ns_1@172.23.105.118' disconnected. Check logs for details."} 2021-09-30 14:12:21,632 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.104.76', u'tstamp': 1633036002120L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:06:42.120Z', u'text': u'Bucket "bucket1" rebalance does not seem to be swap rebalance'} 2021-09-30 14:12:21,632 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.104.76', u'tstamp': 1633036000849L, u'shortText': u'message', u'serverTime': u'2021-09-30T14:06:40.849Z', u'text': u'Started rebalancing bucket bucket1'} 2021-09-30 14:12:21,632 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.105.112', u'tstamp': 1633036000842L, u'shortText': u'node down', u'serverTime': u'2021-09-30T14:06:40.842Z', u'text': u"Node 'ns_1@172.23.105.112' saw that node 'ns_1@172.23.105.105' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-09-30 14:12:21,632 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.105.109', u'tstamp': 1633036000842L, u'shortText': u'node down', u'serverTime': u'2021-09-30T14:06:40.842Z', u'text': u"Node 'ns_1@172.23.105.109' saw that node 'ns_1@172.23.105.105' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-09-30 14:12:21,632 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.105.118', u'tstamp': 1633036000841L, u'shortText': u'node down', u'serverTime': u'2021-09-30T14:06:40.841Z', u'text': u"Node 'ns_1@172.23.105.118' saw that node 'ns_1@172.23.105.105' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-09-30 14:12:21,634 | test | ERROR | MainProcess | pool-6-thread-30 | [rest_client:print_UI_logs:2786] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.104.76', u'tstamp': 1633036000840L, u'shortText': u'node down', u'serverTime': u'2021-09-30T14:06:40.840Z', u'text': u"Node 'ns_1@172.23.104.76' saw that node 'ns_1@172.23.105.105' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-09-30 14:12:21,634 | test | ERROR | MainProcess | pool-6-thread-30 | [task:call:269] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a1caf36a1278db2a4b8f496a59b06e1a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=6f3240e24c0478b0b5759c34e095f59a', u'status': u'notRunning'} - rebalance failed 2021-09-30 14:12:22,644 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 104114 | 7.81 GiB | 258.57 MiB | 249.32 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 80202 | 3.91 GiB | 532.36 MiB | 604.62 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 11231126 | 1.00 GiB | 897.49 MiB | 6.14 GiB | 1.79634565261 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2021-09-30 14:12:23,342 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-09-30 14:12:23,342 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-09-30 14:12:34,940 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-09-30 14:12:34,940 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-09-30 14:12:44,434 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 1% 2021-09-30 14:12:44,434 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 0% 2021-09-30 14:13:08,657 | test | ERROR | MainProcess | pool-6-thread-16 | [task:execute_tasks:3895] Failed to load 4 docs from 75574 to 85581 of thread_name LoadDocs_MutateDocsFromSpecTask_1633036012.69_default_gYtdRKkEHiGb0N0b-59-754000_aO5pBxcncBSS4D9_TeKu-rdsXAaPyBP%AN0BoH5TtuDZL2W9AQkOMT5bRVTJsmquxpUzE0qB3DLpfYmN7zVBQTTN7tjS%Xxj-59-754000_ttl=0_create_MAJORITY_75574_85581_1633036012.85 2021-09-30 14:13:13,661 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:71] Pool did not terminate 2021-09-30 14:13:15,663 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:71] Pool did not terminate 2021-09-30 14:13:22,984 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-09-30 14:12:17, Test's start timestamp is 2021-09-30 13:05:44.469000 2021-09-30 14:13:22,986 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-09-30 14:12:17, Test's start timestamp is 2021-09-30 13:05:44.469000 2021-09-30 14:13:22,986 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:904] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-09-30 14:13:22,987 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:906] 172.23.105.118: Found ' CRITICAL ' logs - ['[ns_server:info,2021-09-30T14:12:17.215-07:00,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1386). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/08f3ed68-8449-41fa-bb8b799f-87c1e633.dmp before terminating.\n', '[ns_server:info,2021-09-30T14:12:17.468-07:00,babysitter_of_ns_1@cb.local:<0.1358.0>:ns_port_server:log:221]memcached<0.1358.0>: 2021-09-30T14:12:17.268313-07:00 CRITICAL Detected previous crash\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268388-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1386). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/08f3ed68-8449-41fa-bb8b799f-87c1e633.dmp before terminating.\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268399-07:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268402-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x69d858]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268404-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6ed8ca]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268406-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6edc08]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268407-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fb0486e1000+0xf630]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268409-07:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x538600]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268410-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x52f6af]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268434-07:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x5308c4]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268435-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x531236]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268437-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x531a0d]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268466-07:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x531f48]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268491-07:00 CRITICAL #10 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl11syncKVStoreEtb+0x215) [0x400000+0x4dd5c5]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268494-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x4dd740]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268495-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x4e40ae]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268507-07:00 CRITICAL #13 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtRKNS_5SliceES4_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS7_EEtEE+0x29e) [0x400000+0x4de00e]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268510-07:00 CRITICAL #14 /opt/couchbase/bin/memcached(_ZN5magma5Magma14CompactKVStoreEtRKNS_5SliceES3_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS6_EEtEE+0x6d) [0x400000+0x4de0dd]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268513-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x45629e]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268515-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x44c5ea]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268539-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x44da06]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268541-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x3e9ada]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268542-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x3eb0f1]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268544-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x3089c6]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268547-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x61cf32]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268548-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x61a055]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268549-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x7649a0]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268594-07:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x74c75a]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268632-07:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x767959]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268634-07:00 CRITICAL #26 /opt/couchbase/bin/memcached() [0x400000+0x619ce4]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268652-07:00 CRITICAL #27 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb0467f7000+0xcdd40]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268654-07:00 CRITICAL #28 /lib64/libpthread.so.0() [0x7fb0486e1000+0x7ea5]\n', 'memcached<0.1358.0>: 2021-09-30T14:12:17.268655-07:00 CRITICAL #29 /lib64/libc.so.6(clone+0x6d) [0x7fb045f0f000+0xfe8dd]\n'] 2021-09-30 14:13:24,533 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1134] Running cbcollect on node ns_1@172.23.104.76,ns_1@172.23.105.109,ns_1@172.23.105.112,ns_1@172.23.105.118 2021-09-30 14:13:34,562 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1137] ns_1@172.23.104.76,ns_1@172.23.105.109,ns_1@172.23.105.112,ns_1@172.23.105.118 - cbcollect status: True 2021-09-30 14:13:34,562 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1142] Polling active_tasks to check cbcollect status 2021-09-30 14:20:55,608 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.76: Copying cbcollect ZIP file to Client 2021-09-30 14:20:57,224 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.105.109: Copying cbcollect ZIP file to Client 2021-09-30 14:20:58,490 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.105.112: Copying cbcollect ZIP file to Client 2021-09-30 14:20:59,753 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.105.118: Copying cbcollect ZIP file to Client 2021-09-30 14:21:00,999 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:567] Skipping get_trace !! 2021-09-30 14:21:32,859 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.104.76:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-09-30 14:21:33,434 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-09-30 14:21:33,434 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-09-30 14:22:03,592 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.104.76:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-09-30 14:22:04,157 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-09-30 14:22:04,157 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-09-30 14:22:34,285 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.104.76:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-09-30 14:22:34,792 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-09-30 14:22:34,792 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-09-30 14:22:46,127 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1574] Rebalance done. Taken 11.0789999962 seconds to complete 2021-09-30 14:22:46,140 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-09-30 14:22:51,151 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:22:51,155 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:22:56,177 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:22:56,181 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:23:01,204 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:23:01,207 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:23:06,217 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:312] Node 172.23.105.118:8091 is in a broken state! 2021-09-30 14:23:06,217 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:cleanup_cluster:201] Can't create rest connection after rebalance out for ejected nodes, will retry after 10 seconds according to MB-8430: Unable to reach the host @ 172.23.105.118 2021-09-30 14:23:06,230 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: MB-8430 2021-09-30 14:23:16,240 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:23:16,243 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:23:21,263 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:23:21,266 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:23:26,286 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] GET http://172.23.105.118:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-09-30 14:23:26,289 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:307] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-09-30 14:23:31,301 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:312] Node 172.23.105.118:8091 is in a broken state! 2021-09-30 14:23:31,302 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:579] Killing memcached due to Unable to reach the host @ 172.23.105.118