2021-05-12 05:54:15,292 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= MagmaCrashTests setup started for test #1 test_crash_during_ops ========= 2021-05-12 05:54:15,898 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #1 test_crash_during_ops ========= 2021-05-12 05:54:16,630 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:321] Initializing cluster 2021-05-12 05:54:19,877 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:22,536 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:24,586 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:26,607 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:28,628 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:30,674 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.202:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:36,756 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:38,776 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:40,857 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:43,207 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:45,229 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:47,428 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:52,943 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:54,961 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:56,980 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:54:58,996 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:01,026 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:03,062 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.203:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:08,884 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:10,905 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:12,947 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:15,000 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:17,022 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:55:19,108 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.204:8091/nodes/self error [Errno 111] Connection refused 2021-05-12 05:56:07,035 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-12 05:56:19,134 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.106.202, 172.23.106.201, 172.23.106.203, 172.23.106.204 initialized 2021-05-12 05:56:19,151 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #1 test_crash_during_ops ========= 2021-05-12 05:56:43,842 | test | INFO | MainProcess | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.106.202 | kv | 7.0.0-5153-enterprise | 0.915016294811 | Cluster node | | 172.23.106.201 | ['kv'] | | | <--- IN --- | | 172.23.106.203 | ['kv'] | | | <--- IN --- | | 172.23.106.204 | ['kv'] | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-12 05:56:43,858 | test | INFO | MainProcess | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-12 05:56:53,938 | test | INFO | MainProcess | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-12 05:56:53,959 | test | INFO | MainProcess | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 10.1180000305 sec 2021-05-12 05:56:59,401 | test | INFO | MainProcess | MainThread | [magma_base:setUp:96] Scopes list is ['_default'] 2021-05-12 05:57:03,203 | test | INFO | MainProcess | MainThread | [magma_base:get_disk_usage:715] Disk usage stats for bucekt default is below 2021-05-12 05:57:03,207 | test | INFO | MainProcess | MainThread | [magma_base:get_disk_usage:716] Total Disk usage for kvstore is 0MB 2021-05-12 05:57:03,625 | test | INFO | MainProcess | MainThread | [magma_base:get_bucket_dgm:667] Active Resident Threshold of default is 100 2021-05-12 05:57:03,627 | test | INFO | MainProcess | MainThread | [magma_base:setUp:153] disk usage before loading 0 2021-05-12 05:57:41,779 | test | INFO | MainProcess | pool-2-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................****X | 12782.600 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-12 05:58:06,002 | test | INFO | MainProcess | MainThread | [magma_base:get_disk_usage:715] Disk usage stats for bucekt default is below 2021-05-12 05:58:06,003 | test | INFO | MainProcess | MainThread | [magma_base:get_disk_usage:716] Total Disk usage for kvstore is 0MB 2021-05-12 05:58:06,483 | test | INFO | MainProcess | MainThread | [magma_base:get_bucket_dgm:667] Active Resident Threshold of default is 100 2021-05-12 05:58:06,484 | test | INFO | MainProcess | MainThread | [magma_base:initial_load:238] For bucket default disk usage after initial creation is 0MB 2021-05-12 05:58:06,484 | test | INFO | MainProcess | MainThread | [magma_base:setUp:200] ==========Finished magma base setup======== 2021-05-12 05:58:06,486 | test | INFO | MainProcess | MainThread | [magma_crash_recovery:test_crash_during_ops:88] ====test_crash_during_ops starts==== 2021-05-12 05:58:08,869 | test | INFO | MainProcess | Thread-1 | [common_lib:sleep:22] Sleep 55 seconds. Reason: Iteration:1 waiting for 55 sec to kill memcached on all nodes 2021-05-12 05:59:40,006 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 05:59:49,683 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 05:59:52,706 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 05:59:55,808 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 05:59:59,209 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:03,404 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:06,539 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:09,648 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:10,214 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:00:12,835 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:15,862 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:20,240 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:00:21,509 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:00:37,109 | test | INFO | MainProcess | Thread-1 | [common_lib:sleep:22] Sleep 31 seconds. Reason: Iteration:2 waiting for 31 sec to kill memcached on all nodes 2021-05-12 06:01:44,161 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:01:54,980 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:01:58,513 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:01,530 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:04,575 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:12,566 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:14,117 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:02:15,588 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:18,605 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:19,736 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:02:21,953 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:25,178 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:02:37,862 | test | INFO | MainProcess | Thread-1 | [common_lib:sleep:22] Sleep 30 seconds. Reason: Iteration:3 waiting for 30 sec to kill memcached on all nodes 2021-05-12 06:03:36,226 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:03:49,431 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:03:51,563 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:03:59,907 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:03,015 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:06,028 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:07,388 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:04:09,045 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:12,523 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:12,608 | infra | ERROR | MainProcess | Thread-1 | [remote_util:log_command_output:3079] Redirecting to /bin/systemctl restart couchbase-server.service 2021-05-12 06:04:15,543 | infra | ERROR | MainProcess | pool-2-thread-21 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.202:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 111] Connection refused 2021-05-12 06:04:25,191 | test | INFO | MainProcess | Thread-1 | [basetestcase:check_if_new_messages:704] Comparing timestamps: Log's latest timestamp: 2021-05-12 06:03:06, Test's start timestamp is 2021-05-12 05:54:15.289000 2021-05-12 06:04:25,194 | test | INFO | MainProcess | Thread-1 | [basetestcase:check_if_new_messages:704] Comparing timestamps: Log's latest timestamp: 2021-05-12 06:03:06, Test's start timestamp is 2021-05-12 05:54:15.289000 2021-05-12 06:04:25,194 | test | INFO | MainProcess | Thread-1 | [basetestcase:check_coredump_exist:808] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-05-12 06:04:25,197 | test | CRITICAL | MainProcess | Thread-1 | [basetestcase:check_coredump_exist:810] 172.23.106.203: Found ' ERROR ' logs - ['2021-05-12T06:02:59.405617-07:00 ERROR 76: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'cid:0x0:test_docs-26008268\' seqno:14648 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-6195935 and prepare seqno: 14652, highSeqno:14937, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af11080 qi:[key:\'cid:0x0:test_docs-6195935\' seqno:14652 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":38744} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":34,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-6195935","keylen":18,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:01.218538-07:00 ERROR 2081: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'cid:0x0:test_docs-26008268\' seqno:14648 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-85393274 and prepare seqno: 14653, highSeqno:14938, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af110b0 qi:[key:\'cid:0x0:test_docs-85393274\' seqno:14653 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":57185} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-85393274","keylen":19,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:06.923068-07:00 ERROR 2079: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:154 Pending resolution for \'SW @0x7f45a87bb690 qi:[key:\'cid:0x0:test_docs-85392583\' seqno:14546 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-60637898 and prepare seqno: 14554, highSeqno:15078, HPS:14855, HCS:12457, foundKeyWithSeqno:\'SW @0x7f452b502ac0 qi:[key:\'cid:0x0:test_docs-60637898\' seqno:14554 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'15011{Disk}\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":55175} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-60637898","keylen":19,"magic":"ClientRequest","opaque":71,"opcode":"DCP_COMMIT","vbucket":154},"refcount":1}]\n'] 2021-05-12 06:04:26,875 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | 172.23.106.202 | kv | 25.664527956 | 12294172672 | 10020073472 | 0 / 0 | 0 / 0 | 7.0.0-5153-enterprise | | 172.23.106.203 | kv | 8.78220140515 | 12294176768 | 10559987712 | 0 / 0 | 1406591 / 1305136 | 7.0.0-5153-enterprise | | 172.23.106.204 | kv | 11.3504992118 | 12294176768 | 10371198976 | 0 / 0 | 1685691 / 1589170 | 7.0.0-5153-enterprise | | 172.23.106.201 | kv | 20.5348004094 | 12294172672 | 10260774912 | 0 / 0 | 1403775 / 1674524 | 7.0.0-5153-enterprise | +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ 2021-05-12 06:04:27,305 | test | INFO | MainProcess | MainThread | [magma_base:tearDown:285] ## Active Resident Threshold of default is 100 ## 2021-05-12 06:04:33,707 | test | INFO | MainProcess | MainThread | [basetestcase:check_if_new_messages:704] Comparing timestamps: Log's latest timestamp: 2021-05-12 06:03:06, Test's start timestamp is 2021-05-12 05:54:15.289000 2021-05-12 06:04:33,709 | test | INFO | MainProcess | MainThread | [basetestcase:check_if_new_messages:704] Comparing timestamps: Log's latest timestamp: 2021-05-12 06:03:06, Test's start timestamp is 2021-05-12 05:54:15.289000 2021-05-12 06:04:33,710 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:808] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-05-12 06:04:33,710 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:810] 172.23.106.203: Found ' ERROR ' logs - ['2021-05-12T06:02:59.405617-07:00 ERROR 76: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'cid:0x0:test_docs-26008268\' seqno:14648 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-6195935 and prepare seqno: 14652, highSeqno:14937, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af11080 qi:[key:\'cid:0x0:test_docs-6195935\' seqno:14652 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":38744} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":34,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-6195935","keylen":18,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:01.218538-07:00 ERROR 2081: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'cid:0x0:test_docs-26008268\' seqno:14648 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-85393274 and prepare seqno: 14653, highSeqno:14938, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af110b0 qi:[key:\'cid:0x0:test_docs-85393274\' seqno:14653 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":57185} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-85393274","keylen":19,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:06.923068-07:00 ERROR 2079: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:154 Pending resolution for \'SW @0x7f45a87bb690 qi:[key:\'cid:0x0:test_docs-85392583\' seqno:14546 reqs:{majority, timeout=infinite}] status:Pending\', but received unexpected commit for key cid:0x0:test_docs-60637898 and prepare seqno: 14554, highSeqno:15078, HPS:14855, HCS:12457, foundKeyWithSeqno:\'SW @0x7f452b502ac0 qi:[key:\'cid:0x0:test_docs-60637898\' seqno:14554 reqs:{majority, timeout=infinite}] status:Pending\' lastReceivedSnapEnd:\'15011{Disk}\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":55175} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, @ns_server) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":".test_docs-60637898","keylen":19,"magic":"ClientRequest","opaque":71,"opcode":"DCP_COMMIT","vbucket":154},"refcount":1}]\n'] 2021-05-12 06:04:34,536 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.106.201,ns_1@172.23.106.202,ns_1@172.23.106.203,ns_1@172.23.106.204 2021-05-12 06:04:44,569 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.106.201,ns_1@172.23.106.202,ns_1@172.23.106.203,ns_1@172.23.106.204 - cbcollect status: True 2021-05-12 06:04:44,571 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-05-12 06:07:25,069 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.201: Copying cbcollect ZIP file to Client 2021-05-12 06:07:26,230 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.202: Copying cbcollect ZIP file to Client 2021-05-12 06:07:27,358 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.203: Copying cbcollect ZIP file to Client 2021-05-12 06:07:27,993 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.204: Copying cbcollect ZIP file to Client 2021-05-12 06:07:30,644 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:492] Skipping get_trace !! 2021-05-12 06:07:54,280 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 12.5719997883 seconds to complete 2021-05-12 06:07:54,305 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete