./testrunner -i /tmp/testexec.9702.ini sdk_timeout=60,rerun=False,get-cbcollect-info=False,upgrade_version=7.0.0-4023 -t rebalance_new.rebalance_out.RebalanceOutTests.rebalance_out_with_ops,nodes_init=3,nodes_out=1,replicas=0,num_items=20000,doc_ops=create:update:delete,durability=MAJORITY,sdk_client_pool=True Test Input params: {'cluster_name': 'testexec.9702', 'ini': '/tmp/testexec.9702.ini', 'get-cbcollect-info': 'False', 'replicas': '0', 'durability': 'MAJORITY', 'conf_file': 'conf/sanity/durability.conf', 'upgrade_version': '7.0.0-4023', 'sdk_client_pool': 'True', 'spec': 'durability', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos8-p0-os_certify-vset00-00-durability/logs/testrunner-20-Dec-14_17-02-07/test_10', 'nodes_init': '3', 'doc_ops': 'create:update:delete', 'nodes_out': '1', 'num_items': '20000', 'sdk_timeout': '60', 'case_number': 10} rebalance_out_with_ops (rebalance_new.rebalance_out.RebalanceOutTests) ... 2020-12-14 17:26:21,575 | test | INFO | MainThread | [basetestcase:log_setup_status:553] ========= BaseTestCase setup started for test #10 rebalance_out_with_ops ========= 2020-12-14 17:26:21,915 | test | INFO | MainThread | [basetestcase:log_setup_status:553] ========= BaseTestCase setup started for test #10 rebalance_out_with_ops ========= 2020-12-14 17:26:22,206 | test | INFO | MainThread | [basetestcase:setUp:329] Initializing cluster 2020-12-14 17:26:25,062 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:597] RAM quota was defined less than 100 MB: 2020-12-14 17:26:38,717 | test | INFO | MainThread | [basetestcase:setUp:351] Couchbase Cluster: default, Nodes: 172.23.121.25, 172.23.120.29, 172.23.120.255, 172.23.121.145, 172.23.121.148, 172.23.121.171 initialized 2020-12-14 17:26:38,730 | test | INFO | MainThread | [basetestcase:log_setup_status:553] ========= BaseTestCase setup finished for test #10 rebalance_out_with_ops ========= 2020-12-14 17:26:38,730 | test | WARNING | MainThread | [basetestcase:_BaseTestCase__log:550] Exception during REST log_client_error: type object 'datetime' has no attribute 'datetime' 2020-12-14 17:26:43,282 | test | INFO | pool-11-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 172.23.121.25 | kv | Cluster node | | 172.23.120.29 | None | <--- IN --- | | 172.23.120.255 | None | <--- IN --- | +----------------+----------+--------------+ 2020-12-14 17:26:43,292 | test | INFO | pool-11-thread-7 | [task:check:320] Rebalance - status: running, progress: 0 2020-12-14 17:26:53,334 | test | INFO | pool-11-thread-7 | [task:check:320] Rebalance - status: none, progress: 100 2020-12-14 17:26:53,346 | test | INFO | pool-11-thread-7 | [task:check:377] Rebalance completed with progress: 100% in 10.0649998188 sec 2020-12-14 17:26:56,980 | test | INFO | MainThread | [rebalance_base:setUp:116] Creating SDK clients for client_pool 2020-12-14 17:26:58,585 | test | INFO | MainThread | [rebalance_base:setUp:128] Verifying num_items counts after doc_ops 2020-12-14 17:26:59,648 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4371] Validating collection stats and item counts 2020-12-14 17:27:04,789 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.120.29 | kv | 3.05764411028 | 4353015808 | 3658510336 | 17825792 / 3758092288 | 0 / 0 | 7.0.0-4023-enterprise | | 172.23.120.255 | kv | 1.20240480962 | 4353015808 | 3690004480 | 56360960 / 3758092288 | 0 / 0 | 7.0.0-4023-enterprise | | 172.23.121.25 | kv | 1.32898696088 | 4353015808 | 3695616000 | 81526784 / 3758092288 | 0 / 0 | 7.0.0-4023-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2020-12-14 17:27:04,838 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | default | couchbase | 0 | none | 0 | 0 | 4643094528 | 19180504 | 1433621 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2020-12-14 17:27:04,839 | test | INFO | MainThread | [basetestcase:log_setup_status:553] ========= RebalanceBase setup complete for test #10 rebalance_out_with_ops ========= 2020-12-14 17:27:04,875 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 15 seconds. Reason: Wait for rebalance to start 2020-12-14 17:27:04,894 | test | INFO | pool-11-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 172.23.120.29 | kv | Cluster node | | 172.23.120.255 | [u'kv'] | --- OUT ---> | | 172.23.121.25 | kv | Cluster node | +----------------+----------+--------------+ 2020-12-14 17:27:04,963 | test | INFO | pool-11-thread-2 | [task:check:320] Rebalance - status: running, progress: 0 2020-12-14 17:27:10,053 | test | ERROR | pool-11-thread-2 | [rest_client:_rebalance_status_and_progress:1484] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2020-12-14 17:27:10,091 | test | INFO | pool-11-thread-2 | [rest_client:print_UI_logs:2594] Latest logs from UI on 172.23.121.25: 2020-12-14 17:27:10,092 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995625418L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:27:05.418Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.4969.0>,\n {{dcp_wait_for_data_move_failed,"default",\n 338,\'ns_1@172.23.120.255\',\n [\'ns_1@172.23.121.25\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,465}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = f979e3f27acecff3b0d4b1b952f971f7'} 2020-12-14 17:27:10,092 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995625383L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:27:05.383Z', u'text': u'Worker <0.4893.0> (for action {move,{338,\n [\'ns_1@172.23.120.255\'],\n [\'ns_1@172.23.121.25\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.4969.0>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 338,\n \'ns_1@172.23.120.255\',\n [\'ns_1@172.23.121.25\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 465}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2020-12-14 17:27:10,092 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995625212L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:27:05.212Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2020-12-14 17:27:10,092 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995624978L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:27:04.978Z', u'text': u'Started rebalancing bucket default'} 2020-12-14 17:27:10,092 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995624888L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:27:04.888Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.120.29','ns_1@172.23.121.25'], EjectNodes = ['ns_1@172.23.120.255'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f979e3f27acecff3b0d4b1b952f971f7"} 2020-12-14 17:27:10,094 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995614333L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:26:54.333Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.121.25\' in 0 seconds.'} 2020-12-14 17:27:10,094 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.29', u'tstamp': 1607995613830L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:26:53.830Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.120.29\' in 0 seconds.'} 2020-12-14 17:27:10,094 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.255', u'tstamp': 1607995613820L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:26:53.820Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.120.255\' in 0 seconds.'} 2020-12-14 17:27:10,094 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 12, u'module': u'menelaus_web', u'type': u'info', u'node': u'ns_1@172.23.121.25', u'tstamp': 1607995613697L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:26:53.697Z', u'text': u'Created bucket "default" of type: couchbase\n[{num_replicas,0},\n {replica_index,true},\n {ram_quota,1547698176},\n {durability_min_level,none},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {flush_enabled,false},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,off}]'} 2020-12-14 17:27:10,095 | test | ERROR | pool-11-thread-2 | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.120.255', u'tstamp': 1607995603337L, u'shortText': u'message', u'serverTime': u'2020-12-14T17:26:43.337Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2020-12-14 17:27:10,095 | test | ERROR | pool-11-thread-2 | [task:call:236] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed FAIL 2020-12-14 17:27:19,905 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.120.29 | kv | 9.47950716621 | 4353015808 | 3541356544 | 17825792 / 3758092288 | 9994 / 0 | 7.0.0-4023-enterprise | | 172.23.120.255 | kv | 14.2102615694 | 4353015808 | 3562987520 | 56360960 / 3758092288 | 10026 / 0 | 7.0.0-4023-enterprise | | 172.23.121.25 | kv | 9.94210923735 | 4353015808 | 3564367872 | 81526784 / 3758092288 | 9980 / 0 | 7.0.0-4023-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2020-12-14 17:27:20,242 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.121.25: Looking for crash / dump files 2020-12-14 17:27:20,292 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:20,377 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2020-12-14 17:27:20,467 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2020-12-14 17:27:20,549 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2020-12-14 17:27:20,621 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.121.25: No 'sanitizers.log.*' files found 2020-12-14 17:27:20,888 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.120.29: Looking for crash / dump files 2020-12-14 17:27:20,938 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:21,023 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2020-12-14 17:27:21,104 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2020-12-14 17:27:21,198 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.120.29: No 'sanitizers.log.*' files found 2020-12-14 17:27:21,473 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.120.255: Looking for crash / dump files 2020-12-14 17:27:21,523 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:21,607 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2020-12-14 17:27:21,665 | test | INFO | MainThread | [basetestcase:check_if_new_messages:695] Comparing timestamps: Log's latest timestamp: 2020-12-14 17:27:05, Test's start timestamp is 2020-12-14 17:26:21.575000 2020-12-14 17:27:21,665 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:793] 172.23.120.255: Found 'ERROR' logs - ['2020-12-14T17:27:05.358539-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358598-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358614-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358630-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358645-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358667-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358824-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358881-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358971-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358986-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359007-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359021-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359038-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359137-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359158-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359184-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.373581-08:00 ERROR 111: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.121.25:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":18,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.121.25:default, vb:338, state:in-memory\n'] 2020-12-14 17:27:21,683 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.120.255: No 'sanitizers.log.*' files found 2020-12-14 17:27:21,963 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.121.145: Looking for crash / dump files 2020-12-14 17:27:22,010 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:22,105 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.121.145: No 'sanitizers.log.*' files found 2020-12-14 17:27:22,385 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.121.148: Looking for crash / dump files 2020-12-14 17:27:22,434 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:22,516 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.121.148: No 'sanitizers.log.*' files found 2020-12-14 17:27:22,799 | test | INFO | MainThread | [basetestcase:check_coredump_exist:706] 172.23.121.171: Looking for crash / dump files 2020-12-14 17:27:22,848 | test | INFO | MainThread | [basetestcase:check_coredump_exist:761] Looking for CRITICAL|WARN messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-12-14 17:27:22,920 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:750] 172.23.121.171: No 'sanitizers.log.*' files found 2020-12-14 17:27:22,957 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1062] Running cbcollect on node ns_1@172.23.120.255,ns_1@172.23.120.29,ns_1@172.23.121.25 2020-12-14 17:27:32,976 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1065] ns_1@172.23.120.255,ns_1@172.23.120.29,ns_1@172.23.121.25 - cbcollect status: True 2020-12-14 17:27:32,977 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1070] Polling active_tasks to check cbcollect status 2020-12-14 17:29:13,226 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1093] ns_1@172.23.120.255: Copying cbcollect ZIP file to Client 2020-12-14 17:29:13,674 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] 172.23.120.255 node cb collect zip copied on client : True 2020-12-14 17:29:13,676 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1093] ns_1@172.23.120.29: Copying cbcollect ZIP file to Client 2020-12-14 17:29:14,115 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] 172.23.120.29 node cb collect zip copied on client : True 2020-12-14 17:29:14,117 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1093] ns_1@172.23.121.25: Copying cbcollect ZIP file to Client 2020-12-14 17:29:14,604 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] 172.23.121.25 node cb collect zip copied on client : True 2020-12-14 17:29:14,615 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:498] Skipping get_trace !! 2020-12-14 17:29:26,381 | test | INFO | MainThread | [rest_client:monitorRebalance:1438] Rebalance done. Taken 11.0709998608 seconds to complete 2020-12-14 17:29:26,392 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: rebalance_out_with_ops (rebalance_new.rebalance_out.RebalanceOutTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_out.py", line 14, in tearDown super(RebalanceOutTests, self).tearDown() File "pytests/rebalance_new/rebalance_base.py", line 190, in tearDown super(RebalanceBaseTest, self).tearDown() File "pytests/basetestcase.py", line 458, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ====================================================================== FAIL: rebalance_out_with_ops (rebalance_new.rebalance_out.RebalanceOutTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_out.py", line 106, in rebalance_out_with_ops self.fail("Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 218.985s FAILED (failures=1, errors=1) During the test, Remote Connections: 64, Disconnections: 64 SDK Connections: 1, Disconnections: 1