2020-05-29 16:27:11,496 | root | WARNING | MainProcess | MainThread | [data_helper::31] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially 2020-05-29 16:27:12,049 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #1 test_sub_doc_op_with_bucket_level_durability ========= 2020-05-29 16:27:13,118 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #1 test_sub_doc_op_with_bucket_level_durability ========= 2020-05-29 16:27:13,815 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:351] CLEANUP WAS SKIPPED 2020-05-29 16:27:13,818 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:266] Initializing cluster 2020-05-29 16:27:14,081 | test | ERROR | MainProcess | pool-2-thread-1 | [rest_client:_http_request:737] POST http://10.112.191.101:8091//node/controller/setupServices body: services=kv&hostname=10.112.191.101&password=password&user=Administrator headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password 2020-05-29 16:27:14,094 | test | WARNING | MainProcess | pool-2-thread-1 | [rest_client:init_node_services:833] This node is already provisioned with services, we do not consider this as failure for test case 2020-05-29 16:27:14,187 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:443] RAM quota was defined less than 100 MB: 2020-05-29 16:27:14,667 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:288] Couchbase Cluster: default, Nodes: 10.112.191.101, 10.112.191.102 initialized 2020-05-29 16:27:14,688 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup finished for test #1 test_sub_doc_op_with_bucket_level_durability ========= 2020-05-29 16:27:14,982 | test | INFO | MainProcess | MainThread | [basetestcase:sleep:514] Reason: None. Sleep for 5 secs ... 2020-05-29 16:27:20,130 | test | INFO | MainProcess | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 10.112.191.101 | kv | Cluster node | +----------------+----------+--------------+ 2020-05-29 16:27:20,160 | test | INFO | MainProcess | pool-2-thread-3 | [task:check:304] Rebalance - status: running, progress: 0 2020-05-29 16:27:35,354 | test | INFO | MainProcess | pool-2-thread-3 | [task:check:304] Rebalance - status: none, progress: 100 2020-05-29 16:27:35,381 | test | INFO | MainProcess | pool-2-thread-3 | [task:check:363] Rebalance completed with progress: 100% in 15.2510001659 sec 2020-05-29 16:27:35,704 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ | 10.112.191.101 | kv | 1.0101010101 | 1930829824 | 1395826688 | 1810432 / 1107292160 | 0 / 0 | 6.6.0-7687-enterprise | +----------------+----------+-----------------+------------+------------+----------------------+-------------------+-----------------------+ 2020-05-29 16:27:36,061 | test | INFO | MainProcess | MainThread | [durability_base:setUp:177] ===== BucketDurabilityBase setup complete ===== 2020-05-29 16:27:36,063 | test | INFO | MainProcess | MainThread | [bucket_level_durability:test_sub_doc_op_with_bucket_level_durability:260] Creating couchbase bucket with level 'MAJORITY' 2020-05-29 16:27:47,999 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in sync_write_committed_count stats. 2 != 1 2020-05-29 16:27:48,000 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in ops_get stats. 2 != 1 2020-05-29 16:27:48,000 | test | ERROR | MainProcess | MainThread | [basetestcase:log_failure:518] Cbstat vbucket-details verification failed 2020-05-29 16:27:50,082 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in sync_write_committed_count stats. 3 != 2 2020-05-29 16:27:50,082 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in ops_get stats. 4 != 2 2020-05-29 16:27:50,082 | test | ERROR | MainProcess | MainThread | [basetestcase:log_failure:518] Cbstat vbucket-details verification failed 2020-05-29 16:27:51,914 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in sync_write_committed_count stats. 4 != 3 2020-05-29 16:27:51,915 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in ops_get stats. 6 != 3 2020-05-29 16:27:51,915 | test | ERROR | MainProcess | MainThread | [basetestcase:log_failure:518] Cbstat vbucket-details verification failed 2020-05-29 16:27:53,849 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in sync_write_committed_count stats. 5 != 4 2020-05-29 16:27:53,851 | test | ERROR | MainProcess | MainThread | [durability_helper:verify_vbucket_details_stats:268] Mismatch in ops_get stats. 8 != 4 2020-05-29 16:27:53,851 | test | ERROR | MainProcess | MainThread | [basetestcase:log_failure:518] Cbstat vbucket-details verification failed 2020-05-29 16:27:54,460 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:543] Initializing core dump check on all the nodes 2020-05-29 16:27:54,969 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:351] CLEANUP WAS SKIPPED 2020-05-29 16:27:54,970 | test | INFO | MainProcess | MainThread | [test_summary:display:40] Test summary: Creating couchbase bucket with level 'MAJORITY'.....[OK] subdoc_insert for key nUiCRNw9sUlQ..................[OK] subdoc_upsert for key nUiCRNw9sUlQ..................[FAILED] subdoc_replace for key nUiCRNw9sUlQ.................[FAILED] subdoc_delete for key nUiCRNw9sUlQ..................[FAILED] Delete couchbase bucket.............................[FAILED]