> Task :testrunner Filename: debug.conf Prefix: epengine.basic_ops.basic_ops Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'num_nodes': 3, 'rerun': 'False', 'skip_cluster_reset': 'False', 'spec': 'debug', 'upgrade_version': '7.2.1-5799'} Logs will be stored at /data/workspace/temp_job_ashwin/logs/testrunner-23-Jun-26_03-28-11/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,get-cbcollect-info=False,skip_cluster_reset=False,upgrade_version=7.2.1-5799 -t epengine.basic_ops.basic_ops.test_oso_backfill_not_sending_duplicate_items,nodes_init=3,services_init=kv-kv-index:n1ql,bucket_storage=magma,sdk_client_pool=True,sdk_pool_capacity=6,num_items=300000,vbuckets=2' Test Input params: {'cluster_name': 'node', 'ini': 'node.ini', 'get-cbcollect-info': 'False', 'bucket_storage': 'magma', 'conf_file': 'debug.conf', 'vbuckets': '2', 'upgrade_version': '7.2.1-5799', 'sdk_client_pool': 'True', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 3, 'logs_folder': '/data/workspace/temp_job_ashwin/logs/testrunner-23-Jun-26_03-28-11/test_1', 'nodes_init': '3', 'skip_cluster_reset': 'False', 'num_items': '300000', 'services_init': 'kv-kv-index:n1ql', 'case_number': 1, 'sdk_pool_capacity': '6'} test_oso_backfill_not_sending_duplicate_items (epengine.basic_ops.basic_ops) ... 2023-06-26 03:28:16,632 | test | INFO | MainThread | [basetestcase:log_setup_status:639] ========= basic_ops setup started for test #1 test_oso_backfill_not_sending_duplicate_items ========= 2023-06-26 03:28:16,960 | test | INFO | MainThread | [basetestcase:log_setup_status:639] ========= BaseTestCase setup started for test #1 test_oso_backfill_not_sending_duplicate_items ========= 2023-06-26 03:28:17,696 | test | INFO | MainThread | [basetestcase:initialize_cluster:465] Initializing cluster : C1 2023-06-26 03:28:33,992 | infra | CRITICAL | pool-5-thread-10 | [task:call:6274] mcdMemoryReserved reported in nodes/self is: 9620 2023-06-26 03:28:33,994 | test | CRITICAL | pool-5-thread-10 | [table_view:display:72] Memory quota allocated: +---------------------+---------+ | Service | RAM MiB | +---------------------+---------+ | eventingMemoryQuota | 256 | | cbasMemoryQuota | 1024 | | indexMemoryQuota | 256 | | ftsMemoryQuota | 256 | | memoryQuota | 9520 | +---------------------+---------+ 2023-06-26 03:28:36,628 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.108.72:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["Could not set the storage path. It must be a directory writable by 'couchbase' user."] auth: Administrator:password 2023-06-26 03:28:36,634 | test | ERROR | MainThread | [rest_client:set_data_path:2188] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["Could not set the storage path. It must be a directory writable by 'couchbase' user."] 2023-06-26 03:28:36,635 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:694] RAM quota was defined less than 100 MB: 2023-06-26 03:28:54,329 | test | INFO | MainThread | [basetestcase:initialize_cluster:491] Cluster C1 initialized 2023-06-26 03:28:54,387 | test | INFO | MainThread | [basetestcase:log_setup_status:639] ========= BaseTestCase setup finished for test #1 test_oso_backfill_not_sending_duplicate_items ========= 2023-06-26 03:28:54,421 | test | INFO | MainThread | [basetestcase:log_setup_status:639] ========= ClusterSetup setup started for test #1 test_oso_backfill_not_sending_duplicate_items ========= 2023-06-26 03:29:09,040 | test | INFO | pool-5-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+----------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------------+-----------------------+----------------+--------------+-----------------------+ | 172.23.107.111 | kv | 7.2.1-5799-enterprise | 0.778595515287 | Cluster node | active / none | | 172.23.107.112 | ['kv'] | | | <--- IN --- | | | 172.23.108.72 | ['index,n1ql'] | | | <--- IN --- | | +----------------+----------------+-----------------------+----------------+--------------+-----------------------+ 2023-06-26 03:29:09,076 | test | INFO | pool-5-thread-11 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-06-26 03:29:19,137 | test | INFO | pool-5-thread-11 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-06-26 03:29:34,229 | test | INFO | pool-5-thread-11 | [task:check:421] Rebalance - status: none, progress: 100 2023-06-26 03:29:34,253 | test | INFO | pool-5-thread-11 | [task:check:480] Rebalance completed with progress: 100% in 25.2130000591 sec 2023-06-26 03:29:34,530 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+-------------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+-------------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | 172.23.107.111 | kv | 0.75925497006 | 11.74 GiB | 11.11 GiB | 111.50 MiB / 4.10 GiB | 0 / 0 | 7.2.1-5799-enterprise | | 172.23.107.112 | kv | 0.70648744699 | 11.74 GiB | 11.05 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.2.1-5799-enterprise | | 172.23.108.72 | index, n1ql | 1.70493373035 | 4.03 GiB | 3.26 GiB | 55.26 MiB / 4.24 GiB | 0 / 0 | 7.2.1-5799-enterprise | +----------------+-------------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ 2023-06-26 03:29:34,532 | test | INFO | MainThread | [basetestcase:log_setup_status:639] ========= ClusterSetup setup complete for test #1 test_oso_backfill_not_sending_duplicate_items ========= 2023-06-26 03:29:37,654 | test | INFO | MainThread | [basic_ops:setUp:90] Creating SDK client pool 2023-06-26 03:29:41,079 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | default | couchbase | magma | 1 | none | 0 | 0 | 18.59 GiB | 3.63 MiB | 34.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2023-06-26 03:29:41,082 | test | INFO | MainThread | [basic_ops:setUp:98] ==========Finished Basic_ops base setup======== 2023-06-26 03:29:41,111 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2258] Setting index mem. quota=256M 2023-06-26 03:29:41,138 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2260] Setting indexerThreads=1 for creating DCP pause/resume 2023-06-26 03:29:41,167 | test | INFO | MainThread | [rest_client:set_indexer_params:2647] Indexer indexerThreads set to 1 2023-06-26 03:29:41,168 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2264] Creating collections c1 2023-06-26 03:29:41,239 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2264] Creating collections c2 2023-06-26 03:29:41,325 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2264] Creating collections c3 2023-06-26 03:30:24,753 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2287] Creating GSI index on collection 'c1' 2023-06-26 03:30:40,900 | test | INFO | MainThread | [basic_ops:test_oso_backfill_not_sending_duplicate_items:2297] Index created 2023-06-26 03:30:40,920 | test | INFO | MainThread | [common_lib:sleep:23] Sleep 15 seconds. Reason: Wait before fetching index stats 2023-06-26 03:31:06,713 | test | INFO | MainThread | [basetestcase:check_logs:815] Comparing timestamps: Log's latest timestamp: 2023-06-26 03:30:25, Test's start timestamp is 2023-06-26 03:28:16.631000 2023-06-26 03:31:06,716 | test | INFO | MainThread | [basetestcase:check_logs:815] Comparing timestamps: Log's latest timestamp: 2023-06-26 03:30:25, Test's start timestamp is 2023-06-26 03:28:16.631000 2023-06-26 03:31:06,717 | test | INFO | MainThread | [basetestcase:check_coredump_exist:930] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2023-06-26 03:31:06,717 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:932] 172.23.107.111: Found ' ERROR ' logs - ["2023-06-26T03:30:25.240229-07:00 ERROR BucketLogger::logInner: Failed to log message with format string '{} Skipping OSO backfill for cid:{} as collection item count ({}) is too large a percentage of the vBucket item count ({:.2f}%)'\n"] 2023-06-26 03:31:07,378 | test | INFO | MainThread | [basetestcase:check_logs:815] Comparing timestamps: Log's latest timestamp: 2023-06-26 03:30:25, Test's start timestamp is 2023-06-26 03:28:16.631000 2023-06-26 03:31:07,382 | test | INFO | MainThread | [basetestcase:check_logs:815] Comparing timestamps: Log's latest timestamp: 2023-06-26 03:30:25, Test's start timestamp is 2023-06-26 03:28:16.631000 2023-06-26 03:31:07,384 | test | INFO | MainThread | [basetestcase:check_coredump_exist:930] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2023-06-26 03:31:07,385 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:932] 172.23.107.112: Found ' ERROR ' logs - ["2023-06-26T03:30:25.282915-07:00 ERROR BucketLogger::logInner: Failed to log message with format string '{} Skipping OSO backfill for cid:{} as collection item count ({}) is too large a percentage of the vBucket item count ({:.2f}%)'\n"] 2023-06-26 03:31:08,328 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.107.111,ns_1@172.23.107.112,ns_1@172.23.108.72 2023-06-26 03:31:18,365 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.107.111,ns_1@172.23.107.112,ns_1@172.23.108.72 - cbcollect status: True 2023-06-26 03:31:18,368 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-06-26 03:33:54,114 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.107.111: Copying cbcollect ZIP file to Client 2023-06-26 03:33:54,382 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.107.112: Copying cbcollect ZIP file to Client 2023-06-26 03:33:54,756 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.108.72: Copying cbcollect ZIP file to Client ERROR ====================================================================== ERROR: test_oso_backfill_not_sending_duplicate_items (epengine.basic_ops.basic_ops) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/basic_ops.py", line 101, in tearDown super(basic_ops, self).tearDown() File "pytests/basetestcase.py", line 1106, in tearDown super(ClusterSetup, self).tearDown() File "pytests/basetestcase.py", line 550, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed During the test, Remote Connections: 28, Disconnections: 28 ---------------------------------------------------------------------- SDK Connections: 7, Disconnections: 7 Ran 1 test in 339.289s FAILED (errors=1) summary so far suite epengine.basic_ops.basic_ops , pass 0 , fail 1 failures so far... epengine.basic_ops.basic_ops.test_oso_backfill_not_sending_duplicate_items testrunner logs, diags and results are available under /data/workspace/temp_job_ashwin/logs/testrunner-23-Jun-26_03-28-11/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 5m 58s 2 actionable tasks: 2 executed + '[' false == true ']' Archiving artifacts Recording test results [Checks API] No suitable checks publisher found. Build step 'Publish JUnit test result report' changed build result to UNSTABLE Finished: UNSTABLE