[2019-12-02 13:18:59,989] - [eventing_base] [139636873811712] - INFO - Starting Test: test_partial_rollback [2019-12-02 13:18:59,993] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:19:05,647] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:19:06,501] - [rest_client] [139636873811712] - INFO - Node version in cluster 6.5.0-4908-enterprise [2019-12-02 13:19:06,529] - [rest_client] [139636873811712] - INFO - Node versions in cluster [u'6.5.0-4908-enterprise', u'6.5.0-4908-enterprise', u'6.5.0-4908-enterprise', u'6.5.0-4908-enterprise'] [2019-12-02 13:19:06,529] - [basetestcase] [139636873811712] - INFO - ============== basetestcase setup was started for test #35 test_partial_rollback============== [2019-12-02 13:19:06,645] - [bucket_helper] [139636873811712] - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 172.23.104.100 [2019-12-02 13:19:06,646] - [bucket_helper] [139636873811712] - INFO - remove bucket dst_bucket ... [2019-12-02 13:19:08,323] - [bucket_helper] [139636873811712] - INFO - deleted bucket : dst_bucket from 172.23.104.100 [2019-12-02 13:19:08,323] - [bucket_helper] [139636873811712] - INFO - waiting for bucket deletion to complete.... [2019-12-02 13:19:08,375] - [rest_client] [139636873811712] - INFO - node 172.23.104.100 existing buckets : [u'metadata', u'src_bucket'] [2019-12-02 13:19:08,376] - [bucket_helper] [139636873811712] - INFO - remove bucket metadata ... [2019-12-02 13:19:10,549] - [bucket_helper] [139636873811712] - INFO - deleted bucket : metadata from 172.23.104.100 [2019-12-02 13:19:10,549] - [bucket_helper] [139636873811712] - INFO - waiting for bucket deletion to complete.... [2019-12-02 13:19:10,577] - [rest_client] [139636873811712] - INFO - node 172.23.104.100 existing buckets : [u'src_bucket'] [2019-12-02 13:19:10,579] - [bucket_helper] [139636873811712] - INFO - remove bucket src_bucket ... [2019-12-02 13:19:11,974] - [bucket_helper] [139636873811712] - INFO - deleted bucket : src_bucket from 172.23.104.100 [2019-12-02 13:19:11,975] - [bucket_helper] [139636873811712] - INFO - waiting for bucket deletion to complete.... [2019-12-02 13:19:11,982] - [rest_client] [139636873811712] - INFO - node 172.23.104.100 existing buckets : [] [2019-12-02 13:19:11,982] - [bucket_helper] [139636873811712] - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. [2019-12-02 13:19:14,056] - [cluster_helper] [139636873811712] - INFO - rebalancing all nodes in order to remove nodes [2019-12-02 13:19:14,061] - [rest_client] [139636873811712] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.103', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.100,ns_1@172.23.104.103'} [2019-12-02 13:19:14,071] - [rest_client] [139636873811712] - INFO - rebalance operation started [2019-12-02 13:19:14,085] - [rest_client] [139636873811712] - INFO - rebalance percentage : 0.00 % [2019-12-02 13:19:24,101] - [rest_client] [139636873811712] - INFO - rebalance percentage : 75.00 % [2019-12-02 13:19:34,120] - [rest_client] [139636873811712] - INFO - rebalance percentage : 75.00 % [2019-12-02 13:19:54,157] - [rest_client] [139636873811712] - INFO - rebalance progress took 40.09 seconds [2019-12-02 13:19:54,157] - [rest_client] [139636873811712] - INFO - sleep for 10 seconds after rebalance... [2019-12-02 13:20:04,202] - [cluster_helper] [139636873811712] - INFO - removed all the nodes from cluster associated with ip:172.23.104.100 port:8091 ssh_username:root ? [(u'ns_1@172.23.104.107', 8091), (u'ns_1@172.23.104.105', 8091), (u'ns_1@172.23.104.103', 8091)] [2019-12-02 13:20:04,214] - [cluster_helper] [139636873811712] - INFO - waiting for ns_server @ 172.23.104.100:8091 [2019-12-02 13:20:04,223] - [cluster_helper] [139636873811712] - INFO - ns_server @ 172.23.104.100:8091 is running [2019-12-02 13:20:04,228] - [cluster_helper] [139636873811712] - INFO - waiting for ns_server @ 172.23.104.103:8091 [2019-12-02 13:20:04,233] - [cluster_helper] [139636873811712] - INFO - ns_server @ 172.23.104.103:8091 is running [2019-12-02 13:20:04,238] - [cluster_helper] [139636873811712] - INFO - waiting for ns_server @ 172.23.104.105:8091 [2019-12-02 13:20:04,243] - [cluster_helper] [139636873811712] - INFO - ns_server @ 172.23.104.105:8091 is running [2019-12-02 13:20:04,250] - [cluster_helper] [139636873811712] - INFO - waiting for ns_server @ 172.23.104.107:8091 [2019-12-02 13:20:04,256] - [cluster_helper] [139636873811712] - INFO - ns_server @ 172.23.104.107:8091 is running [2019-12-02 13:20:04,256] - [basetestcase] [139636873811712] - INFO - initializing cluster [2019-12-02 13:20:04,257] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:20:10,377] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:20:11,138] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:20:16,895] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:20:17,775] - [remote_util] [139636873811712] - INFO - 172.23.104.100 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:20:17,904] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:20:17,904] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.100: systemctl stop couchbase-server.service [2019-12-02 13:20:20,378] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:20,379] - [basetestcase] [139636873811712] - INFO - Couchbase stopped [2019-12-02 13:20:20,443] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.100: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 13:20:20,479] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:20,480] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.100: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 13:20:20,578] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:20,580] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:20:26,206] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:20:27,086] - [remote_util] [139636873811712] - INFO - 172.23.104.100 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:20:27,223] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:20:27,223] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.100: systemctl start couchbase-server.service [2019-12-02 13:20:27,261] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:27,261] - [basetestcase] [139636873811712] - INFO - Couchbase started [2019-12-02 13:20:27,377] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:20:32,849] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.103 [2019-12-02 13:20:33,639] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:20:39,813] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.103 [2019-12-02 13:20:40,655] - [remote_util] [139636873811712] - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:20:40,723] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:20:40,724] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.103: systemctl stop couchbase-server.service [2019-12-02 13:20:42,345] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:42,345] - [basetestcase] [139636873811712] - INFO - Couchbase stopped [2019-12-02 13:20:42,409] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 13:20:42,428] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:42,429] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 13:20:42,510] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:42,513] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:20:47,677] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.103 [2019-12-02 13:20:48,518] - [remote_util] [139636873811712] - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:20:48,647] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:20:48,648] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.103: systemctl start couchbase-server.service [2019-12-02 13:20:48,684] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:20:48,684] - [basetestcase] [139636873811712] - INFO - Couchbase started [2019-12-02 13:20:48,801] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:20:54,285] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.105 [2019-12-02 13:20:55,078] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:21:01,766] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.105 [2019-12-02 13:21:02,602] - [remote_util] [139636873811712] - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:21:02,727] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:21:02,727] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: systemctl stop couchbase-server.service [2019-12-02 13:21:04,298] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:04,298] - [basetestcase] [139636873811712] - INFO - Couchbase stopped [2019-12-02 13:21:04,412] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 13:21:04,434] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:04,434] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 13:21:04,516] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:04,519] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:21:10,116] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.105 [2019-12-02 13:21:10,918] - [remote_util] [139636873811712] - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:21:11,042] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:21:11,042] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: systemctl start couchbase-server.service [2019-12-02 13:21:11,080] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:11,080] - [basetestcase] [139636873811712] - INFO - Couchbase started [2019-12-02 13:21:11,197] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.107 with username:root [2019-12-02 13:21:16,538] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.107 [2019-12-02 13:21:18,103] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.107 with username:root [2019-12-02 13:21:23,488] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.107 [2019-12-02 13:21:25,046] - [remote_util] [139636873811712] - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:21:25,256] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:21:25,256] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.107: systemctl stop couchbase-server.service [2019-12-02 13:21:27,230] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:27,230] - [basetestcase] [139636873811712] - INFO - Couchbase stopped [2019-12-02 13:21:27,344] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.107: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 13:21:27,414] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:27,415] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.107: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 13:21:27,546] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:27,548] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.107 with username:root [2019-12-02 13:21:32,952] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.107 [2019-12-02 13:21:34,563] - [remote_util] [139636873811712] - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 13:21:34,846] - [remote_util] [139636873811712] - INFO - Running systemd command on this server [2019-12-02 13:21:34,846] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.107: systemctl start couchbase-server.service [2019-12-02 13:21:34,966] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:21:34,966] - [basetestcase] [139636873811712] - INFO - Couchbase started [2019-12-02 13:21:35,080] - [basetestcase] [139636873811712] - INFO - sleep for 10 secs. ... [2019-12-02 13:21:45,825] - [task] [139636882204416] - INFO - server: ip:172.23.104.100 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.100', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.100', 'uptime': u'71', 'mcdMemoryReserved': 3349, 'storageTotalRam': 4187, 'hostname': u'172.23.104.100:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3513544704, 'memoryTotal': 4390723584, 'memoryQuota': 1846, 'mcdMemoryAllocated': 3349, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 13:21:45,830] - [rest_client] [139636882204416] - ERROR - GET http://172.23.104.100:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 13:21:45,830] - [rest_client] [139636882204416] - INFO - pools/default params : memoryQuota=2243 [2019-12-02 13:21:45,842] - [rest_client] [139636882204416] - INFO - /node/controller/setupServices params on 172.23.104.100:8091:services=kv&password=password&user=Administrator&hostname=172.23.104.100 [2019-12-02 13:21:45,858] - [rest_client] [139636882204416] - INFO - settings/web params on 172.23.104.100:8091:username=Administrator&password=password&port=8091 [2019-12-02 13:21:45,950] - [remote_util] [139636882204416] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:21:51,691] - [remote_util] [139636882204416] - INFO - Connected to 172.23.104.100 [2019-12-02 13:21:52,478] - [remote_util] [139636882204416] - INFO - running command.raw on 172.23.104.100: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 13:21:52,577] - [remote_util] [139636882204416] - INFO - command executed successfully [2019-12-02 13:21:52,582] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.100:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:21:52,586] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.100:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:21:52,586] - [rest_client] [139636882204416] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 13:21:52,690] - [task] [139636882204416] - INFO - server: ip:172.23.104.103 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.103', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.103', 'uptime': u'59', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4024, 'hostname': u'172.23.104.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3809763328, 'memoryTotal': 4220149760, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 13:21:52,694] - [rest_client] [139636882204416] - ERROR - GET http://172.23.104.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 13:21:52,695] - [rest_client] [139636882204416] - INFO - pools/default params : memoryQuota=2156 [2019-12-02 13:21:52,702] - [rest_client] [139636882204416] - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 [2019-12-02 13:21:52,751] - [remote_util] [139636882204416] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:21:58,520] - [remote_util] [139636882204416] - INFO - Connected to 172.23.104.103 [2019-12-02 13:21:59,311] - [remote_util] [139636882204416] - INFO - running command.raw on 172.23.104.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 13:21:59,360] - [remote_util] [139636882204416] - INFO - command executed successfully [2019-12-02 13:21:59,366] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.103:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:21:59,372] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.103:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:21:59,372] - [rest_client] [139636882204416] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 13:21:59,475] - [task] [139636882204416] - INFO - server: ip:172.23.104.105 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.105', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.105', 'uptime': u'43', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.105:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3761692672, 'memoryTotal': 4220854272, 'memoryQuota': 1749, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 13:21:59,479] - [rest_client] [139636882204416] - ERROR - GET http://172.23.104.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 13:21:59,480] - [rest_client] [139636882204416] - INFO - pools/default params : memoryQuota=2157 [2019-12-02 13:21:59,488] - [rest_client] [139636882204416] - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 [2019-12-02 13:21:59,537] - [remote_util] [139636882204416] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:22:05,523] - [remote_util] [139636882204416] - INFO - Connected to 172.23.104.105 [2019-12-02 13:22:06,378] - [remote_util] [139636882204416] - INFO - running command.raw on 172.23.104.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 13:22:06,444] - [remote_util] [139636882204416] - INFO - command executed successfully [2019-12-02 13:22:06,450] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:22:06,456] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:22:06,456] - [rest_client] [139636882204416] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 13:22:06,567] - [task] [139636882204416] - INFO - server: ip:172.23.104.107 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.107', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.107', 'uptime': u'27', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.107:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3829833728, 'memoryTotal': 4220854272, 'memoryQuota': 1749, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 13:22:06,574] - [rest_client] [139636882204416] - ERROR - GET http://172.23.104.107:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 13:22:06,574] - [rest_client] [139636882204416] - INFO - pools/default params : memoryQuota=2157 [2019-12-02 13:22:06,588] - [rest_client] [139636882204416] - INFO - settings/web params on 172.23.104.107:8091:username=Administrator&password=password&port=8091 [2019-12-02 13:22:06,681] - [remote_util] [139636882204416] - INFO - connecting to 172.23.104.107 with username:root [2019-12-02 13:22:11,799] - [remote_util] [139636882204416] - INFO - Connected to 172.23.104.107 [2019-12-02 13:22:13,334] - [remote_util] [139636882204416] - INFO - running command.raw on 172.23.104.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 13:22:13,451] - [remote_util] [139636882204416] - INFO - command executed successfully [2019-12-02 13:22:13,456] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:22:13,460] - [rest_client] [139636882204416] - INFO - /diag/eval status on 172.23.104.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 13:22:13,461] - [rest_client] [139636882204416] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 13:22:13,536] - [basetestcase] [139636873811712] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.100 **** [2019-12-02 13:22:13,558] - [rest_client] [139636873811712] - ERROR - DELETE http://172.23.104.100:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-12-02 13:22:13,559] - [internal_user] [139636873811712] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-12-02 13:22:13,708] - [basetestcase] [139636873811712] - INFO - sleep for 5 secs. ... [2019-12-02 13:22:18,713] - [basetestcase] [139636873811712] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2019-12-02 13:22:18,741] - [basetestcase] [139636873811712] - INFO - done initializing cluster [2019-12-02 13:22:19,524] - [task] [139636882204416] - INFO - adding node 172.23.104.103:8091 to cluster [2019-12-02 13:22:19,524] - [rest_client] [139636882204416] - INFO - adding remote node @172.23.104.103:8091 to this cluster @172.23.104.100:8091 [2019-12-02 13:22:21,917] - [task] [139636882204416] - INFO - adding node 172.23.104.105:8091 to cluster [2019-12-02 13:22:21,918] - [rest_client] [139636882204416] - INFO - adding remote node @172.23.104.105:8091 to this cluster @172.23.104.100:8091 [2019-12-02 13:22:24,002] - [task] [139636882204416] - INFO - adding node 172.23.104.107:8091 to cluster [2019-12-02 13:22:24,002] - [rest_client] [139636882204416] - INFO - adding remote node @172.23.104.107:8091 to this cluster @172.23.104.100:8091 [2019-12-02 13:22:27,247] - [rest_client] [139636882204416] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.100,ns_1@172.23.104.103'} [2019-12-02 13:22:27,260] - [rest_client] [139636882204416] - INFO - rebalance operation started [2019-12-02 13:22:27,272] - [rest_client] [139636882204416] - INFO - rebalance percentage : 0.00 % [2019-12-02 13:22:27,272] - [task] [139636882204416] - INFO - Rebalance - status: running, progress: 0.00% [2019-12-02 13:22:37,306] - [task] [139636882204416] - INFO - Rebalance - status: none, progress: 100.00% [2019-12-02 13:22:37,321] - [task] [139636882204416] - INFO - rebalancing was completed with progress: 100% in 10.0607180595 sec [2019-12-02 13:22:37,324] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:22:42,835] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:22:43,645] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.100: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 13:22:43,719] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:22:43,719] - [basetestcase] [139636873811712] - INFO - Enabled diag/eval for non-local hosts from 172.23.104.100 [2019-12-02 13:22:43,834] - [basetestcase] [139636873811712] - INFO - ============== basetestcase setup was finished for test #35 test_partial_rollback ============== [2019-12-02 13:22:43,877] - [basetestcase] [139636873811712] - INFO - ------- Cluster statistics ------- [2019-12-02 13:22:43,877] - [basetestcase] [139636873811712] - INFO - 172.23.104.107:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.7518796992481203, 'mem_free': 3718742016, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'index', u'n1ql']} [2019-12-02 13:22:43,878] - [basetestcase] [139636873811712] - INFO - 172.23.104.105:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3711426560, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'eventing']} [2019-12-02 13:22:43,878] - [basetestcase] [139636873811712] - INFO - 172.23.104.100:8091 => {'swap_mem_used': 0, 'cpu_utilization': 4.896907216494846, 'mem_free': 3452280832, 'swap_mem_total': 0, 'mem_total': 4390723584, 'services': [u'kv']} [2019-12-02 13:22:43,878] - [basetestcase] [139636873811712] - INFO - 172.23.104.103:8091 => {'swap_mem_used': 0, 'cpu_utilization': 6.54911838790932, 'mem_free': 3765907456, 'swap_mem_total': 0, 'mem_total': 4220149760, 'services': [u'kv']} [2019-12-02 13:22:43,878] - [basetestcase] [139636873811712] - INFO - --- End of cluster statistics --- [2019-12-02 13:22:43,881] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:22:49,383] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.100 [2019-12-02 13:22:54,573] - [basetestcase] [139636873811712] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-12-02 13:22:54,587] - [basetestcase] [139636873811712] - INFO - list of eventing nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root] [2019-12-02 13:22:54,593] - [rest_client] [139636873811712] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 13:22:54,600] - [eventing_base] [139636873811712] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2019-12-02 13:22:54,600] - [rest_client] [139636873811712] - INFO - pools/default params : memoryQuota=330 [2019-12-02 13:22:54,607] - [rest_client] [139636873811712] - INFO - pools/default params : indexMemoryQuota=256 [2019-12-02 13:22:54,613] - [rest_client] [139636873811712] - INFO - pools/default params : eventingMemoryQuota=512 [2019-12-02 13:22:54,641] - [basetestcase] [139636873811712] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-12-02 13:22:54,641] - [rest_client] [139636873811712] - INFO - pools/default params : memoryQuota=700 [2019-12-02 13:22:54,648] - [eventing_recovery] [139636873811712] - INFO - 100 [2019-12-02 13:22:55,383] - [rest_client] [139636882204416] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2019-12-02 13:22:55,396] - [rest_client] [139636882204416] - INFO - 0.01 seconds to create bucket src_bucket [2019-12-02 13:22:55,396] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:22:56,172] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:56,407] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:56,607] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:56,847] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:56,955] - [task] [139636882204416] - WARNING - vbucket map not ready after try 0 [2019-12-02 13:22:56,956] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:22:57,228] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:57,470] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:57,673] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:57,914] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:58,009] - [task] [139636882204416] - WARNING - vbucket map not ready after try 1 [2019-12-02 13:22:58,009] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:22:58,393] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:58,544] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:58,852] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:58,994] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:59,089] - [task] [139636882204416] - WARNING - vbucket map not ready after try 2 [2019-12-02 13:22:59,089] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:22:59,407] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:59,551] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:22:59,827] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:22:59,989] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:00,082] - [task] [139636882204416] - WARNING - vbucket map not ready after try 3 [2019-12-02 13:23:00,082] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:00,379] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:00,523] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:00,806] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:00,957] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:01,052] - [task] [139636882204416] - WARNING - vbucket map not ready after try 4 [2019-12-02 13:23:01,053] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:01,367] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:01,512] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:01,798] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:01,904] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:02,047] - [task] [139636882204416] - WARNING - vbucket map not ready after try 5 [2019-12-02 13:23:03,099] - [rest_client] [139636882204416] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2019-12-02 13:23:03,115] - [rest_client] [139636882204416] - INFO - 0.02 seconds to create bucket dst_bucket [2019-12-02 13:23:03,116] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:03,851] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:04,049] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:04,211] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:04,411] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:04,479] - [task] [139636882204416] - WARNING - vbucket map not ready after try 0 [2019-12-02 13:23:04,480] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:04,729] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:04,861] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:05,036] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:05,296] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:05,389] - [task] [139636882204416] - WARNING - vbucket map not ready after try 1 [2019-12-02 13:23:05,389] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:05,729] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:05,878] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:06,170] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:06,331] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:06,423] - [task] [139636882204416] - WARNING - vbucket map not ready after try 2 [2019-12-02 13:23:06,424] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:06,754] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:06,912] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:07,210] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:07,382] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:07,479] - [task] [139636882204416] - WARNING - vbucket map not ready after try 3 [2019-12-02 13:23:07,480] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:07,831] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:07,993] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:08,295] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:08,453] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:08,645] - [task] [139636882204416] - WARNING - vbucket map not ready after try 4 [2019-12-02 13:23:08,646] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-12-02 13:23:08,896] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:09,050] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:09,361] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-12-02 13:23:09,518] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 dst_bucket [2019-12-02 13:23:09,706] - [task] [139636882204416] - WARNING - vbucket map not ready after try 5 [2019-12-02 13:23:10,773] - [rest_client] [139636882204416] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=400 [2019-12-02 13:23:10,793] - [rest_client] [139636882204416] - INFO - 0.02 seconds to create bucket metadata [2019-12-02 13:23:10,793] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:11,625] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:11,825] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:11,985] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:12,176] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:12,242] - [task] [139636882204416] - WARNING - vbucket map not ready after try 0 [2019-12-02 13:23:12,243] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:12,460] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:12,690] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:12,889] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:13,086] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:13,158] - [task] [139636882204416] - WARNING - vbucket map not ready after try 1 [2019-12-02 13:23:13,159] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:13,469] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:13,584] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:13,742] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:13,935] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:13,999] - [task] [139636882204416] - WARNING - vbucket map not ready after try 2 [2019-12-02 13:23:13,999] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:14,255] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:14,370] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:14,615] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:14,739] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:14,812] - [task] [139636882204416] - WARNING - vbucket map not ready after try 3 [2019-12-02 13:23:14,812] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:15,087] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:15,215] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:15,450] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:15,568] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:15,630] - [task] [139636882204416] - WARNING - vbucket map not ready after try 4 [2019-12-02 13:23:15,631] - [bucket_helper] [139636882204416] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-12-02 13:23:15,936] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:16,089] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:16,395] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-12-02 13:23:16,562] - [data_helper] [139636882204416] - INFO - creating direct client 172.23.104.103:11210 metadata [2019-12-02 13:23:16,751] - [task] [139636882204416] - WARNING - vbucket map not ready after try 5 [2019-12-02 13:23:17,024] - [basetestcase] [139636873811712] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-12-02 13:23:17,025] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:17,037] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:17,521] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 479.834612ms [2019-12-02 13:23:17,521] - [tuq_helper] [139636873811712] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2019-12-02 13:23:17,533] - [rest_client] [139636873811712] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2019-12-02 13:23:21,193] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 3.656314297s [2019-12-02 13:23:21,193] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:21,205] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:21,281] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 74.17415ms [2019-12-02 13:23:21,281] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:21,292] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:21,350] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 55.763571ms [2019-12-02 13:23:21,351] - [tuq_helper] [139636873811712] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2019-12-02 13:23:21,360] - [rest_client] [139636873811712] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2019-12-02 13:23:25,106] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 3.742481922s [2019-12-02 13:23:25,106] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:25,117] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:25,191] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 70.461586ms [2019-12-02 13:23:25,191] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:25,201] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:25,255] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 51.690583ms [2019-12-02 13:23:25,256] - [tuq_helper] [139636873811712] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2019-12-02 13:23:25,265] - [rest_client] [139636873811712] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2019-12-02 13:23:28,946] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 3.678698969s [2019-12-02 13:23:28,947] - [tuq_helper] [139636873811712] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 13:23:28,960] - [rest_client] [139636873811712] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 13:23:29,061] - [tuq_helper] [139636873811712] - INFO - TOTAL ELAPSED TIME: 98.126572ms [2019-12-02 13:23:29,093] - [basetestcase] [139636873811712] - INFO - list of kv nodes in cluster: [ip:172.23.104.100 port:8091 ssh_username:root, ip:172.23.104.103 port:8091 ssh_username:root] [2019-12-02 13:23:29,093] - [eventing_recovery] [139636873811712] - INFO - kv nodes:[ip:172.23.104.100 port:8091 ssh_username:root, ip:172.23.104.103 port:8091 ssh_username:root] [2019-12-02 13:23:29,111] - [data_helper] [139636873811712] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:29,304] - [data_helper] [139636873811712] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:29,516] - [data_helper] [139636873811712] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:29,778] - [data_helper] [139636873811712] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:29,903] - [eventing_base] [139636873811712] - INFO - Deploying the following handler code : Function_976230229_test_partial_rollback with {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} [2019-12-02 13:23:29,904] - [eventing_base] [139636873811712] - INFO - function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { var docID = context.docID; while (true) { try { var query = DELETE FROM dst_bucket where meta().id = $docID; break; } catch (e) { log(e); } } } function timerCallback(context) { var docID = context.docID; while (true) { try { var query = INSERT INTO dst_bucket ( KEY, VALUE ) VALUES ( $docID ,'timerCallback'); break; } catch (e) { log(e); } } } [2019-12-02 13:23:31,377] - [eventing_base] [139636873811712] - INFO - deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_976230229_test_partial_rollback' in metakv", "warnings": null } } [2019-12-02 13:23:31,377] - [eventing_recovery] [139636873811712] - INFO - Killing memcached on ip:172.23.104.103 port:8091 ssh_username:root [2019-12-02 13:23:31,382] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:23:36,829] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.103 [2019-12-02 13:23:37,691] - [data_helper] [139636907382528] - ERROR - Connection reset with error: [Errno 104] Connection reset by peer while _send_op, server is alive? [2019-12-02 13:23:37,713] - [eventing_recovery] [139636873811712] - INFO - Starting persistence on ip:172.23.104.100 port:8091 ssh_username:root [2019-12-02 13:23:37,780] - [data_helper] [139636873811712] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:38,191] - [data_helper] [139636907382528] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-12-02 13:23:38,201] - [basetestcase] [139636873811712] - INFO - sleep for 20 secs. Waiting for Function_976230229_test_partial_rollback to deployed... ... [2019-12-02 13:23:38,371] - [data_helper] [139636907382528] - INFO - creating direct client 172.23.104.103:11210 src_bucket [2019-12-02 13:23:58,521] - [basetestcase] [139636873811712] - INFO - sleep for 20 secs. Waiting for Function_976230229_test_partial_rollback to deployed... ... [2019-12-02 13:24:18,810] - [eventing_recovery] [139636873811712] - INFO - {u'cpu_cores_available': 16, u'ep_dcp_fts_backoff': 0, u'vb_active_sync_write_aborted_count': 0, u'ep_dcp_2i_total_backlog_size': 0, u'ep_dcp_views+indexes_total_backlog_size': 0, u'decr_misses': 0, u'ep_dcp_cbas_items_remaining': 0, u'ep_dcp_xdcr_items_sent': 0, u'ep_oom_errors': 0, u'ep_num_value_ejects': 0, u'cas_hits': 0, u'ep_vb_total': 2048, u'ep_dcp_replica_total_bytes': 0, u'disk_update_count': 0, u'ep_diskqueue_drain': 0, u'ep_num_ops_del_ret_meta': 0, u'ep_dcp_other_backoff': 0, u'bg_wait_total': 0, u'ep_ops_create': 0, u'ep_replica_ahead_exceptions': 0, u'ep_dcp_2i_backoff': 0, u'ep_dcp_other_count': 0, u'disk_update_total': 0, u'ep_dcp_fts_producer_count': 0, u'ops': 0, u'ep_dcp_fts_total_backlog_size': 0, u'ep_diskqueue_fill': 0, u'ep_dcp_eventing_backoff': 0, u'vb_active_resident_items_ratio': 100, u'ep_dcp_other_producer_count': 0, u'ep_dcp_replica_items_remaining': 0, u'ep_dcp_replica_count': 4, u'ep_dcp_eventing_producer_count': 8, u'ep_dcp_views_producer_count': 0, u'vb_replica_queue_age': 6991316001114126439, u'ep_dcp_fts_count': 0, u'ep_dcp_fts_items_remaining': 0, u'mem_free': 13014384640, u'ep_dcp_fts_total_bytes': 0, u'evictions': 0, u'vb_active_sync_write_committed_count': 0, u'ep_cache_miss_rate': 0, u'vb_avg_pending_queue_age': 0, u'ep_dcp_xdcr_items_remaining': 0, u'timestamp': 1575321857440, u'bytes_written': 122432.5674325674, u'vb_active_num': 1024, u'ep_bg_fetched': 0, u'vb_pending_queue_fill': 0, u'mem_limit': 17052581888, u'vb_avg_active_queue_age': 0, u'ep_dcp_cbas_total_backlog_size': 0, u'ep_dcp_eventing_count': 8, u'vb_active_meta_data_memory': 59687, u'vb_replica_itm_memory': 327625, u'ep_dcp_views_items_sent': 0, u'ep_dcp_2i_count': 12, u'ep_dcp_views+indexes_producer_count': 12, u'avg_active_timestamp_drift': 0, u'ep_clock_cas_drift_threshold_exceeded': 0, u'ep_diskqueue_items': 0, u'mem_total': 17052581888, u'ep_num_ops_set_meta': 0, u'cpu_stolen_rate': 1.307346999523328, u'ep_active_hlc_drift': 0, u'ep_dcp_views_count': 0, u'ep_dcp_cbas_items_sent': 0, u'avg_replica_timestamp_drift': 0, u'ep_replica_hlc_drift': 0, u'cpu_utilization_rate': 15.28497409326425, u'vb_total_queue_age': 13982631796237326218L, u'vb_active_itm_memory': 327625, u'ep_dcp_views_total_bytes': 0, u'misses': 0, u'ep_dcp_cbas_backoff': 0, u'couch_views_disk_size': 0, u'bytes_read': 553.4465534465535, u'ep_dcp_replica_backoff': 0, u'ep_mem_high_wat': 178257920, u'vb_pending_num_non_resident': 0, u'vb_pending_queue_drain': 0, u'cmd_lookup': 0, u'ep_active_hlc_drift_count': 0, u'cmd_get': 0, u'curr_connections': 130, u'ep_num_non_resident': 0, u'ep_dcp_xdcr_count': 0, u'vb_active_ops_update': 0, u'couch_docs_fragmentation': 0, u'curr_items': 657, u'ep_dcp_xdcr_producer_count': 0, u'ep_queue_size': 0, u'delete_misses': 0, u'vb_replica_queue_drain': 0, u'vb_avg_total_queue_age': 0, u'ep_dcp_eventing_items_remaining': 0, u'couch_docs_disk_size': 16170328, u'ep_dcp_2i_items_remaining': 0, u'vb_replica_queue_size': 0, u'avg_disk_update_time': 0, u'ep_overhead': 10345145, u'mem_actual_free': 13014384640, u'disk_commit_total': 0, u'vb_active_eject': 0, u'ep_resident_items_rate': 100, u'ep_dcp_eventing_total_bytes': 0, u'couch_total_disk_size': 23399624, u'vb_pending_queue_size': 0, u'ep_dcp_xdcr_total_bytes': 0, u'cas_misses': 0, u'ep_dcp_2i_producer_count': 12, u'hit_ratio': 0, u'rest_requests': 10, u'ep_ops_update': 0, u'ep_kv_size': 1018574, u'vb_replica_ops_create': 0, u'get_misses': 0, u'disk_commit_count': 0, u'vb_active_ops_create': 0, u'couch_docs_actual_disk_size': 23399624, u'cpu_sys_rate': 8.76401065077018, u'vb_pending_eject': 0, u'vb_active_queue_fill': 0, u'hibernated_requests': 17, u'mem_actual_used': 4038197248, u'vb_avg_replica_queue_age': 0, u'vb_pending_curr_items': 0, u'couch_views_fragmentation': 0, u'ep_num_ops_set_ret_meta': 0, u'ep_dcp_other_total_backlog_size': 0, u'couch_docs_data_size': 16170328, u'vb_pending_resident_items_ratio': 100, u'couch_views_ops': 0, u'ep_dcp_views_total_backlog_size': 0, u'couch_views_data_size': 0, u'mem_used': 51609616, u'vb_active_queue_drain': 0, u'ep_active_ahead_exceptions': 0, u'ep_dcp_other_items_sent': 0, u'bg_wait_count': 0, u'vb_replica_eject': 0, u'vb_active_sync_write_accepted_count': 0, u'vb_active_num_non_resident': 0, u'cmd_set': 0, u'vb_active_queue_size': 0, u'xdc_ops': 0, u'incr_misses': 0, u'ep_mem_low_wat': 157286400, u'cas_badval': 0, u'ep_flusher_todo': 0, u'ep_dcp_cbas_count': 0, u'ep_dcp_eventing_total_backlog_size': 0, u'vb_pending_queue_age': 0, u'avg_disk_commit_time': 0, u'ep_dcp_cbas_total_bytes': 0, u'ep_dcp_replica_items_sent': 0, u'vb_active_queue_age': 6991315795123199779, u'ep_dcp_xdcr_total_backlog_size': 0, u'ep_dcp_views_backoff': 0, u'disk_write_queue': 0, u'ep_dcp_views+indexes_items_remaining': 0, u'vb_replica_num': 1024, u'vb_replica_resident_items_ratio': 100, u'couch_spatial_disk_size': 0, u'ep_dcp_other_total_bytes': 0, u'ep_data_write_failed': 0, u'ep_num_ops_get_meta': 0, u'vb_pending_itm_memory': 0, u'mem_used_sys': 9243820032, u'ep_item_commit_failed': 0, u'ep_dcp_replica_producer_count': 2, u'ep_dcp_replica_total_backlog_size': 0, u'ep_dcp_views_items_remaining': 0, u'cpu_irq_rate': 0.7808999628659898, u'avg_bg_wait_time': 0, u'odp_report_failed': 0, u'couch_views_actual_disk_size': 0, u'vb_pending_ops_create': 0, u'swap_total': 7516184576, u'ep_dcp_cbas_producer_count': 0, u'couch_spatial_ops': 0, u'decr_hits': 0, u'incr_hits': 0, u'hibernated_waked': 1, u'cpu_user_rate': 23.96504016599826, u'curr_items_tot': 1314, u'vb_replica_num_non_resident': 0, u'ep_dcp_fts_items_sent': 0, u'ep_dcp_2i_items_sent': 0, u'ep_dcp_views+indexes_backoff': 0, u'swap_used': 0, u'ep_dcp_other_items_remaining': 0, u'ep_dcp_eventing_items_sent': 0, u'ep_dcp_views+indexes_items_sent': 0, u'vb_replica_meta_data_memory': 59687, u'ep_dcp_2i_total_bytes': 0, u'vb_pending_meta_data_memory': 0, u'get_hits': 0, u'delete_hits': 0, u'vb_pending_num': 0, u'ep_meta_data_memory': 119374, u'ep_num_ops_del_meta': 0, u'ep_replica_hlc_drift_count': 0, u'vb_replica_curr_items': 657, u'ep_tmp_oom_errors': 0, u'ep_dcp_views+indexes_count': 12, u'ep_dcp_views+indexes_total_bytes': 0, u'vb_replica_queue_fill': 0, u'couch_spatial_data_size': 0, u'vb_pending_ops_update': 0, u'ep_data_read_failed': 0, u'ep_max_size': 209715200, u'vb_replica_ops_update': 0, u'ep_dcp_xdcr_backoff': 0} [2019-12-02 13:24:18,849] - [basetestcase] [139636873811712] - INFO - list of eventing nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root] [2019-12-02 13:24:18,876] - [basetestcase] [139636873811712] - INFO - list of eventing nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root] [2019-12-02 13:24:18,936] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:24:49,025] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:25:19,112] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:25:49,203] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:26:19,291] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:26:49,386] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:27:19,474] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:27:49,564] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:28:19,641] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:28:49,731] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 0 Expected : 657 ... [2019-12-02 13:29:19,789] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 536 Expected : 657 ... [2019-12-02 13:29:49,880] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:30:19,969] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:30:50,063] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:31:20,160] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:31:50,250] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:32:20,344] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:32:50,402] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:33:20,512] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:33:50,602] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:34:20,692] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:34:50,775] - [basetestcase] [139636873811712] - INFO - sleep for 30 secs. Waiting for handler code Function_976230229_test_partial_rollback to complete bucket operations... Current : 563 Expected : 657 ... [2019-12-02 13:35:20,919] - [eventing_base] [139636873811712] - INFO - Documents in source bucket : 657 [2019-12-02 13:35:22,190] - [eventing_base] [139636873811712] - INFO - Stats for Node 172.23.104.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 12169, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 657, "dcp_mutation_sent_to_worker": 657, "dcp_snapshot": 379, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 2094, "failure_stats": 2094, "latency_stats": 2094, "lcb_exception_stats": 2094, "log_level": 3, "num_processed_events": 657, "processed_events_size": 440464, "thr_count": 3, "thr_map": 3, "timer_events": 563, "timer_responses_received": 657, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 657, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 657, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 12157, "num_processed_events": 657, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 657, "processed_events_size": 440464, "timer_create_counter": 657, "timer_create_failure": 0, "timer_msg_counter": 563, "timer_responses_sent": 0, "timestamp": { "17439": "2019-12-02T21:35:20Z", "17444": "2019-12-02T21:35:20Z", "17445": "2019-12-02T21:35:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "17439": "2019-12-02T21:35:20Z", "17444": "2019-12-02T21:35:20Z", "17445": "2019-12-02T21:35:20Z" }, "v8worker_events_lost": 0 }, "function_id": 3492760573, "function_name": "Function_976230229_test_partial_rollback", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_976230229_test_partial_rollback_0": "[0-341]", "worker_Function_976230229_test_partial_rollback_1": "[342-682]", "worker_Function_976230229_test_partial_rollback_2": "[683-1023]" }, "latency_percentile_stats": { "100": 26600, "50": 2800, "80": 4900, "90": 6500, "95": 7900, "99": 12800 }, "lcb_creds_request_counter": 48, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.104.105:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.105:8096": { "worker_Function_976230229_test_partial_rollback_0": "[0-341]", "worker_Function_976230229_test_partial_rollback_1": "[342-682]", "worker_Function_976230229_test_partial_rollback_2": "[683-1023]" } }, "worker_pids": { "worker_Function_976230229_test_partial_rollback_0": 17444, "worker_Function_976230229_test_partial_rollback_1": 17445, "worker_Function_976230229_test_partial_rollback_2": 17439 } } ] [2019-12-02 13:35:22,781] - [basetestcase] [139636873811712] - INFO - list of eventing nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root] [2019-12-02 13:35:22,783] - [remote_util] [139636873811712] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:35:29,266] - [remote_util] [139636873811712] - INFO - Connected to 172.23.104.105 [2019-12-02 13:35:30,017] - [rest_client] [139636873811712] - INFO - /diag/eval status on 172.23.104.105:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-12-02 13:35:30,018] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2019-12-02 13:35:30,115] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:35:30,116] - [remote_util] [139636873811712] - INFO - running command.raw on 172.23.104.105: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2019-12-02 13:35:30,149] - [remote_util] [139636873811712] - INFO - command executed successfully [2019-12-02 13:35:30,546] - [eventing_base] [139636873811712] - INFO - Bucket dst_bucket DGM is 100 [2019-12-02 13:35:30,620] - [eventing_base] [139636873811712] - INFO - Bucket metadata DGM is 100 [2019-12-02 13:35:30,694] - [eventing_base] [139636873811712] - INFO - Bucket src_bucket DGM is 100 [2019-12-02 13:35:30,730] - [basetestcase] [139636873811712] - INFO - ------- Cluster statistics ------- [2019-12-02 13:35:30,730] - [basetestcase] [139636873811712] - INFO - 172.23.104.107:8091 => {'swap_mem_used': 0, 'cpu_utilization': 2.659574468085106, 'mem_free': 3491262464, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'index', u'n1ql']} [2019-12-02 13:35:30,731] - [basetestcase] [139636873811712] - INFO - 172.23.104.105:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.01010101010101, 'mem_free': 3483238400, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'eventing']} [2019-12-02 13:35:30,731] - [basetestcase] [139636873811712] - INFO - 172.23.104.100:8091 => {'swap_mem_used': 0, 'cpu_utilization': 13.98963730569948, 'mem_free': 2705952768, 'swap_mem_total': 0, 'mem_total': 4390723584, 'services': [u'kv']} [2019-12-02 13:35:30,731] - [basetestcase] [139636873811712] - INFO - 172.23.104.103:8091 => {'swap_mem_used': 0, 'cpu_utilization': 11.58690176322418, 'mem_free': 3178151936, 'swap_mem_total': 0, 'mem_total': 4220149760, 'services': [u'kv']} [2019-12-02 13:35:30,731] - [basetestcase] [139636873811712] - INFO - --- End of cluster statistics --- [2019-12-02 13:35:30,732] - [basetestcase] [139636873811712] - WARNING - CLEANUP WAS SKIPPED [2019-12-02 13:35:30,839] - [remote_util] [139637140539200] - INFO - connecting to 172.23.104.100 with username:root [2019-12-02 13:35:36,331] - [remote_util] [139637140539200] - INFO - Connected to 172.23.104.100 [2019-12-02 13:35:37,065] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.100: /opt/couchbase/bin/cbcollect_info 172.23.104.100-20191202-1335-diag.zip [2019-12-02 13:50:56,911] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:51:26,391] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.100: rm -f /root/172.23.104.100-20191202-1335-diag.zip [2019-12-02 13:51:26,766] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:51:26,832] - [remote_util] [139637140539200] - INFO - connecting to 172.23.104.103 with username:root [2019-12-02 13:51:32,876] - [remote_util] [139637140539200] - INFO - Connected to 172.23.104.103 [2019-12-02 13:51:33,636] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.103: /opt/couchbase/bin/cbcollect_info 172.23.104.103-20191202-1351-diag.zip [2019-12-02 13:55:12,301] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:55:15,245] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.103: rm -f /root/172.23.104.103-20191202-1351-diag.zip [2019-12-02 13:55:15,283] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:55:15,399] - [remote_util] [139637140539200] - INFO - connecting to 172.23.104.105 with username:root [2019-12-02 13:55:20,871] - [remote_util] [139637140539200] - INFO - Connected to 172.23.104.105 [2019-12-02 13:55:21,589] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.105: /opt/couchbase/bin/cbcollect_info 172.23.104.105-20191202-1355-diag.zip [2019-12-02 13:56:44,666] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:56:45,792] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.105: rm -f /root/172.23.104.105-20191202-1355-diag.zip [2019-12-02 13:56:45,813] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:56:45,929] - [remote_util] [139637140539200] - INFO - connecting to 172.23.104.107 with username:root [2019-12-02 13:56:51,596] - [remote_util] [139637140539200] - INFO - Connected to 172.23.104.107 [2019-12-02 13:56:52,994] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.107: /opt/couchbase/bin/cbcollect_info 172.23.104.107-20191202-1356-diag.zip [2019-12-02 13:59:12,624] - [remote_util] [139637140539200] - INFO - command executed successfully [2019-12-02 13:59:13,395] - [remote_util] [139637140539200] - INFO - running command.raw on 172.23.104.107: rm -f /root/172.23.104.107-20191202-1356-diag.zip [2019-12-02 13:59:13,515] - [remote_util] [139637140539200] - INFO - command executed successfully