warning: Debugger speedups using cython not found. Run '"/Users/lfc/mobile-testkit/venv/bin/python2.7" "/Users/lfc/.p2/pool/plugins/org.python.pydev.core_7.4.0.201910251334/pysrc/setup_cython.py" build_ext --inplace' to build. pydev debugger: starting (pid: 30882) ============================= test session starts ============================== platform darwin -- Python 2.7.17, pytest-3.0.2, py-1.4.31, pluggy-0.3.1 -- /Users/lfc/mobile-testkit/venv/bin/python2.7 cachedir: .cache rootdir: /Users/lfc/mobile-testkit, inifile: pytest.ini plugins: timeout-1.0.0, html-1.10.0 collecting ... collected 70 items testsuites/CBLTester/CBL_Functional_tests/TestSetup_FunctionalTests/test_replication.py::test_replication_configuration_valid_values[10-True-True] {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} Extracted host (172.16.1.201) from url (http://172.16.1.201:4984) Running tests with load balancer disabled Running test with server version 6.5.0-4821 Running test with sync_gateway version 2.7.0-137 Running test with cbl_log_decoder_platform None Running test with cbl_log_decoder_platform None Running test with xattrs for sync meta storage Running with no conflicts Running SG tests using views Running without delta sync {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} Extracted host (172.16.1.205) from url (http://172.16.1.205:8091) Installing Sync Gateway + Couchbase Server + Accels ('di' only) >>> Cluster info: Running test with server version 6.5.0-4821 Running test with sync_gateway version 2.7.0-137 [pool] ma1 ansible_host=172.16.1.205 ma2 ansible_host=172.16.1.201 ma3 ansible_host=172.16.1.196 ma4 ansible_host=172.16.1.84 [couchbase_servers] cb1 ansible_host=172.16.1.205 [sync_gateways] sg1 ansible_host=172.16.1.201 [sg_accels] [load_generators] [load_balancers] [webhook_ip] tf1 ansible_host=172.16.1.138 [environment] cbs_ssl_enabled=False xattrs_enabled=True sg_lb_enabled=False ipv6_enabled=False x509_certs=False delta_sync_enabled=False sync_gateway_ssl=False server_version=6.5.0-4821 sync_gateway_version=2.7.0-137 cbl_log_decoder_platform=None no_conflicts_enabled=True sg_use_views=True sg_platform=centos number_replicas=0 Couchbase Server configuration ------------------------------------------ version: 6.5.0 sync_gateway configuration ------------------------------------------ version: 2.7.0 build number: 137 commit: None config path: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json build flags: skip bucketcreation: False resources/cluster_configs/base_cc cbs: ['http://172.16.1.205:8091'] sgs: [{'ip': u'172.16.1.201', 'name': u'sg1'}] acs: [] ssl: False >>> Provisioning cluster... {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} >>> Server package: http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/mad-hatter/4821/couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm >>> Using sync_gateway config: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json Cleaning cluster: resources/cluster_configs/base_cc Running playbook: libraries/provision/ansible/playbooks/remove-previous-installs.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway] **************************************** skipping: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway rpm] ******************************* changed: [sg1] [WARNING]: Consider using yum, dnf or zypper module rather than running rpm TASK [SYNC GATEWAY | Uninstall sync_gateway deb] ******************************* skipping: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway on Macos] ************************** skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway on Macos] ***************************** skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway service] ****************************** ok: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway binary] ******************************* ok: [sg1] TASK [SYNC GATEWAY | Remove sync gateway source] ******************************* changed: [sg1] [WARNING]: Consider using file module with state=absent rather than running rm TASK [SYNC GATEWAY | Remove sync gateway source .repo directory] *************** changed: [sg1] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** changed: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway] *********************************** skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway install directory] ******************** skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway install directory] ******************** skipping: [sg1] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** skipping: [sg1] TASK [SG ACCEL | Stop sg_accel service] **************************************** fatal: [sg1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sg_accel: cannot check nor set state"} ...ignoring TASK [SG ACCEL | Stop sg_accel] ************************************************ skipping: [sg1] TASK [SG ACCEL | Uninstall sg_accel rpm] *************************************** fatal: [sg1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sg-accel", "delta": "0:00:00.210078", "end": "2019-11-20 12:06:29.959470", "failed": true, "rc": 1, "start": "2019-11-20 12:06:29.749392", "stderr": "error: package couchbase-sg-accel is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [SG ACCEL | Uninstall sg_accel deb] *************************************** skipping: [sg1] TASK [SG ACCEL | Remove sg_accel service] ************************************** ok: [sg1] TASK [SG ACCEL | Delete sg_accel binary] *************************************** ok: [sg1] TASK [SG ACCEL | Remove sg accel source] *************************************** changed: [sg1] TASK [SG ACCEL | Remove sg accel source .repo directory] *********************** changed: [sg1] TASK [SG ACCEL | Remove tmp logging dir (log rotation tests)] ****************** changed: [sg1] TASK [SG ACCEL | Uninstall sync_gateway_accel] ********************************* skipping: [sg1] TASK [SG ACCEL | Delete sync_gateway_accel install directory] ****************** skipping: [sg1] TASK [SG ACCEL | Delete sync_gateway_accel install directory] ****************** skipping: [sg1] TASK [SG ACCEL | Remove tmp logging dir (log rotation tests)] ****************** skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway user on Centos] *********************** changed: [sg1] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [sg1] TASK [NGINX | stop nginx service] ********************************************** fatal: [sg1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service nginx: cannot check nor set state"} ...ignoring TASK [NGINX | Remove nginx package] ******************************************** ok: [sg1] TASK [ALL | verify no sg_accel or sgs are running on 4985] ********************* ok: [sg1] PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] TASK [Stop Couchbase Service] ************************************************** changed: [cb1] TASK [COUCHBASE SERVER | Try to stop Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server on CentOS] ***************** changed: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server on Ubuntu] ***************** skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server on Ubuntu] ********************* skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server community on CentOS] ******* fatal: [cb1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server-community", "delta": "0:00:00.200140", "end": "2019-11-20 12:06:49.121284", "failed": true, "rc": 1, "start": "2019-11-20 12:06:48.921144", "stderr": "error: package couchbase-server-community is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server community on Ubuntu] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server community on Ubuntu] *********** skipping: [cb1] TASK [COUCHBASE SERVER | Kill epmd] ******************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill epmd", "delta": "0:00:00.204727", "end": "2019-11-20 12:06:50.096273", "failed": true, "rc": 1, "start": "2019-11-20 12:06:49.891546", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill beam.smp] **************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill beam.smp", "delta": "0:00:00.197033", "end": "2019-11-20 12:06:50.700861", "failed": true, "rc": 1, "start": "2019-11-20 12:06:50.503828", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill memcached] *************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill memcached", "delta": "0:00:00.185508", "end": "2019-11-20 12:06:51.282868", "failed": true, "rc": 1, "start": "2019-11-20 12:06:51.097360", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Remove all couchbase server residue] ****************** changed: [cb1] TASK [NGINX | stop nginx service] ********************************************** fatal: [cb1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service nginx: cannot check nor set state"} ...ignoring TASK [NGINX | Remove nginx package] ******************************************** ok: [cb1] TASK [ALL | verify no servers not listening on port 8081] ********************** ok: [cb1] PLAY RECAP ********************************************************************* cb1 : ok=11 changed=7 unreachable=0 failed=0 sg1 : ok=20 changed=9 unreachable=0 failed=0 Running playbook: libraries/provision/ansible/playbooks/flush-firewall.yml PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] TASK [OS | Flush Firewall] ***************************************************** changed: [cb1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [OS | Flush Firewall] ***************************************************** changed: [sg1] PLAY RECAP ********************************************************************* cb1 : ok=2 changed=1 unreachable=0 failed=0 sg1 : ok=2 changed=1 unreachable=0 failed=0 Installing Couchbase Server resources/cluster_configs/base_cc Couchbase Server configuration ------------------------------------------ version: 6.5.0 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} >>> Installing Couchbase Server Running playbook: libraries/provision/ansible/playbooks/install-couchbase-server-package.yml PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase server primary node 172.16.1.205" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase server node 172.16.1.205" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Downloading Couchbase server v. http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/mad-hatter/4821/couchbase-server-enterprise-6.5.0-4821-centos7.x86_64.rpm" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "IPv6 Enabled - False" } TASK [Stop Couchbase Service] ************************************************** fatal: [cb1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service couchbase-server: cannot check nor set state"} ...ignoring TASK [COUCHBASE SERVER | Try to stop Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server on CentOS] ***************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server", "delta": "0:00:00.200311", "end": "2019-11-20 12:07:12.889572", "failed": true, "rc": 1, "start": "2019-11-20 12:07:12.689261", "stderr": "error: package couchbase-server is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server on Ubuntu] ***************** skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server on Ubuntu] ********************* skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server community on CentOS] ******* fatal: [cb1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server-community", "delta": "0:00:00.198069", "end": "2019-11-20 12:07:13.877297", "failed": true, "rc": 1, "start": "2019-11-20 12:07:13.679228", "stderr": "error: package couchbase-server-community is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server community on Ubuntu] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server community on Ubuntu] *********** skipping: [cb1] TASK [COUCHBASE SERVER | Kill epmd] ******************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill epmd", "delta": "0:00:00.186654", "end": "2019-11-20 12:07:14.861671", "failed": true, "rc": 1, "start": "2019-11-20 12:07:14.675017", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill beam.smp] **************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill beam.smp", "delta": "0:00:00.186120", "end": "2019-11-20 12:07:15.448269", "failed": true, "rc": 1, "start": "2019-11-20 12:07:15.262149", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill memcached] *************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill memcached", "delta": "0:00:00.191865", "end": "2019-11-20 12:07:16.059523", "failed": true, "rc": 1, "start": "2019-11-20 12:07:15.867658", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Remove all couchbase server residue] ****************** ok: [cb1] TASK [SYNC GATEWAY | Remove sync_gateway user on Centos] *********************** ok: [cb1] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [cb1] TASK [COUCHBASE SERVER | Download couchbase server] **************************** ok: [cb1] TASK [Install Couchbase Server on CentOS] ************************************** changed: [cb1] TASK [Install Couchbase Server on Ubuntu] ************************************** skipping: [cb1] TASK [Starting CBS for Ubuntu] ************************************************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Stopping CBS] ************************************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Enabling IPv6 in static_config] ******************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Deleting config.dat] ****************************** skipping: [cb1] TASK [Enabling CBS on IPV6 | Starting CBS] ************************************* skipping: [cb1] TASK [COUCHBASE SERVER | raise max file descriptors] *************************** ok: [cb1] TASK [COUCHBASE SERVER | Enabled Transparent Huge Pages] *********************** changed: [cb1] TASK [COUCHBASE SERVER | Disable Transparent Huge Pages] *********************** changed: [cb1] TASK [COUCHBASE SERVER | Enabled Transparent Huge Pages again] ***************** changed: [cb1] TASK [COUCHBASE SERVER | Start Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [cb1] TASK [COUCHBASE SERVER | Restart Couchbase Service] **************************** changed: [cb1] TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase cluster RAM 958" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase bucket RAM 479" } TASK [COUCHBASE SERVER | Get major version] ************************************ changed: [cb1] TASK [COUCHBASE SERVER | Get full version] ************************************* changed: [cb1] TASK [COUCHBASE SERVER | Wait for node to be listening on port 8091] *********** ok: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (3.0.X and 3.1.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.0.X and 4.1.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.5.X and 4.6.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.7.X and up) | configure IPv4] *** changed: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.7.X and up) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (3.x only)] ******************* skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (4.X and 5.X) | configure IPv4] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (4.X and 5.X) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (6.X and up) | configure IPv4] changed: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (6.X and up)| configure IPv6] * skipping: [cb1] TASK [COUCHBASE SERVER | Wait for node to be listening on port 8091] *********** ok: [cb1] TASK [set_fact] **************************************************************** skipping: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv4] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv6] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (6.X and up) | configure IPv4] ****** changed: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (6.X and up) | configure IPv6] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (3.X to 5.X) | configure IPv4] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (3.X to 5.X) | configure IPv6] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (6.X and up) | configure IPv4] *** changed: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (6.X and up) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (3.X to 5.X) | configure IPv4] *** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (3.X to 5.X) | configure IPv6] *** skipping: [cb1] PLAY RECAP ********************************************************************* cb1 : ok=31 changed=16 unreachable=0 failed=0 >>> Waiting for server to be in 'healthy' state GET http://172.16.1.205:8091/pools/nodes 200 All nodes are healthy Installing Sync Gateway sync_gateway configuration ------------------------------------------ version: 2.7.0 build number: 137 commit: None config path: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json build flags: skip bucketcreation: False resources/cluster_configs/base_cc cbs: ['http://172.16.1.205:8091'] sgs: [{'ip': u'172.16.1.201', 'name': u'sg1'}] acs: [] ssl: False {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} GET http://172.16.1.205:8091/pools/default/buckets 200 Found buckets: [] GET http://172.16.1.205:8091/pools/default/buckets 200 Found buckets: [] Creating buckets: [u'travel-sample'] total_avail_ram_mb: 1838 effective_avail_ram_mb: 958 Creating bucket travel-sample with RAM 958 GET http://Administrator:password@172.16.1.205:8091/pools 200 POST http://172.16.1.205:8091/pools/default/buckets 202 Creating RBAC user travel-sample with password password and roles ro_admin,bucket_full_access[travel-sample] PUT http://172.16.1.205:8091/settings/rbac/users/local/travel-sample 200 Key not found error: Bucket is ready! GET http://172.16.1.205:8091/pools/nodes 200 All nodes are healthy Keyerror in getting loggingredactlevel revs_limit not found in resources/cluster_configs/base_cc, Ignoring Running playbook: libraries/provision/ansible/playbooks/install-sync-gateway-package.yml [WARNING]: While constructing a mapping from /Users/lfc/mobile- testkit/libraries/provision/ansible/playbooks/install-sync-gateway-package.yml, line 63, column 3, found a duplicate dict key (vars). Using last defined value only. PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** fatal: [sg1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sync_gateway: cannot check nor set state"} ...ignoring TASK [SYNC GATEWAY | Stop sync_gateway] **************************************** skipping: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway rpm] ******************************* fatal: [sg1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sync-gateway", "delta": "0:00:00.205637", "end": "2019-11-20 12:08:39.534462", "failed": true, "rc": 1, "start": "2019-11-20 12:08:39.328825", "stderr": "error: package couchbase-sync-gateway is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [SYNC GATEWAY | Uninstall sync_gateway deb] ******************************* skipping: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway on Macos] ************************** skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway on Macos] ***************************** skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway service] ****************************** ok: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway binary] ******************************* ok: [sg1] TASK [SYNC GATEWAY | Remove sync gateway source] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Remove sync gateway source .repo directory] *************** changed: [sg1] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** changed: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway user on Centos] *********************** ok: [sg1] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [sg1] TASK [SYNC GATEWAY | verify no service on 4985] ******************************** ok: [sg1] PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Create sync gateway user on Centos/linux/ubuntu/debian] *** changed: [sg1] TASK [SYNC GATEWAY | Add sync gateway user on MacOS] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Create /tmp/sg_logs] ************************************** changed: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Download sync_gateway rpm http://latestbuilds.service.couchbase.com/builds/latestbuilds/sync_gateway/2.7.0/137/couchbase-sync-gateway-enterprise_2.7.0-137_x86_64.rpm] *** ok: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Install sync_gateway rpm] ********************************* changed: [sg1] TASK [SYNC GATEWAY | Install sync_gateway deb] ********************************* skipping: [sg1] TASK [SYNC GATEWAY | Install sync_gateway tar.gz] ****************************** skipping: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway for CentOS 6] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway on MacOS] ******************************* skipping: [sg1] TASK [SYNC GATEWAY | Verify sync_gateway not listening on port] **************** ok: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs] *************************************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* changed: [sg1] TASK [Deleting /home/sync_gateway/certs directory] ***************************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs.zip] *********************************** skipping: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config] ************************************ changed: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ ok: [sg1] => { "out.stdout": { "adminInterface": "0.0.0.0:4985", "compressResponses": false, "databases": { "db": { "allow_conflicts": false, "bucket": "travel-sample", "enable_shared_bucket_access": true, "import_docs": "continuous", "password": "password", "server": "http://172.16.1.205:8091", "use_views": true, "username": "travel-sample" } }, "interface": ":4984", "logging": { "debug": { "enabled": true } }, "maxFileDescriptors": 90000, "maxIncomingConnections": 0 } } TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** changed: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** changed: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [debug] ******************************************************************* ok: [sg1] => { "msg": "Starting sync_gateway" } TASK [SYNC GATEWAY | Start sync gateway service] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** ok: [sg1] PLAY RECAP ********************************************************************* cb1 : ok=1 changed=0 unreachable=0 failed=0 sg1 : ok=30 changed=13 unreachable=0 failed=0 Running playbook: libraries/provision/ansible/playbooks/install-sg-accel-package.yml PLAY RECAP ********************************************************************* Running playbook: libraries/provision/ansible/playbooks/configure-sync-gateway-awslogs-forwarder.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | upload default awslogs config] **************************** fatal: [sg1]: FAILED! => {"changed": false, "checksum": "cc05bb8b42d0f7241ed333b559ff0d77554e66b2", "failed": true, "msg": "Destination directory /var/awslogs/etc does not exist"} ...ignoring TASK [SYNC GATEWAY | restart awslogs service] ********************************** fatal: [sg1]: FAILED! => {"changed": true, "cmd": "service awslogs restart", "delta": "0:00:00.228152", "end": "2019-11-20 12:09:36.465111", "failed": true, "rc": 5, "start": "2019-11-20 12:09:36.236959", "stderr": "Redirecting to /bin/systemctl restart awslogs.service\nFailed to restart awslogs.service: Unit not found.", "stdout": "", "stdout_lines": [], "warnings": ["Consider using service module rather than running service"]} ...ignoring PLAY RECAP ********************************************************************* sg1 : ok=3 changed=1 unreachable=0 failed=0 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} Upstream definition: server 172.16.1.201:4984; Upstream definition admin: server 172.16.1.201:4985; Running playbook: libraries/provision/ansible/playbooks/install-nginx.yml PLAY RECAP ********************************************************************* >>> Done provisioning cluster... Verfying versions for cluster: resources/cluster_configs/base_cc GET http://Administrator:password@172.16.1.205:8091/pools 200 Expected Server Version: 6.5.0-4821 Running Server Version: 6.5.0-4821 GET http://172.16.1.201:4984/ 200 'server' header: Couchbase Sync Gateway/2.7.0 EE vendor name: Couchbase Sync Gateway GET http://172.16.1.201:4984/ 200 Expected sync_gateway Version: 2.7.0-137 Running sync_gateway Version: 2.7.0-137 Running sync_gateway Vendor Version: 2.7 Verfying versions for cluster: resources/cluster_configs/base_cc GET http://Administrator:password@172.16.1.205:8091/pools 200 Expected Server Version: 6.5.0-4821 Running Server Version: 6.5.0-4821 GET http://172.16.1.201:4984/ 200 'server' header: Couchbase Sync Gateway/2.7.0 EE vendor name: Couchbase Sync Gateway GET http://172.16.1.201:4984/ 200 Expected sync_gateway Version: 2.7.0-137 Running sync_gateway Version: 2.7.0-137 Running sync_gateway Vendor Version: 2.7 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'172.16.1.205', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'172.16.1.201', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sg_use_views': True, u'no_conflicts_enabled': True, u'sg_lb_enabled': False, u'sync_gateway_ssl': False, u'cbl_log_decoder_platform': None, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.7.0-137', u'ipv6_enabled': False, u'sg_platform': u'centos', u'cbs_ssl_enabled': False, u'server_version': u'6.5.0-4821'}, u'hosts': [{u'ip': u'172.16.1.205', u'name': u'host1'}, {u'ip': u'172.16.1.201', u'name': u'host2'}, {u'ip': u'172.16.1.196', u'name': u'host3'}, {u'ip': u'172.16.1.84', u'name': u'host4'}], u'load_balancers': [], u'sg_accels': []} Running test 'test_replication_configuration_valid_values[10-True-True]' cluster_config: resources/cluster_configs/base_cc cluster_topology: {'load_balancers': [], 'sg_accels': [], 'sync_gateways': [{'admin': 'http://172.16.1.201:4985', 'public': 'http://172.16.1.201:4984'}], 'couchbase_servers': ['http://172.16.1.205:8091']} mode: cc xattrs_enabled: True Got response: @1_172.16.1.138_iOS Log files available at - /Users/lfc/Library/Developer/CoreSimulator/Devices/C6333739-DE9F-4DA7-9FDF-43B574092A11/data/Containers/Data/Application/EC3FBA43-A16D-44BD-99A8-84AD7B1254F5/logs1574231985.050415 Creating a Database cbl-test1574231985.13 at test setup Got response: @2_172.16.1.138_iOS Got response: @3_172.16.1.138_iOS Getting the database name Got response: "cbl-test1574231985.13" resources/cluster_configs/base_cc cbs: ['http://172.16.1.205:8091'] sgs: [{'ip': u'172.16.1.201', 'name': u'sg1'}] acs: [] ssl: False >>> Reseting cluster ... >>> CBS SSL enabled: False >>> Using xattrs: True >>> Stopping sync_gateway Running playbook: libraries/provision/ansible/playbooks/stop-sync-gateway.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway for CentOS 6] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway on MacOS] ******************************* skipping: [sg1] TASK [SYNC GATEWAY | Verify sync_gateway not listening on port] **************** ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service on Windows] ********************* skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=3 changed=1 unreachable=0 failed=0 >>> Stopping sg_accel Running playbook: libraries/provision/ansible/playbooks/stop-sg-accel.yml PLAY RECAP ********************************************************************* >>> Deleting sync_gateway artifacts Running playbook: libraries/provision/ansible/playbooks/delete-sync-gateway-artifacts.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs] ********************************* changed: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs] ********************************* skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Create sync_gateway logs directory on Windows] ************ skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs directory under Temp on Windows] * skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Create sync_gateway logs directory under c:/tmp on Windows] *** skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=2 changed=1 unreachable=0 failed=0 >>> Deleting sg_accel artifacts Running playbook: libraries/provision/ansible/playbooks/delete-sg-accel-artifacts.yml PLAY RECAP ********************************************************************* >>> Deleting buckets on: http://172.16.1.205:8091 GET http://172.16.1.205:8091/pools/default/buckets 200 Found buckets: [u'travel-sample'] Existing buckets: [u'travel-sample'] Deleting buckets: [u'travel-sample'] GET http://Administrator:password@172.16.1.205:8091/pools 200 Deleting RBAC user travel-sample DELETE http://172.16.1.205:8091/settings/rbac/users/local/travel-sample 200 DELETE http://172.16.1.205:8091/pools/default/buckets/travel-sample 200 GET http://172.16.1.205:8091/pools/default/buckets 200 Found buckets: [] >>> Creating buckets on: http://172.16.1.205:8091 >>> Creating buckets [u'travel-sample'] Creating buckets: [u'travel-sample'] total_avail_ram_mb: 1838 effective_avail_ram_mb: 958 Creating bucket travel-sample with RAM 958 GET http://Administrator:password@172.16.1.205:8091/pools 200 POST http://172.16.1.205:8091/pools/default/buckets 202 Creating RBAC user travel-sample with password password and roles admin PUT http://172.16.1.205:8091/settings/rbac/users/local/travel-sample 200 Key not found error: Bucket is ready! GET http://172.16.1.205:8091/pools/nodes 200 All nodes are healthy >>> Waiting for Server: http://172.16.1.205:8091 to be in a healthy state GET http://172.16.1.205:8091/pools/nodes 200 All nodes are healthy >>> Starting sync_gateway with configuration: /Users/lfc/mobile-testkit/resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json Keyerror in getting loggingredactlevel ./gen_keystore.sh 172.16.1.205 travel-sample Generate ROOT CA Generate Intermediate Generate RSA Get node IPs Loop through nodes Setup Certificate for 172.16.1.205 Adding ROOT CA Adding Intermediate Signature ok subject=/C=UA/O=My Company/CN=travel-sample Getting CA Private Key Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 360 days for: CN=travel-sample, OU=None, O=None, L=None, ST=None, C=US Signature ok subject=/C=US/ST=None/L=None/O=None/OU=None/CN=travel-sample Getting CA Private Key Certificate was added to keystore Certificate was added to keystore Certificate reply was installed in keystore scp certs.zip root@172.16.1.201:/tmp revs_limit not found in resources/cluster_configs/base_cc, Ignoring Running playbook: libraries/provision/ansible/playbooks/start-sync-gateway.yml [WARNING]: While constructing a mapping from /Users/lfc/mobile- testkit/libraries/provision/ansible/playbooks/start-sync-gateway.yml, line 23, column 5, found a duplicate dict key (sudo). Using last defined value only. [DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* changed: [sg1] TASK [Deleting /home/sync_gateway/certs directory] ***************************** ok: [sg1] TASK [Deleting /home/sync_gateway/certs.zip] *********************************** ok: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** changed: [sg1] TASK [Unzipping certs.zip] ***************************************************** changed: [sg1] [WARNING]: Consider using unarchive module rather than running unzip TASK [SYNC GATEWAY | Check deployed config] ************************************ changed: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ ok: [sg1] => { "out.stdout": { "adminInterface": "0.0.0.0:4985", "compressResponses": false, "databases": { "db": { "allow_conflicts": false, "bucket": "travel-sample", "cacertpath": "/home/sync_gateway/certs/ca.pem", "certpath": "/home/sync_gateway/certs/chain.pem", "enable_shared_bucket_access": true, "import_docs": "continuous", "keypath": "/home/sync_gateway/certs/pkey.key", "server": "couchbases://172.16.1.205:", "use_views": true } }, "interface": ":4984", "logging": { "debug": { "enabled": true } }, "maxFileDescriptors": 90000, "maxIncomingConnections": 0 } } TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** changed: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* skipping: [sg1] TASK [Deleting /Users/sync_gateway/certs directory] **************************** skipping: [sg1] TASK [Deleting /Users/sync_gateway/certs.zip] ********************************** skipping: [sg1] TASK [Copying certs.zip to /Users/sync_gateway directory] ********************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config] ************************************ skipping: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ skipping: [sg1] TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** skipping: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** skipping: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~2\Couchbase is present on Windows] ***** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~2\Couchbase] ******** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~2\Couchbase\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~1\Couchbase\Sync Gateway is present on Windows] *** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~1\Couchbase\Sync Gateway] *** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~1\Couchbase\Sync Gateway\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ skipping: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [debug] ******************************************************************* ok: [sg1] => { "msg": "Starting sync_gateway" } TASK [SYNC GATEWAY | Start sync gateway service] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] ****************