(venv) eunices-mbp:mobile-testkit eunicehuang$ pytest -s -rsx -k test_default_conflict_withConflicts_and_sgOffline --timeout 1800 --liteserv-version=2.5.0-190 --liteserv-host=localhost --liteserv-port=8080 --xattrs --sg-ssl --sync-gateway-version=2.5.0-271 --mode=cc --server-version=6.0.1-2037 --liteserv-platform=xamarin-ios --create-db-per-test=cbl-test testsuites/CBLTester/CBL_Functional_tests/TestSetup_FunctionalTests ========================================== test session starts =========================================== platform darwin -- Python 2.7.10, pytest-3.0.2, py-1.4.31, pluggy-0.3.1 -- /Users/eunicehuang/dev/mobile-testkit/venv/bin/python cachedir: .cache rootdir: /Users/eunicehuang/dev/mobile-testkit, inifile: pytest.ini plugins: timeout-1.0.0, html-1.10.0 collected 182 items testsuites/CBLTester/CBL_Functional_tests/TestSetup_FunctionalTests/test_replication.py::test_default_conflict_withConflicts_and_sgOffline Downloading TestServer ... {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': False, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} Extracted host (192.168.33.11) from url (http://192.168.33.11:4984) Enabling SSL on sync gateway Running tests with load balancer disabled Running test with server version 6.0.1-2037 Running test with sync_gateway version 2.5.0-271 Running test with xattrs for sync meta storage Running with allow conflicts Running tests with cbs <-> sg ssl disabled Running without delta sync {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} Extracted host (192.168.33.10) from url (http://192.168.33.10:8091) Installing Sync Gateway + Couchbase Server + Accels ('di' only) >>> Cluster info: Running test with server version 6.0.1-2037 Running test with sync_gateway version 2.5.0-271 [pool] ma1 ansible_host=192.168.33.10 ma2 ansible_host=192.168.33.11 [couchbase_servers] cb1 ansible_host=192.168.33.10 [sync_gateways] sg1 ansible_host=192.168.33.11 [sg_accels] [load_generators] [load_balancers] [webhook_ip] tf1 ansible_host=10.100.172.185 [environment] cbs_ssl_enabled=False xattrs_enabled=True sg_lb_enabled=False ipv6_enabled=False x509_certs=False delta_sync_enabled=False sync_gateway_ssl=True server_version=6.0.1-2037 sync_gateway_version=2.5.0-271 no_conflicts_enabled=False sg_use_views=False number_replicas=0 Couchbase Server configuration ------------------------------------------ version: 6.0.1 sync_gateway configuration ------------------------------------------ version: 2.5.0 build number: 271 commit: None config path: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json build flags: skip bucketcreation: False resources/cluster_configs/base_cc cbs: ['http://192.168.33.10:8091'] sgs: [{'ip': u'192.168.33.11', 'name': u'sg1'}] acs: [] ssl: False >>> Provisioning cluster... {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} >>> Server package: http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/alice/2037/couchbase-server-enterprise-6.0.1-2037-centos7.x86_64.rpm >>> Using sync_gateway config: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json Cleaning cluster: resources/cluster_configs/base_cc Running playbook: libraries/provision/ansible/playbooks/remove-previous-installs.yml PLAY [pool] ******************************************************************** TASK [setup] ******************************************************************* ok: [ma2] ok: [ma1] TASK [Stop Couchbase Service] ************************************************** fatal: [ma2]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service couchbase-server: cannot check nor set state"} ...ignoring changed: [ma1] TASK [COUCHBASE SERVER | Try to stop Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [ma1] skipping: [ma2] TASK [COUCHBASE SERVER | Uninstall couchbase server on CentOS] ***************** fatal: [ma2]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server", "delta": "0:00:00.159424", "end": "2019-04-16 23:30:09.623468", "failed": true, "rc": 1, "start": "2019-04-16 23:30:09.464044", "stderr": "error: package couchbase-server is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring changed: [ma1] [WARNING]: Consider using yum, dnf or zypper module rather than running rpm TASK [COUCHBASE SERVER | Uninstall couchbase server on Ubuntu] ***************** skipping: [ma1] skipping: [ma2] TASK [COUCHBASE SERVER | Purge couchbase server on Ubuntu] ********************* skipping: [ma1] skipping: [ma2] TASK [COUCHBASE SERVER | Uninstall couchbase server community on CentOS] ******* fatal: [ma2]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server-community", "delta": "0:00:00.142728", "end": "2019-04-16 23:30:10.760001", "failed": true, "rc": 1, "start": "2019-04-16 23:30:10.617273", "stderr": "error: package couchbase-server-community is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring fatal: [ma1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server-community", "delta": "0:00:00.143041", "end": "2019-04-16 23:30:10.760875", "failed": true, "rc": 1, "start": "2019-04-16 23:30:10.617834", "stderr": "error: package couchbase-server-community is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server community on Ubuntu] ******* skipping: [ma1] skipping: [ma2] TASK [COUCHBASE SERVER | Purge couchbase server community on Ubuntu] *********** skipping: [ma1] skipping: [ma2] TASK [COUCHBASE SERVER | Kill epmd] ******************************************** fatal: [ma1]: FAILED! => {"changed": true, "cmd": "pkill epmd", "delta": "0:00:00.122642", "end": "2019-04-16 23:30:11.254924", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.132282", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring fatal: [ma2]: FAILED! => {"changed": true, "cmd": "pkill epmd", "delta": "0:00:00.122289", "end": "2019-04-16 23:30:11.255828", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.133539", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill beam.smp] **************************************** fatal: [ma1]: FAILED! => {"changed": true, "cmd": "pkill beam.smp", "delta": "0:00:00.124757", "end": "2019-04-16 23:30:11.664518", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.539761", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring fatal: [ma2]: FAILED! => {"changed": true, "cmd": "pkill beam.smp", "delta": "0:00:00.125007", "end": "2019-04-16 23:30:11.666012", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.541005", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill memcached] *************************************** fatal: [ma1]: FAILED! => {"changed": true, "cmd": "pkill memcached", "delta": "0:00:00.124951", "end": "2019-04-16 23:30:12.075660", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.950709", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring fatal: [ma2]: FAILED! => {"changed": true, "cmd": "pkill memcached", "delta": "0:00:00.127278", "end": "2019-04-16 23:30:12.081535", "failed": true, "rc": 1, "start": "2019-04-16 23:30:11.954257", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Remove all couchbase server residue] ****************** ok: [ma2] changed: [ma1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** fatal: [ma1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sync_gateway: cannot check nor set state"} ...ignoring changed: [ma2] TASK [SYNC GATEWAY | Stop sync_gateway] **************************************** skipping: [ma1] skipping: [ma2] TASK [SYNC GATEWAY | Uninstall sync_gateway rpm] ******************************* fatal: [ma1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sync-gateway", "delta": "0:00:00.139915", "end": "2019-04-16 23:30:13.521045", "failed": true, "rc": 1, "start": "2019-04-16 23:30:13.381130", "stderr": "error: package couchbase-sync-gateway is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring changed: [ma2] TASK [SYNC GATEWAY | Uninstall sync_gateway deb] ******************************* skipping: [ma1] skipping: [ma2] TASK [SYNC GATEWAY | Remove sync_gateway service] ****************************** ok: [ma1] ok: [ma2] TASK [SYNC GATEWAY | Delete sync_gateway binary] ******************************* ok: [ma1] ok: [ma2] TASK [SYNC GATEWAY | Remove sync gateway source] ******************************* changed: [ma1] [WARNING]: Consider using file module with state=absent rather than running rm changed: [ma2] TASK [SYNC GATEWAY | Remove sync gateway source .repo directory] *************** changed: [ma1] changed: [ma2] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** changed: [ma1] changed: [ma2] TASK [SYNC GATEWAY | Uninstall sync_gateway] *********************************** skipping: [ma1] skipping: [ma2] TASK [SYNC GATEWAY | Delete sync_gateway install directory] ******************** skipping: [ma1] skipping: [ma2] TASK [SYNC GATEWAY | Delete sync_gateway install directory] ******************** skipping: [ma1] skipping: [ma2] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Stop sg_accel service] **************************************** fatal: [ma1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sg_accel: cannot check nor set state"} ...ignoring fatal: [ma2]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sg_accel: cannot check nor set state"} ...ignoring TASK [SG ACCEL | Stop sg_accel] ************************************************ skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Uninstall sg_accel rpm] *************************************** fatal: [ma2]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sg-accel", "delta": "0:00:00.139363", "end": "2019-04-16 23:30:16.820544", "failed": true, "rc": 1, "start": "2019-04-16 23:30:16.681181", "stderr": "error: package couchbase-sg-accel is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring fatal: [ma1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sg-accel", "delta": "0:00:00.142317", "end": "2019-04-16 23:30:16.825732", "failed": true, "rc": 1, "start": "2019-04-16 23:30:16.683415", "stderr": "error: package couchbase-sg-accel is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [SG ACCEL | Uninstall sg_accel deb] *************************************** skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Remove sg_accel service] ************************************** ok: [ma2] ok: [ma1] TASK [SG ACCEL | Delete sg_accel binary] *************************************** ok: [ma2] ok: [ma1] TASK [SG ACCEL | Remove sg accel source] *************************************** changed: [ma2] changed: [ma1] TASK [SG ACCEL | Remove sg accel source .repo directory] *********************** changed: [ma2] changed: [ma1] TASK [SG ACCEL | Remove tmp logging dir (log rotation tests)] ****************** changed: [ma2] changed: [ma1] TASK [SG ACCEL | Uninstall sync_gateway_accel] ********************************* skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Delete sync_gateway_accel install directory] ****************** skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Delete sync_gateway_accel install directory] ****************** skipping: [ma1] skipping: [ma2] TASK [SG ACCEL | Remove tmp logging dir (log rotation tests)] ****************** skipping: [ma1] skipping: [ma2] TASK [NGINX | stop nginx service] ********************************************** fatal: [ma1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service nginx: cannot check nor set state"} ...ignoring fatal: [ma2]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service nginx: cannot check nor set state"} ...ignoring TASK [NGINX | Remove nginx package] ******************************************** ok: [ma2] ok: [ma1] TASK [SYNC GATEWAY | Remove sync_gateway user] ********************************* ok: [ma1] changed: [ma2] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [ma2] ok: [ma1] TASK [ALL | verify no sg_accel or sgs are running on 4985] ********************* ok: [ma2] ok: [ma1] TASK [ALL | verify no servers not listening on port 8081] ********************** ok: [ma1] ok: [ma2] PLAY RECAP ********************************************************************* ma1 : ok=28 changed=15 unreachable=0 failed=0 ma2 : ok=28 changed=15 unreachable=0 failed=0 Running playbook: libraries/provision/ansible/playbooks/flush-firewall.yml PLAY [pool] ******************************************************************** TASK [setup] ******************************************************************* ok: [ma2] ok: [ma1] TASK [OS | Flush Firewall] ***************************************************** changed: [ma1] changed: [ma2] PLAY RECAP ********************************************************************* ma1 : ok=2 changed=1 unreachable=0 failed=0 ma2 : ok=2 changed=1 unreachable=0 failed=0 Installing Couchbase Server resources/cluster_configs/base_cc Couchbase Server configuration ------------------------------------------ version: 6.0.1 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} >>> Installing Couchbase Server Running playbook: libraries/provision/ansible/playbooks/install-couchbase-server-package.yml [WARNING]: While constructing a mapping from /Users/eunicehuang/dev/mobile- testkit/libraries/provision/ansible/playbooks/install-couchbase-server-package.yml, line 80, column 7, found a duplicate dict key (shell). Using last defined value only. PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase server primary node 192.168.33.10" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase server node 192.168.33.10" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Downloading Couchbase server v. http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/alice/2037/couchbase-server-enterprise-6.0.1-2037-centos7.x86_64.rpm" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "IPv6 Enabled - False" } TASK [Stop Couchbase Service] ************************************************** ok: [cb1] TASK [COUCHBASE SERVER | Try to stop Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server on CentOS] ***************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server", "delta": "0:00:00.138661", "end": "2019-04-16 23:30:29.120062", "failed": true, "rc": 1, "start": "2019-04-16 23:30:28.981401", "stderr": "error: package couchbase-server is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server on Ubuntu] ***************** skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server on Ubuntu] ********************* skipping: [cb1] TASK [COUCHBASE SERVER | Uninstall couchbase server community on CentOS] ******* fatal: [cb1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-server-community", "delta": "0:00:00.137316", "end": "2019-04-16 23:30:29.592736", "failed": true, "rc": 1, "start": "2019-04-16 23:30:29.455420", "stderr": "error: package couchbase-server-community is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [COUCHBASE SERVER | Uninstall couchbase server community on Ubuntu] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Purge couchbase server community on Ubuntu] *********** skipping: [cb1] TASK [COUCHBASE SERVER | Kill epmd] ******************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill epmd", "delta": "0:00:00.121773", "end": "2019-04-16 23:30:30.032405", "failed": true, "rc": 1, "start": "2019-04-16 23:30:29.910632", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill beam.smp] **************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill beam.smp", "delta": "0:00:00.122075", "end": "2019-04-16 23:30:30.394000", "failed": true, "rc": 1, "start": "2019-04-16 23:30:30.271925", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Kill memcached] *************************************** fatal: [cb1]: FAILED! => {"changed": true, "cmd": "pkill memcached", "delta": "0:00:00.127409", "end": "2019-04-16 23:30:30.758743", "failed": true, "rc": 1, "start": "2019-04-16 23:30:30.631334", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []} ...ignoring TASK [COUCHBASE SERVER | Remove all couchbase server residue] ****************** ok: [cb1] TASK [SYNC GATEWAY | Remove sync_gateway user] ********************************* ok: [cb1] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [cb1] TASK [COUCHBASE SERVER | Download couchbase server] **************************** ok: [cb1] TASK [Install Couchbase Server on CentOS] ************************************** changed: [cb1] TASK [Install Couchbase Server on Ubuntu] ************************************** skipping: [cb1] TASK [Starting CBS for Ubuntu] ************************************************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Stopping CBS] ************************************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Enabling IPv6 in static_config] ******************* skipping: [cb1] TASK [Enabling CBS on IPV6 | Deleting config.dat] ****************************** skipping: [cb1] TASK [Enabling CBS on IPV6 | Starting CBS] ************************************* skipping: [cb1] TASK [COUCHBASE SERVER | raise max file descriptors] *************************** ok: [cb1] TASK [COUCHBASE SERVER | Disable Transparent Huge Pages] *********************** changed: [cb1] TASK [COUCHBASE SERVER | Start Couchbase Service (Hack around 7.2 and CBS 4.1.0 issue)] *** skipping: [cb1] TASK [COUCHBASE SERVER | Restart Couchbase Service] **************************** changed: [cb1] TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase cluster RAM 2519" } TASK [debug] ******************************************************************* ok: [cb1] => { "msg": "Couchbase bucket RAM 1259" } TASK [COUCHBASE SERVER | Get major version] ************************************ changed: [cb1] TASK [COUCHBASE SERVER | Wait for node to be listening on port 8091] *********** ok: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (3.0.X and 3.1.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.0.X and 4.1.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.5.X and 4.6.X)] ********* skipping: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.7.X and up) | configure IPv4] *** changed: [cb1] TASK [COUCHBASE SERVER | Configure cluster settings (4.7.X and up) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (3.x only)] ******************* skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (4.X and 5.X) | configure IPv4] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (4.X and 5.X) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (6.X and up) | configure IPv4] changed: [cb1] TASK [COUCHBASE SERVER | Initialize primary node (6.X and up)| configure IPv6] * skipping: [cb1] TASK [COUCHBASE SERVER | Wait for node to be listening on port 8091] *********** ok: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv4] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv6] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv4] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Join additional cluster nodes | configure IPv6] ******* skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (6.X and up) | configure IPv4] ****** changed: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (6.X and up) | configure IPv6] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (3.X to 5.X) | configure IPv4] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Rebalance cluster (3.X to 5.X) | configure IPv6] ****** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (6.X and up) | configure IPv4] *** changed: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (6.X and up) | configure IPv6] *** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (3.X to 5.X) | configure IPv4] *** skipping: [cb1] TASK [COUCHBASE SERVER | Enable auto failover (3.X to 5.X) | configure IPv6] *** skipping: [cb1] PLAY RECAP ********************************************************************* cb1 : ok=28 changed=13 unreachable=0 failed=0 >>> Waiting for server to be in 'healthy' state GET http://192.168.33.10:8091/pools/nodes 200 All nodes are healthy Installing Sync Gateway sync_gateway configuration ------------------------------------------ version: 2.5.0 build number: 271 commit: None config path: resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json build flags: skip bucketcreation: False resources/cluster_configs/base_cc cbs: ['http://192.168.33.10:8091'] sgs: [{'ip': u'192.168.33.11', 'name': u'sg1'}] acs: [] ssl: False {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} GET http://192.168.33.10:8091/pools/default/buckets 200 Found buckets: [] GET http://192.168.33.10:8091/pools/default/buckets 200 Found buckets: [] Creating buckets: [u'travel-sample'] total_avail_ram_mb: 3789 effective_avail_ram_mb: 2519 Creating bucket travel-sample with RAM 2519 GET http://Administrator:password@192.168.33.10:8091/pools 200 POST http://192.168.33.10:8091/pools/default/buckets 202 Creating RBAC user travel-sample with password password and roles ro_admin,bucket_full_access[travel-sample] PUT http://192.168.33.10:8091/settings/rbac/users/local/travel-sample 200 Key not found error: Bucket is ready! GET http://192.168.33.10:8091/pools/nodes 200 All nodes are healthy Keyerror in getting loggingredactlevel revs_limit not found in resources/cluster_configs/base_cc, Ignoring Running playbook: libraries/provision/ansible/playbooks/install-sync-gateway-package.yml [WARNING]: While constructing a mapping from /Users/eunicehuang/dev/mobile- testkit/libraries/provision/ansible/playbooks/install-sync-gateway-package.yml, line 63, column 3, found a duplicate dict key (vars). Using last defined value only. PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** fatal: [sg1]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service sync_gateway: cannot check nor set state"} ...ignoring TASK [SYNC GATEWAY | Stop sync_gateway] **************************************** skipping: [sg1] TASK [SYNC GATEWAY | Uninstall sync_gateway rpm] ******************************* fatal: [sg1]: FAILED! => {"changed": true, "cmd": "rpm -e couchbase-sync-gateway", "delta": "0:00:00.139130", "end": "2019-04-16 23:31:27.856398", "failed": true, "rc": 1, "start": "2019-04-16 23:31:27.717268", "stderr": "error: package couchbase-sync-gateway is not installed", "stdout": "", "stdout_lines": [], "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]} ...ignoring TASK [SYNC GATEWAY | Uninstall sync_gateway deb] ******************************* skipping: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway service] ****************************** ok: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway binary] ******************************* ok: [sg1] TASK [SYNC GATEWAY | Remove sync gateway source] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Remove sync gateway source .repo directory] *************** changed: [sg1] TASK [SYNC GATEWAY | Remove tmp logging dir (log rotation tests)] ************** changed: [sg1] TASK [SYNC GATEWAY | Remove sync_gateway user] ********************************* ok: [sg1] TASK [SG ACCEL | Remove sg_accel user] ***************************************** ok: [sg1] TASK [SYNC GATEWAY | verify no service on 4985] ******************************** ok: [sg1] PLAY [couchbase_servers] ******************************************************* TASK [setup] ******************************************************************* ok: [cb1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Create sync gateway user] ********************************* changed: [sg1] TASK [SYNC GATEWAY | Create /tmp/sg_logs] ************************************** changed: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Download sync_gateway rpm http://latestbuilds.service.couchbase.com/builds/latestbuilds/sync_gateway/2.5.0/271/couchbase-sync-gateway-enterprise_2.5.0-271_x86_64.rpm] *** ok: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Install sync_gateway rpm] ********************************* changed: [sg1] TASK [SYNC GATEWAY | Install sync_gateway deb] ********************************* skipping: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway for CentOS 6] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Verify sync_gateway not listening on port] **************** ok: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs] *************************************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* changed: [sg1] TASK [Deleting /home/sync_gateway/certs directory] ***************************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs.zip] *********************************** skipping: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config] ************************************ changed: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ ok: [sg1] => { "out.stdout": { "SSLCert": "sg_cert.pem", "SSLKey": "sg_privkey.pem", "adminInterface": "0.0.0.0:4985", "compressResponses": false, "databases": { "db": { "bucket": "travel-sample", "enable_shared_bucket_access": true, "import_docs": "continuous", "num_index_replicas": 0, "password": "password", "server": "http://192.168.33.10:8091", "username": "travel-sample" } }, "interface": ":4984", "logging": { "debug": { "enabled": true } }, "maxCouchbaseConnections": 16, "maxFileDescriptors": 90000, "maxIncomingConnections": 0, "slowServerCallWarningThreshold": 500 } } TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** changed: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** changed: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [debug] ******************************************************************* ok: [sg1] => { "msg": "Starting sync_gateway" } TASK [SYNC GATEWAY | Start sync gateway service] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** ok: [sg1] PLAY RECAP ********************************************************************* cb1 : ok=1 changed=0 unreachable=0 failed=0 sg1 : ok=30 changed=13 unreachable=0 failed=0 Running playbook: libraries/provision/ansible/playbooks/install-sg-accel-package.yml PLAY RECAP ********************************************************************* Running playbook: libraries/provision/ansible/playbooks/configure-sync-gateway-awslogs-forwarder.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | upload default awslogs config] **************************** ok: [sg1] TASK [SYNC GATEWAY | restart awslogs service] ********************************** changed: [sg1] [WARNING]: Consider using service module rather than running service PLAY RECAP ********************************************************************* sg1 : ok=3 changed=1 unreachable=0 failed=0 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} Upstream definition: server https://192.168.33.11:4984; Upstream definition admin: server https://192.168.33.11:4985; Running playbook: libraries/provision/ansible/playbooks/install-nginx.yml PLAY RECAP ********************************************************************* >>> Done provisioning cluster... Verfying versions for cluster: resources/cluster_configs/base_cc GET http://Administrator:password@192.168.33.10:8091/pools 200 Expected Server Version: 6.0.1-2037 Running Server Version: 6.0.1-2037 GET https://192.168.33.11:4984/ 200 'server' header: Couchbase Sync Gateway/2.5.0 EE vendor name: Couchbase Sync Gateway GET https://192.168.33.11:4984/ 200 Expected sync_gateway Version: 2.5.0-271 Running sync_gateway Version: 2.5.0-271 Running sync_gateway Vendor Version: 2.5 Verfying versions for cluster: resources/cluster_configs/base_cc GET http://Administrator:password@192.168.33.10:8091/pools 200 Expected Server Version: 6.0.1-2037 Running Server Version: 6.0.1-2037 GET https://192.168.33.11:4984/ 200 'server' header: Couchbase Sync Gateway/2.5.0 EE vendor name: Couchbase Sync Gateway GET https://192.168.33.11:4984/ 200 Expected sync_gateway Version: 2.5.0-271 Running sync_gateway Version: 2.5.0-271 Running sync_gateway Vendor Version: 2.5 {u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}], u'sync_gateways': [{u'ip': u'192.168.33.11', u'name': u'sg1'}], u'environment': {u'delta_sync_enabled': False, u'x509_certs': False, u'sync_gateway_ssl': True, u'sg_lb_enabled': False, u'sg_use_views': False, u'no_conflicts_enabled': False, u'xattrs_enabled': True, u'number_replicas': 0, u'sync_gateway_version': u'2.5.0-271', u'ipv6_enabled': False, u'cbs_ssl_enabled': False, u'server_version': u'6.0.1-2037'}, u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}], u'load_balancers': [], u'sg_accels': []} Running test 'test_default_conflict_withConflicts_and_sgOffline' cluster_config: resources/cluster_configs/base_cc cluster_topology: {'load_balancers': [], 'sg_accels': [], 'sync_gateways': [{'admin': 'https://192.168.33.11:4985', 'public': 'https://192.168.33.11:4984'}], 'couchbase_servers': ['http://192.168.33.10:8091']} mode: cc xattrs_enabled: True Creating a Database cbl-test1555457520.63 at test setup Got response: "@1_10.100.172.185_net" Got response: "@2_10.100.172.185_net" Getting the database name Got response: "cbl-test1555457520.63" resources/cluster_configs/base_cc cbs: ['http://192.168.33.10:8091'] sgs: [{'ip': u'192.168.33.11', 'name': u'sg1'}] acs: [] ssl: False >>> Reseting cluster ... >>> CBS SSL enabled: False >>> Using xattrs: True >>> Stopping sync_gateway Running playbook: libraries/provision/ansible/playbooks/stop-sync-gateway.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway for CentOS 6] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Verify sync_gateway not listening on port] **************** ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service on Windows] ********************* skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=3 changed=1 unreachable=0 failed=0 >>> Stopping sg_accel Running playbook: libraries/provision/ansible/playbooks/stop-sg-accel.yml PLAY RECAP ********************************************************************* >>> Deleting sync_gateway artifacts Running playbook: libraries/provision/ansible/playbooks/delete-sync-gateway-artifacts.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs] ********************************* changed: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Create sync_gateway logs directory on Windows] ************ skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs directory under Temp on Windows] * skipping: [sg1] TASK [SYNC GATEWAY | Delete sync_gateway logs on Windows] ********************** skipping: [sg1] TASK [SYNC GATEWAY | Create sync_gateway logs directory under c:/tmp on Windows] *** skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=2 changed=1 unreachable=0 failed=0 >>> Deleting sg_accel artifacts Running playbook: libraries/provision/ansible/playbooks/delete-sg-accel-artifacts.yml PLAY RECAP ********************************************************************* >>> Deleting buckets on: http://192.168.33.10:8091 GET http://192.168.33.10:8091/pools/default/buckets 200 Found buckets: [u'travel-sample'] Existing buckets: [u'travel-sample'] Deleting buckets: [u'travel-sample'] GET http://Administrator:password@192.168.33.10:8091/pools 200 Deleting RBAC user travel-sample DELETE http://192.168.33.10:8091/settings/rbac/users/local/travel-sample 200 DELETE http://192.168.33.10:8091/pools/default/buckets/travel-sample 200 GET http://192.168.33.10:8091/pools/default/buckets 200 Found buckets: [] >>> Creating buckets on: http://192.168.33.10:8091 >>> Creating buckets [u'travel-sample'] Creating buckets: [u'travel-sample'] total_avail_ram_mb: 3789 effective_avail_ram_mb: 2519 Creating bucket travel-sample with RAM 2519 GET http://Administrator:password@192.168.33.10:8091/pools 200 POST http://192.168.33.10:8091/pools/default/buckets 202 Creating RBAC user travel-sample with password password and roles ro_admin,bucket_full_access[travel-sample] PUT http://192.168.33.10:8091/settings/rbac/users/local/travel-sample 200 Key not found error: Bucket is ready! GET http://192.168.33.10:8091/pools/nodes 200 Node is still not healthy. Status: warmup Retrying ... GET http://192.168.33.10:8091/pools/nodes 200 Node is still not healthy. Status: warmup Retrying ... GET http://192.168.33.10:8091/pools/nodes 200 All nodes are healthy >>> Waiting for Server: http://192.168.33.10:8091 to be in a healthy state GET http://192.168.33.10:8091/pools/nodes 200 All nodes are healthy >>> Starting sync_gateway with configuration: /Users/eunicehuang/dev/mobile-testkit/resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json Keyerror in getting loggingredactlevel revs_limit not found in resources/cluster_configs/base_cc, Ignoring Running playbook: libraries/provision/ansible/playbooks/start-sync-gateway.yml [WARNING]: While constructing a mapping from /Users/eunicehuang/dev/mobile- testkit/libraries/provision/ansible/playbooks/start-sync-gateway.yml, line 23, column 5, found a duplicate dict key (sudo). Using last defined value only. [DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* changed: [sg1] TASK [Deleting /home/sync_gateway/certs directory] ***************************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs.zip] *********************************** skipping: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config] ************************************ changed: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ ok: [sg1] => { "out.stdout": { "SSLCert": "sg_cert.pem", "SSLKey": "sg_privkey.pem", "adminInterface": "0.0.0.0:4985", "compressResponses": false, "databases": { "db": { "bucket": "travel-sample", "enable_shared_bucket_access": true, "import_docs": "continuous", "num_index_replicas": 0, "password": "password", "server": "http://192.168.33.10:8091", "username": "travel-sample" } }, "interface": ":4984", "logging": { "debug": { "enabled": true } }, "maxCouchbaseConnections": 16, "maxFileDescriptors": 90000, "maxIncomingConnections": 0, "slowServerCallWarningThreshold": 500 } } TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** changed: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~2\Couchbase is present on Windows] ***** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~2\Couchbase] ******** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~2\Couchbase\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~1\Couchbase\Sync Gateway is present on Windows] *** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~1\Couchbase\Sync Gateway] *** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~1\Couchbase\Sync Gateway\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ skipping: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [debug] ******************************************************************* ok: [sg1] => { "msg": "Starting sync_gateway" } TASK [SYNC GATEWAY | Start sync gateway service] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** ok: [sg1] TASK [debug] ******************************************************************* skipping: [sg1] TASK [SYNC GATEWAY | Start sync_gateway service on Windows] ******************** skipping: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=10 changed=5 unreachable=0 failed=0 >>> Running in channel cache POST https://192.168.33.11:4985/db/_user/ 201 POST https://192.168.33.11:4985/db/_session 200 cookie name SyncGatewaySession, session id 11ca490d855a5d639720e26485680abaaad95fb4 GET https://192.168.33.11:4984/ 200 POST https://192.168.33.11:4984/db/_bulk_docs 201 GET https://192.168.33.11:4984/db/sg_docs_0?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_0?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_0?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_0?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_1?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_1?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_1?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_1?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_2?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_2?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_2?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_2?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_3?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_3?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_3?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_3?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_4?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_4?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_4?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_4?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_5?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_5?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_5?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_5?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_6?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_6?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_6?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_6?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_7?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_7?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_7?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_7?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_8?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_8?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_8?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_8?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_9?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_9?new_edits=false 201 GET https://192.168.33.11:4984/db/sg_docs_9?conflicts=true&revs=true&show_exp=true 200 PUT https://192.168.33.11:4984/db/sg_docs_9?new_edits=false 201 GET https://192.168.33.11:4984/db/_all_docs 200 GET https://192.168.33.11:4984/db/sg_docs_0?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_0: 1 times GET https://192.168.33.11:4984/db/sg_docs_1?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_1: 1 times GET https://192.168.33.11:4984/db/sg_docs_2?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_2: 1 times GET https://192.168.33.11:4984/db/sg_docs_3?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_3: 1 times GET https://192.168.33.11:4984/db/sg_docs_4?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_4: 1 times GET https://192.168.33.11:4984/db/sg_docs_5?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_5: 1 times GET https://192.168.33.11:4984/db/sg_docs_6?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_6: 1 times GET https://192.168.33.11:4984/db/sg_docs_7?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_7: 1 times GET https://192.168.33.11:4984/db/sg_docs_8?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_8: 1 times GET https://192.168.33.11:4984/db/sg_docs_9?conflicts=true&revs=true&show_exp=true 200 Updating https://192.168.33.11:4984/db/sg_docs_9: 1 times Got response: "@3_10.100.172.185_net" Got response: "@4_10.100.172.185_net" Got response: "@5_10.100.172.185_net" Got response: I-1 Got response: "connecting" Activity level: connecting Got response: I-1 Got response: "idle" Got response: L10 Got response: L10 Activity level: idle Got response: L10 Got response: L10 Running playbook: libraries/provision/ansible/playbooks/stop-sync-gateway.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service] ******************************** changed: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway for CentOS 6] *************************** skipping: [sg1] TASK [SYNC GATEWAY | Verify sync_gateway not listening on port] **************** ok: [sg1] TASK [SYNC GATEWAY | Stop sync_gateway service on Windows] ********************* skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=3 changed=1 unreachable=0 failed=0 updating bulk docs Got response: I-1 Got response: "@6_10.100.172.185_net" Got response: I-1 Got response: "@7_10.100.172.185_net" Got response: I-1 Got response: "@8_10.100.172.185_net" Got response: I-1 Got response: "@9_10.100.172.185_net" Got response: I-1 Got response: "@10_10.100.172.185_net" Got response: I-1 Got response: "@11_10.100.172.185_net" Got response: I-1 Got response: "@12_10.100.172.185_net" Got response: I-1 Got response: "@13_10.100.172.185_net" Got response: I-1 Got response: "@14_10.100.172.185_net" Got response: I-1 Got response: "@15_10.100.172.185_net" Got response: I-1 2019-04-16 16:32:42,387 - INFO - start - >>> Starting sync_gateway with configuration: /Users/eunicehuang/dev/mobile-testkit/resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json INFO:test_framework:>>> Starting sync_gateway with configuration: /Users/eunicehuang/dev/mobile-testkit/resources/sync_gateway_configs/sync_gateway_travel_sample_cc.json Keyerror in getting loggingredactlevel revs_limit no found in resources/cluster_configs/base_cc, Ignoring Running playbook: libraries/provision/ansible/playbooks/start-sync-gateway.yml PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to host] ************************* ok: [sg1] TASK [Deleting /home/sync_gateway/certs directory] ***************************** skipping: [sg1] TASK [Deleting /home/sync_gateway/certs.zip] *********************************** skipping: [sg1] TASK [Copying certs.zip to /home/sync_gateway directory] *********************** skipping: [sg1] TASK [Unzipping certs.zip] ***************************************************** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config] ************************************ changed: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ ok: [sg1] => { "out.stdout": { "SSLCert": "sg_cert.pem", "SSLKey": "sg_privkey.pem", "adminInterface": "0.0.0.0:4985", "compressResponses": false, "databases": { "db": { "bucket": "travel-sample", "enable_shared_bucket_access": true, "import_docs": "continuous", "num_index_replicas": 0, "password": "password", "server": "http://192.168.33.10:8091", "username": "travel-sample" } }, "interface": ":4984", "logging": { "debug": { "enabled": true } }, "maxCouchbaseConnections": 16, "maxFileDescriptors": 90000, "maxIncomingConnections": 0, "slowServerCallWarningThreshold": 500 } } TASK [SYNC GATEWAY | Deploying SSL Cert] *************************************** ok: [sg1] TASK [SYNC GATEWAY | Deploying SSL Private Key] ******************************** ok: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~2\Couchbase is present on Windows] ***** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~2\Couchbase] ******** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~2\Couchbase\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Check if C:\PROGRA~1\Couchbase\Sync Gateway is present on Windows] *** skipping: [sg1] TASK [SYNC GATEWAY | Copy sync gateway config to C:\PROGRA~1\Couchbase\Sync Gateway] *** skipping: [sg1] TASK [SYNC GATEWAY | Check deployed config C:\PROGRA~1\Couchbase\Sync Gateway\serviceconfig.json] *** skipping: [sg1] TASK [SYNC GATEWAY | Print deployed config] ************************************ skipping: [sg1] PLAY [sync_gateways] *********************************************************** TASK [setup] ******************************************************************* ok: [sg1] TASK [debug] ******************************************************************* ok: [sg1] => { "msg": "Starting sync_gateway" } TASK [SYNC GATEWAY | Start sync gateway service] ******************************* changed: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** ok: [sg1] TASK [debug] ******************************************************************* skipping: [sg1] TASK [SYNC GATEWAY | Start sync_gateway service on Windows] ******************** skipping: [sg1] TASK [SYNC GATEWAY | Wait until sync gateway to listen on port] **************** skipping: [sg1] PLAY RECAP ********************************************************************* sg1 : ok=10 changed=2 unreachable=0 failed=0 Got response: "stopped" replicator.getActivitylevel(repl)=stopped Got response: "stopped" Got response: "stopped" Activity level: stopped FAILEDTearing down test Deleting the database cbl-test at test teardown Got response: true Got response: I-1 Flushing server memory Got response: I-1 Stopping the test server per test Clearing png files in resources/data ------------- generated xml file: /Users/eunicehuang/dev/mobile-testkit/results/results.xml -------------- ------------- generated html file: /Users/eunicehuang/dev/mobile-testkit/results/report.html ------------- ================================================ FAILURES ================================================ ___________________________ test_default_conflict_withConflicts_and_sgOffline ____________________________ params_from_base_test_setup = {'base_url': 'http://localhost:8080', 'cbl_db': 'cbl-test1555457520.63', 'cluster_config': 'resources/cluster_configs/..._accels': [], 'sync_gateways': [{'admin': 'https://192.168.33.11:4985', 'public': 'https://192.168.33.11:4984'}]}, ...} @pytest.mark.listener @pytest.mark.replication def test_default_conflict_withConflicts_and_sgOffline(params_from_base_test_setup): """ @summary: 1. create docs in sg. 2. Create two conflicts with 2-hex in sg. 3. update doc in sg to have new revision to one of the conflicted branch of sg, the counter for property updates increments to 1 4. Start replication with push pull and continous true 5. wait until replication is done. 6. Verify that default conflict resolver resolved appropriately. 7. Stop sg. 8. Now delete doc in cbl 9. Wait until replication is done i.e 9. Verify docs deleted in sg. """ sg_db = "db" sg_url = params_from_base_test_setup["sg_url"] sg_admin_url = params_from_base_test_setup["sg_admin_url"] sg_config = params_from_base_test_setup["sg_config"] cluster_config = params_from_base_test_setup["cluster_config"] sg_blip_url = params_from_base_test_setup["target_url"] base_url = params_from_base_test_setup["base_url"] db = params_from_base_test_setup["db"] cbl_db = params_from_base_test_setup["source_db"] no_conflicts_enabled = params_from_base_test_setup["no_conflicts_enabled"] channels = ["replication-channel"] num_of_docs = 10 username = "autotest" password = "password" if no_conflicts_enabled: pytest.skip('Cannot work with no-conflicts enabled') # Reset cluster to clean the data c = cluster.Cluster(config=cluster_config) c.reset(sg_config_path=sg_config) # 1. Create docs in SG. sg_client = MobileRestClient() sg_client.create_user(sg_admin_url, sg_db, username, password, channels=channels) cookie, session_id = sg_client.create_session(sg_admin_url, sg_db, username) session = cookie, session_id sg_docs = document.create_docs(doc_id_prefix='sg_docs', number=num_of_docs, channels=channels) sg_docs = sg_client.add_bulk_docs(url=sg_url, db=sg_db, docs=sg_docs, auth=session) # 2. Create two conflicts with 2-hex in sg. for i in xrange(len(sg_docs)): sg_client.add_conflict(url=sg_url, db=sg_db, doc_id=sg_docs[i]["id"], parent_revisions=sg_docs[i]["rev"], new_revision="2-41fa", auth=session) sg_client.add_conflict(url=sg_url, db=sg_db, doc_id=sg_docs[i]["id"], parent_revisions=sg_docs[i]["rev"], new_revision="2-41fa9b", auth=session) # 3. update doc in sg to have new revision to one of the conflicted branch of sg. sg_docs = sg_client.get_all_docs(url=sg_url, db=sg_db, auth=session) sg_docs = sg_docs["rows"] sg_client.update_docs(url=sg_url, db=sg_db, docs=sg_docs, number_updates=1, delay=None, auth=session, channels=channels) # 4. Start replication with push pull and contiinous true replicator = Replication(base_url) authenticator = Authenticator(base_url) replicator_authenticator = authenticator.authentication(session_id, cookie, authentication_type="session") repl = replicator.configure_and_replicate(source_db=cbl_db, replicator_authenticator=replicator_authenticator, target_url=sg_blip_url, replication_type="push_pull", continuous=True, channels=channels) # 5. Stop sg status = c.sync_gateways[0].stop() assert status == 0, "sync_gateway did not stop" # 6. Now update and delete doc in cbl db.update_bulk_docs(cbl_db) db.cbl_delete_bulk_docs(cbl_db) # 7 . Start sg and wait until replication is idle status = c.sync_gateways[0].start(sg_config) assert status == 0, "sync_gateway did not start" count = 0 print "replicator.getActivitylevel(repl)={}".format(replicator.getActivitylevel(repl)) while replicator.getActivitylevel(repl) == "offline" and count < 10: time.sleep(1) count += 1 > replicator.wait_until_replicator_idle(repl) testsuites/CBLTester/CBL_Functional_tests/TestSetup_FunctionalTests/test_replication.py:2013: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , repl = '@5_10.100.172.185_net' err_check = True, max_times = 150, sleep_time = 2 def wait_until_replicator_idle(self, repl, err_check=True, max_times=150, sleep_time=2): count = 0 # Sleep until replicator completely processed activity_level = self.getActivitylevel(repl) while count < max_times: log_info("Activity level: {}".format(activity_level)) time.sleep(sleep_time) if activity_level == "offline" or activity_level == "connecting" or activity_level == "busy": count += 1 else: if activity_level == "idle": if (self.getCompleted(repl) < self.getTotal(repl)) and self.getTotal(repl) != 0: count += 1 else: time.sleep(sleep_time) break if err_check: err = self.getError(repl) if err is not None and err != 'nil' and err != -1: > raise Exception("Error while replicating", err) E Exception: ('Error while replicating', 'Couchbase.Lite.CouchbaseLiteException: CouchbaseLiteException (LiteCoreDomain / 26): Connection closed..') CBLClient/Replication.py:311: Exception ========================================== 181 tests deselected ========================================== =============================== 1 failed, 181 deselected in 171.63 seconds ===============================