Started by user Arunkumar Senthilnathan
[EnvInject] - Loading node environment variables.
Building remotely on slv-s62221 (P0 jython_slave PY3P0 P0_test) in workspace /data/workspace/centos-p0-temp-vset00-00-temp_job
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Cloning the remote Git repository
Cloning repository git://github.com/couchbase/testrunner
 > /usr/bin/git init /data/workspace/centos-p0-temp-vset00-00-temp_job # timeout=10
Fetching upstream changes from git://github.com/couchbase/testrunner
 > /usr/bin/git --version # timeout=10
 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/*
 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10
 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10
Fetching upstream changes from git://github.com/couchbase/testrunner
 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/*
 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1cd25d65e3b53bf6cf01ee31c372c7feeb56bea1 (origin/master)
 > /usr/bin/git config core.sparsecheckout # timeout=10
 > /usr/bin/git checkout -f 1cd25d65e3b53bf6cf01ee31c372c7feeb56bea1
 > /usr/bin/git rev-list e9fd4ed58141aaa1f6276aa1525aaaf20807d2ea # timeout=10
 > /usr/bin/git tag -a -f -m Jenkins Build #71 jenkins-temp_arun_job-71 # timeout=10
No emails were triggered.
[centos-p0-temp-vset00-00-temp_job] $ /bin/sh -xe /tmp/jenkins1342441759714796632.sh
+ echo version=7.0.0-3739
version=7.0.0-3739
+ echo '[global]
username:root
password:couchbase
port:8091
n1ql_port:8093

[membase]
rest_username:Administrator
rest_password:password

[servers]
1:_1
2:_2
3:_3
4:_4

[_1]
ip:172.23.104.105

[_2]
ip:172.23.104.107

[_3]
ip:172.23.104.136

[_4]
ip:172.23.104.201'
+ echo '
### XDCR ####
xdcr.advFilteringXDCR.XDCRAdvFilterTests.test_xdcr_with_filter,items=1000,pause=C1:C2,rdirection=bidirection,ctopology=chain,default@C1=filter_expression:REGEXP_CONTAINS(META()dotidcomma '\''C1-key-1'\''),default@C2=filter_expression:REGEXP_CONTAINS(META()dotidcomma '\''C2-key-1'\''),demand_encryption=1
xdcr.uniXDCR.unidirectional.load_with_ops,items=1000,ctopology=chain,rdirection=unidirection,update=C1,delete=C1
xdcr.advFilteringXDCR.XDCRAdvFilterTests.test_xdcr_with_filter,items=1000,rdirection=unidirection,ctopology=chain,default@C1=filter_expression:REGEXP_CONTAINS(META()dotidcomma '\''C1-key-1'\'')
xdcr.biXDCR.bidirectional.load_with_async_ops,replicas=1,items=1000,ctopology=chain,rdirection=bidirection,update=C1-C2,delete=C1-C2
# Advance Filtering
xdcr.advFilteringXDCR.XDCRAdvFilterTests.test_xdcr_with_filter,items=1000,rdirection=unidirection,ctopology=chain,replication_type=xmem,default@C1=filter_expression:REGEXP_CONTAINS(META()dotidcomma'\''0$'\''),failure_restart_interval:20,timeout_percentage:60
xdcr.filterXDCR.XDCRFilterTests.test_xdcr_with_filter,items=1000,rdirection=unidirection,ctopology=chain,replication_type=xmem,default@C1=filter_expression:REGEXP_CONTAINS(META()dotidcomma '\''0$'\''),skip_key_validation=True
# Prioritization of replications
xdcr.prioritizationXDCR.XDCRPrioritization.test_priority,initial=True,items=1000,rdirection=unidirection,ctopology=chain,replication_type=xmem,standard_buckets=1,default@C1=priority:Medium,standard_bucket_1@C1=priority:High

rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=1000,doc_ops=create;update;delete
rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=3,replicas=1,items=1000,services_init=kv-kv-kv-kv,services_out=kv
swaprebalance.SwapRebalanceBasicTests.do_test,replica=1,num-buckets=1,num-swap=1,items=1000
failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,withMutationOps=True,services_init=kv-kv-kv-kv,services_out=kv
rackzone.rackzonetests.RackzoneTests.test_check_default_zone_create_by_default,items=1000
rackzone.rackzonetests.RackzoneTests.test_create_zone_with_upper_lower_number_and_space_name,items=1000
view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=update,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=1000,sasl_buckets=1,standard_buckets=1,nodes_init=1

### WATSON NEW FEATURE ###
fts.stable_topology_fts.StableTopFTS.run_default_index_query,items=1000,query={"match":' '"safiya@mcdiabetes.com",' '"field":"email"},expected_hits=1000,cluster=D+F,F

#### backup restore ####
ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_sanity,items=1000
security.x509tests.x509tests.test_basic_ssl_test,default_bucket=False,SSLtype=openssl
### backup restore with collection
ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest.test_backup_restore_collection_sanity,items=1000,map_data_collection=true,number_of_backups=2,load_to_collection=true

#moving tests from py-multi-node-sanity-new-feat.conf to this conf file.
clitest.importexporttest.ImportExportTests.test_export_and_import_back,imex_type=json,format_type=list,import_back=True,nodes_init=2
autofailovertests.AutoFailoverTests.test_30s_timeout_stop,replicas=2,keys-count=1000,num-buckets=2

#ephemeral buckets
buckettests.CreateBucketTests.test_ephemeral_buckets,default_bucket=false,sasl_buckets=2,bucket_size=100

# Collections
tuqquery.n1ql_collections_end2end.QueryCollectionsEnd2EndTests.test_end_to_end,GROUP=P0,test_name=e2e_1bucket_1scope_1collection,sanity_test=True,nodes_init=2,services_init=kv:n1ql:index-kv:n1ql:index,reset_services=True,bucket_size=100
xdcr.uniXDCR.unidirectional.load_with_ops,items=1000,ctopology=chain,rdirection=unidirection,update=C1,delete=C1,java_sdk_client=True

# This test is for eventing sanity
eventing.eventing_sanity.EventingSanity.test_create_mutation_for_dcp_stream_boundary_from_beginning,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True
eventing.eventing_sanity.EventingSanity.test_source_doc_mutations_with_timers,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True,source_bucket_mutation=true
eventing.eventing_sanity.EventingSanity.test_pause_resume_execution,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True
eventing.eventing_curl.EventingCurl.test_curl_get,nodes_init=2,services_init=kv-eventing,dataset=default,reset_services=True,skip_cleanup=True,path=basic-auth,curl_user=postman,curl_password=password,auth_type=basic,curl=True

# n1ql+fts integration - disabled for now as this test fails if it is not the first test in the conf
tuqquery.n1ql_fts_sanity.N1qlFTSSanityTest.test_n1ql_syntax_select_from_let,nodes_init=4,services_init=kv:index:fts:n1ql-kv:index:fts:n1ql-kv:index:fts:n1ql-kv:index:fts:n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True

# Analytics
cbas.cbas_bucket_operations.CBASBucketOperations.load_docs_in_cb_bucket_before_and_after_cbas_connect,cb_bucket_name=default,cbas_bucket_name=default_bucket,cbas_dataset_name=default_ds,items=100000,cluster=D

#Flex sanity job
tuqquery.flex_index_phase1.FlexIndexTests.test_flex_with_napa_dataset,scan_consistency=NOT_BOUNDED,skip_load=True,items=10000,bucket_size=200,cluster=D,D+F,F,index_type=scorch,fts_quota=750,custom_map_add_non_indexed_fields=False,text_analyzer=keyword,use_index_name_in_query=False,flex_query_option=flex_use_fts_gsi_query,flex_dataset=default,nodes_init=3

# These test should always be in the end as it reset the serivce set on the node
gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple,doc-per-day=10,dataset=default,gsi_type=memory_optimized,reset_services=True
gsi.gsi_rebalance_gsi.SecondaryIndexingRebalanceTests.test_gsi_rebalance_out_indexer_node,doc-per-day=1000,dataset=array,nodes_init=4,services_init=kv-index-index-n1ql,groups=simple,reset_services=True
gsi.gsi_index_partitioning.GSIIndexPartitioningTests.test_index_scans,items=10000,nodes_init=4,services_init=kv:n1ql-index-index-index,groups=simple,reset_services=True,doc-per-day=1,dataset=bigdata,create_primary_index=False
gsi.collections_index_basics.CollectionsIndexBasics.test_gsi_for_collection,default_bucket=false,force_clean=true,defer_build=False,gsi_type=plasma,reset_services=True,java_sdk_client=True
'
[centos-p0-temp-vset00-00-temp_job] $ /bin/bash /tmp/jenkins2658778176760477451.sh
Cherrypick is: 
sh -c 
/data/workspace/centos-p0-temp-vset00-00-temp_job
Using Couchbase 3 SDK 
virtualenv 20.0.26 from /usr/local/lib/python3.7/site-packages/virtualenv/__init__.py
created virtual environment CPython3.7.6.final.0-64 in 365ms
  creator CPython3Posix(dest=/data/workspace/centos-p0-temp-vset00-00-temp_job/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Collecting bcrypt~=3.1.7
  Using cached bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl (56 kB)
Collecting beautifulsoup4~=4.9.1
  Using cached beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
Collecting boto~=2.49.0
  Using cached boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
Collecting boto3~=1.14.8
  Downloading boto3-1.14.63-py2.py3-none-any.whl (129 kB)
Collecting botocore~=1.17.8
  Downloading botocore-1.17.63-py2.py3-none-any.whl (6.6 MB)
Processing /root/.cache/pip/wheels/cd/92/e4/538bced44c31b450a21fb8b7c1be0d11f126c2abe87d629b9a/btrc-3.2.1-py3-none-any.whl
Collecting certifi~=2020.6.20
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting cffi~=1.14.0
  Using cached cffi-1.14.3-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting chardet~=3.0.4
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting couchbase~=3.0.0
  Downloading couchbase-3.0.6.tar.gz (1.5 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting cryptography~=2.9.2
  Using cached cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
Collecting decorator~=4.4.2
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting deepdiff~=5.0.0
  Using cached deepdiff-5.0.2-py3-none-any.whl (60 kB)
Collecting docker~=4.2.1
  Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting docutils~=0.16
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Processing /root/.cache/pip/wheels/0e/0d/36/0497b38e63d1cc8d15fa8b4e669650d6f4dada9f254a951673/Geohash-1.0-py3-none-any.whl
Collecting httplib2~=0.18.1
  Using cached httplib2-0.18.1-py3-none-any.whl (95 kB)
Collecting idna~=2.9
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting Jinja2~=2.11.2
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting jmespath~=0.10.0
  Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Processing /root/.cache/pip/wheels/04/00/65/435977d12c4daa4fd57668863c0a3e89d041d5d39227a8b774/logger-1.4-py3-none-any.whl
Collecting MarkupSafe~=1.1.1
  Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
Processing /root/.cache/pip/wheels/73/2b/f6/26e9f84153c25050fe7c09e88f8e32a6be3c7034a38c418319/ordered_set-4.0.2-py2.py3-none-any.whl
Collecting paramiko~=2.7.1
  Using cached paramiko-2.7.2-py2.py3-none-any.whl (206 kB)
Collecting pycparser~=2.20
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Processing /root/.cache/pip/wheels/8c/c3/90/614771ae12c31abc58fab73e4f4d2a631cc7cd5f961f3fc277/pyes-0.99.6-py3-none-any.whl
Collecting pymongo~=3.10.1
  Using cached pymongo-3.10.1-cp37-cp37m-manylinux2014_x86_64.whl (462 kB)
Collecting PyNaCl~=1.4.0
  Using cached PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961 kB)
Collecting python-dateutil~=2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz~=2020.1
  Using cached pytz-2020.4-py2.py3-none-any.whl (509 kB)
Processing /root/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests~=2.24.0
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting s3transfer~=0.3.3
  Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Processing /root/.cache/pip/wheels/73/ad/a4/0dff4a6ef231fc0dfa12ffbac2a36cebfdddfe059f50e019aa/sgmllib3k-1.0.0-py3-none-any.whl
Collecting six~=1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting soupsieve~=2.0.1
  Using cached soupsieve-2.0.1-py3-none-any.whl (32 kB)
Collecting urllib3~=1.25.9
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting websocket-client~=0.57.0
  Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Processing /root/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl
Processing /root/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting mypy-extensions
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting attrs>=19.1.0; python_version > "3.5"
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting typing-extensions; python_version < "3.8"
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Building wheels for collected packages: couchbase
  Building wheel for couchbase (PEP 517): started
  Building wheel for couchbase (PEP 517): finished with status 'done'
  Created wheel for couchbase: filename=couchbase-3.0.6-cp37-cp37m-linux_x86_64.whl size=6977162 sha256=18cf525598ca102cb75518b0e911bd3c456d02125b79842a57a11306cea6185e
  Stored in directory: /root/.cache/pip/wheels/13/e7/da/b740e18ebfebf8a9a8313461072cfb938237c3932334e3015a
Successfully built couchbase
Installing collected packages: six, pycparser, cffi, bcrypt, soupsieve, beautifulsoup4, boto, jmespath, urllib3, docutils, python-dateutil, botocore, s3transfer, boto3, logger, chardet, idna, certifi, requests, btrc, pyrsistent, wrapt, mypy-extensions, attrs, typing-extensions, couchbase, cryptography, decorator, ordered-set, deepdiff, websocket-client, docker, Geohash, httplib2, MarkupSafe, Jinja2, PyNaCl, paramiko, pyes, pymongo, pytz, PyYAML, sgmllib3k
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

botocore 1.17.63 requires docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
btrc 3.2.1 requires requests==2.1.0, but you'll have requests 2.24.0 which is incompatible.
Successfully installed Geohash-1.0 Jinja2-2.11.2 MarkupSafe-1.1.1 PyNaCl-1.4.0 PyYAML-5.3.1 attrs-20.3.0 bcrypt-3.1.7 beautifulsoup4-4.9.3 boto-2.49.0 boto3-1.14.63 botocore-1.17.63 btrc-3.2.1 certifi-2020.6.20 cffi-1.14.3 chardet-3.0.4 couchbase-3.0.6 cryptography-2.9.2 decorator-4.4.2 deepdiff-5.0.2 docker-4.2.2 docutils-0.16 httplib2-0.18.1 idna-2.10 jmespath-0.10.0 logger-1.4 mypy-extensions-0.4.3 ordered-set-4.0.2 paramiko-2.7.2 pycparser-2.20 pyes-0.99.6 pymongo-3.10.1 pyrsistent-0.17.3 python-dateutil-2.8.1 pytz-2020.4 requests-2.24.0 s3transfer-0.3.3 sgmllib3k-1.0.0 six-1.15.0 soupsieve-2.0.1 typing-extensions-3.7.4.3 urllib3-1.25.11 websocket-client-0.57.0 wrapt-1.12.1
WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available.
You should consider upgrading via the '/data/workspace/centos-p0-temp-vset00-00-temp_job/venv/bin/python -m pip install --upgrade pip' command.
setting up for java sdk client
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Cloning into 'java_sdk_client'...
Submodule path 'java_sdk_client': checked out '7a93fe0c93f5b16c3cc90398f1e518f72e33a091'
PYTHON SDK VERSION IS couchbase==3.0.6
INFO:root:__main__
INFO:__main__:TestRunner: parsing args...
INFO:__main__:Checking arguments...
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 1
INFO:__main__:--> Running test: view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=create,test_with_view=True,num_ddocs=3,num_views_per_ddoc=2,items=200000,sasl_buckets=1,GROUP=P1,skip_log_scan=False,bucket_storage=couchstore,skip_cleanup=True
INFO:__main__:Logs folder: /data/workspace/centos-p0-temp-vset00-00-temp_job/logs/testrunner-20-Nov-16_16-26-20/test_1
*** TestRunner ***
{'cluster_name': 'test', 'ini': '/tmp/test.ini', 'num_nodes': 4}
Logs will be stored at /data/workspace/centos-p0-temp-vset00-00-temp_job/logs/testrunner-20-Nov-16_16-26-20/test_1

./testrunner -i /tmp/test.ini -p  -t view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=create,test_with_view=True,num_ddocs=3,num_views_per_ddoc=2,items=200000,sasl_buckets=1,GROUP=P1,skip_log_scan=False,bucket_storage=couchstore,skip_cleanup=True

Test Input params:
{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '3', 'num_views_per_ddoc': '2', 'items': '200000', 'sasl_buckets': '1', 'GROUP': 'P1', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'skip_cleanup': 'True', 'ini': '/tmp/test.ini', 'cluster_name': 'test', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/data/workspace/centos-p0-temp-vset00-00-temp_job/logs/testrunner-20-Nov-16_16-26-20/test_1'}
Run before suite setup for view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops
suite_setUp (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: suite_setUp (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: type object 'CreateDeleteViewTests' has no attribute 'suite_setUp'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
rebalance_in_and_out_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests) ... -->before_suite_name:view.createdeleteview.CreateDeleteViewTests.suite_setUp,suite: <unittest.suite.TestSuite tests=[<unittest.loader._FailedTest testMethod=suite_setUp>]>
-->result: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
2020-11-16 16:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:26:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:21 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:21 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-3739-enterprise
2020-11-16 16:26:21 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-3739-enterprise', '7.0.0-3739-enterprise', '7.0.0-3739-enterprise']
2020-11-16 16:26:21 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 rebalance_in_and_out_with_ddoc_ops==============
2020-11-16 16:26:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['bucket0', 'default'] on 172.23.104.105
2020-11-16 16:26:22 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2020-11-16 16:26:22 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.105 existing buckets : ['default']
2020-11-16 16:26:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : bucket0 from 172.23.104.105
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.105 existing buckets : []
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.104.105
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.107, nothing to delete
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.136, nothing to delete
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.201, nothing to delete
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.105,ns_1@172.23.104.107,ns_1@172.23.104.201', 'ejectedNodes': 'ns_1@172.23.104.107,ns_1@172.23.104.201', 'user': 'Administrator', 'password': 'password'}
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2020-11-16 16:26:23 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2020-11-16 16:26:43 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.03 seconds 
2020-11-16 16:26:43 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance...
2020-11-16 16:26:53 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.105 port:8091 ssh_username:root ? [('ns_1@172.23.104.107', 8091), ('ns_1@172.23.104.201', 8091)]
2020-11-16 16:26:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.105:8091
2020-11-16 16:26:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.105:8091 is running
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.107:8091
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.107:8091 is running
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.136:8091
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.136:8091 is running
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.201:8091
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.201:8091 is running
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.105:8091 => {'services': ['kv'], 'cpu_utilization': 0.7530120481927711, 'mem_free': 3362070528, 'mem_total': 4220854272, 'swap_mem_used': 0, 'swap_mem_total': 3758092288}
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2020-11-16 16:26:54 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.107 with username:root, attempt#1 of 5
2020-11-16 16:26:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.107 as root
2020-11-16 16:26:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.136 with username:root, attempt#1 of 5
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.136 as root
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.201 with username:root, attempt#1 of 5
2020-11-16 16:26:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.201 as root
2020-11-16 16:26:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2020-11-16 16:26:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2020-11-16 16:26:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.105 port:8091 ssh_username:root, nodes/self 
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '4807', 'memoryTotal': 4220854272, 'memoryFree': 3401785344, 'mcdMemoryReserved': 3220, 'mcdMemoryAllocated': 3220, 'status': 'healthy', 'hostname': '172.23.104.105:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3739-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f90394bf890>], 'memoryQuota': 2157, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.105', 'ip': '172.23.104.105', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4025, 'curr_items': 0}
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2157
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.105:8091:port=8091&username=Administrator&password=password
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with root but got an error ['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100    43  100     2  100    41    244   5012 --:--:-- --:--:-- --:--:--  5125'] ...
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok']
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.107 port:8091 ssh_username:root, nodes/self 
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '4731', 'memoryTotal': 4220854272, 'memoryFree': 3642433536, 'mcdMemoryReserved': 3220, 'mcdMemoryAllocated': 3220, 'status': 'healthy', 'hostname': '172.23.104.107:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3739-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f90394bf790>], 'memoryQuota': 1749, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.107', 'ip': '172.23.104.107', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4025, 'curr_items': 0}
2020-11-16 16:27:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8091/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2157
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.107:8091:port=8091&username=Administrator&password=password
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.107 with username:root, attempt#1 of 5
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.107 as root
2020-11-16 16:27:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with root but got an error ['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100    43  100     2  100    41    215   4415 --:--:-- --:--:-- --:--:--  4555'] ...
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok']
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.136 port:8091 ssh_username:root, nodes/self 
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '4808', 'memoryTotal': 3950026752, 'memoryFree': 3267940352, 'mcdMemoryReserved': 3013, 'mcdMemoryAllocated': 3013, 'status': 'healthy', 'hostname': '172.23.104.136:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3739-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f90377624d0>], 'memoryQuota': 1594, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.136', 'ip': '172.23.104.136', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3767, 'curr_items': 0}
2020-11-16 16:27:01 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8091/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2018
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.136:8091:port=8091&username=Administrator&password=password
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.136 with username:root, attempt#1 of 5
2020-11-16 16:27:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.136 as root
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.136: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with root but got an error ['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100    43  100     2  100    41    271   5573 --:--:-- --:--:-- --:--:--  5857'] ...
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok']
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.136:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.136:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.201 port:8091 ssh_username:root, nodes/self 
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '4746', 'memoryTotal': 4201844736, 'memoryFree': 3606941696, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.104.201:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3739-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f9037582950>], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.201', 'ip': '172.23.104.201', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0}
2020-11-16 16:27:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8091/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.201:8091:port=8091&username=Administrator&password=password
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.201 with username:root, attempt#1 of 5
2020-11-16 16:27:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.201 as root
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.201: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with root but got an error ['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100    43  100     2  100    41    205   4209 --:--:-- --:--:-- --:--:--  4555'] ...
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok']
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.201:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.201:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version().
2020-11-16 16:27:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2020-11-16 16:27:03 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.105 ****
2020-11-16 16:27:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with root but got an error ['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100    43  100     2  100    41    245   5023 --:--:-- --:--:-- --:--:--  5125'] ...
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok']
2020-11-16 16:27:09 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.105
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.105:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1078&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4
2020-11-16 16:27:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.105 to accept set ops
2020-11-16 16:27:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:27:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.105:8091/pools/default/buckets with param: name=bucket0&ramQuotaMB=1078&authType=sasl&saslPassword=password&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.00 seconds to create bucket bucket0
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:12 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.104.105 to accept set ops
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:27:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5
2020-11-16 16:27:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 rebalance_in_and_out_with_ddoc_ops ==============
2020-11-16 16:27:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:27:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:27:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.107 with username:root, attempt#1 of 5
2020-11-16 16:27:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.107 as root
2020-11-16 16:27:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.136 with username:root, attempt#1 of 5
2020-11-16 16:27:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.136 as root
2020-11-16 16:27:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.201 with username:root, attempt#1 of 5
2020-11-16 16:27:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.201 as root
2020-11-16 16:27:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:27:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:27:19 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2020-11-16 16:27:19 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.105:8091 => {'services': ['kv'], 'cpu_utilization': 11.57921291624622, 'mem_free': 3234455552, 'mem_total': 4220854272, 'swap_mem_used': 0, 'swap_mem_total': 3758092288}
2020-11-16 16:27:19 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2020-11-16 16:27:19 | INFO | MainProcess | test_thread | [createdeleteview.rebalance_in_and_out_with_ddoc_ops] create a cluster of all the available servers except nodes_in
2020-11-16 16:27:20 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.107:8091 to cluster
2020-11-16 16:27:20 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.107:8091 to this cluster @172.23.104.105:8091
2020-11-16 16:27:25 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.136:8091 to cluster
2020-11-16 16:27:25 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.136:8091 to this cluster @172.23.104.105:8091
2020-11-16 16:27:28 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.105,ns_1@172.23.104.107,ns_1@172.23.104.136', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'}
2020-11-16 16:27:28 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2020-11-16 16:27:28 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2020-11-16 16:27:28 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00%
2020-11-16 16:27:38 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 21.00 %
2020-11-16 16:27:38 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 21.00%
2020-11-16 16:27:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 35.00 %
2020-11-16 16:27:48 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 35.00%
2020-11-16 16:27:58 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 49.00 %
2020-11-16 16:27:58 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 49.00%
2020-11-16 16:28:08 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 71.00 %
2020-11-16 16:28:08 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 71.00%
2020-11-16 16:28:18 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 84.00 %
2020-11-16 16:28:18 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 84.00%
2020-11-16 16:28:28 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 98.00 %
2020-11-16 16:28:28 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 98.00%
2020-11-16 16:28:38 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00%
2020-11-16 16:28:38 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 70.23809957504272 sec
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [basetestcase._load_doc_data_all_buckets] create 200000 documents...
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.136:11210 default
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:28:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:28:39 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.136:11210 bucket0
2020-11-16 16:28:39 | INFO | Process-3   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:28:39 | INFO | Process-4   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:28:39 | INFO | Process-5   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:28:39 | INFO | Process-6   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:28:39 | INFO | Process-3   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:28:39 | INFO | Process-4   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:28:39 | INFO | Process-5   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:28:39 | INFO | Process-6   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:28:39 | INFO | Process-5   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 default
2020-11-16 16:28:39 | INFO | Process-4   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 bucket0
2020-11-16 16:28:39 | INFO | Process-3   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 default
2020-11-16 16:28:39 | INFO | Process-6   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 default
2020-11-16 16:28:46 | INFO | Process-7   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:28:46 | INFO | Process-7   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:28:46 | INFO | Process-8   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-7   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 default
2020-11-16 16:28:46 | INFO | Process-8   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-9   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-8   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-9   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-9   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-10  | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-10  | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:28:46 | INFO | Process-10  | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.136:11210 bucket0
2020-11-16 16:28:54 | INFO | MainProcess | test_thread | [createdeleteview._execute_ddoc_ops] Processing Create DDoc Operation On Bucket default
2020-11-16 16:28:54 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-3739-enterprise', '7.0.0-3739-enterprise', '7.0.0-3739-enterprise']
2020-11-16 16:28:54 | INFO | MainProcess | Cluster_Thread | [task.execute] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:28:54 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.201:8091 to cluster
2020-11-16 16:28:54 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.201:8091 to this cluster @172.23.104.105:8091
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.105,ns_1@172.23.104.107,ns_1@172.23.104.136,ns_1@172.23.104.201', 'ejectedNodes': 'ns_1@172.23.104.136', 'user': 'Administrator', 'password': 'password'}
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2020-11-16 16:29:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00%
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc0/_view/views0?stale=ok
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc0
2020-11-16 16:29:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:00 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:02 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:04 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:06 2020
2020-11-16 16:29:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:06 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:08 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:10 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:12 2020
2020-11-16 16:29:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:12 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:14 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:16 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:18 2020
2020-11-16 16:29:18 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:29:18 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 28.00 %
2020-11-16 16:29:18 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 28.00%
2020-11-16 16:29:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:29:19 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc0/_view/views1?stale=ok
2020-11-16 16:29:19 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc0
2020-11-16 16:29:19 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:19 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:21 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:23 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:25 2020
2020-11-16 16:29:25 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:25 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:27 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:27 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:29 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:29 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.136:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:31 2020
2020-11-16 16:29:31 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:31 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:33 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:35 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 401 reason: unauthorized b'{"error":"unauthorized","reason":"password required"}\n' auth: Administrator:password
2020-11-16 16:29:35 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:37 2020
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.00%
2020-11-16 16:29:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc1/_view/views0?stale=ok
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc1
2020-11-16 16:29:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:37 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:39 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:41 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:43 2020
2020-11-16 16:29:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:43 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:45 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:45 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:29:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:47 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:49 2020
2020-11-16 16:29:49 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:49 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:51 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:29:53 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:53 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:29:55 2020
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 %
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 70.00%
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc1/_view/views1?stale=ok
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc1
2020-11-16 16:29:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:55 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:29:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:29:58 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:00 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:02 2020
2020-11-16 16:30:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:02 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:30:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:04 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:30:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:06 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.136:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:08 2020
2020-11-16 16:30:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:08 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:10 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:12 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:14 2020
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 100.00 %
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 100.00%
2020-11-16 16:30:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc2/_view/views0?stale=ok
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc2
2020-11-16 16:30:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:14 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:16 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:18 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:20 2020
2020-11-16 16:30:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:20 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:30:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:22 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.136:8091. Retrying.
2020-11-16 16:30:24 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://172.23.104.136:8091/nodes/self error [Errno 111] Connection refused 
2020-11-16 16:30:27 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.136:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:27 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.136:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:29 2020
2020-11-16 16:30:29 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:29 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:31 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:31 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:33 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:35 2020
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00%
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 95.50599002838135 sec
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc2/_view/views1?stale=ok
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc2
2020-11-16 16:30:35 | INFO | MainProcess | test_thread | [createdeleteview._execute_ddoc_ops] Processing Create DDoc Operation On Bucket bucket0
2020-11-16 16:30:35 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:35 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:37 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:39 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:41 2020
2020-11-16 16:30:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:41 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:43 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:45 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//default/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:45 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:47 2020
2020-11-16 16:30:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:30:47 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc0/_view/views0?stale=ok
2020-11-16 16:30:47 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc0
2020-11-16 16:30:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:47 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:49 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:49 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:30:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:51 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:30:53 2020
2020-11-16 16:30:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:54 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:56 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:30:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:30:58 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:00 2020
2020-11-16 16:31:00 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:31:00 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc0/_view/views1?stale=ok
2020-11-16 16:31:00 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc0
2020-11-16 16:31:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:00 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:02 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:04 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:06 2020
2020-11-16 16:31:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:06 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:08 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc0 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:10 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc0 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:12 2020
2020-11-16 16:31:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:12 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:31:12 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc1/_view/views0?stale=ok
2020-11-16 16:31:12 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc1
2020-11-16 16:31:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:12 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:14 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:16 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:18 2020
2020-11-16 16:31:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:18 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:20 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:22 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:24 2020
2020-11-16 16:31:24 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:31:24 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc1/_view/views1?stale=ok
2020-11-16 16:31:24 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc1
2020-11-16 16:31:24 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:24 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:26 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:28 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:28 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:30 2020
2020-11-16 16:31:30 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:30 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:32 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:32 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:34 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc1 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:34 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc1 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:36 2020
2020-11-16 16:31:36 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.105:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:36 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:31:36 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc2/_view/views0?stale=ok
2020-11-16 16:31:36 | INFO | MainProcess | Cluster_Thread | [task.check] view : views0 was created successfully in ddoc: dev_ddoc2
2020-11-16 16:31:36 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:36 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:38 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:38 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:40 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:42 2020
2020-11-16 16:31:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:42 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:44 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:44 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:46 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:48 2020
2020-11-16 16:31:48 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document
2020-11-16 16:31:48 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/bucket0/_design/dev_ddoc2/_view/views1?stale=ok
2020-11-16 16:31:48 | INFO | MainProcess | Cluster_Thread | [task.check] view : views1 was created successfully in ddoc: dev_ddoc2
2020-11-16 16:31:48 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-3739-enterprise', '7.0.0-3739-enterprise', '7.0.0-3739-enterprise']
2020-11-16 16:31:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root, ip:172.23.104.107 port:8091 ssh_username:root, ip:172.23.104.201 port:8091 ssh_username:root]
2020-11-16 16:31:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:48 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:50 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:50 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
2020-11-16 16:31:52 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.107:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:52 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.107:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:31:54 2020
2020-11-16 16:31:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:54 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:56 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
2020-11-16 16:31:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.201:8092//bucket0/_design/dev_ddoc2 body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password
2020-11-16 16:31:58 | INFO | MainProcess | Cluster_Thread | [task._check_ddoc_replication_on_nodes] Design Doc dev_ddoc2 not yet available on node 172.23.104.201:8091. Retrying.
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 2841 in check>, <FrameSummary file lib/tasks/task.py, line 2924 in _check_ddoc_replication_on_nodes>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:32:00 2020
2020-11-16 16:32:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.105:8091',default bucket
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.105:8091',bucket0 bucket
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.107:8091',default bucket
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.107:8091',bucket0 bucket
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.201:8091',default bucket
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:32:01 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.201:8091',bucket0 bucket
2020-11-16 16:32:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:32:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:32:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:32:02 | INFO | MainProcess | test_thread | [task.__init__] 200000 items will be verified on default bucket on scope None on collection None
2020-11-16 16:32:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:32:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:32:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:32:02 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified
2020-11-16 16:32:02 | INFO | MainProcess | test_thread | [task.__init__] 200000 items will be verified on bucket0 bucket on scope None on collection None
2020-11-16 16:32:03 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified
2020-11-16 16:32:44 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified
2020-11-16 16:32:44 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified
2020-11-16 16:33:14 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified
2020-11-16 16:33:15 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified
2020-11-16 16:33:44 | INFO | MainProcess | load_gen_task | [task.has_next] 150000 items were verified
2020-11-16 16:33:46 | INFO | MainProcess | load_gen_task | [task.has_next] 150000 items were verified
2020-11-16 16:34:14 | INFO | MainProcess | load_gen_task | [task.has_next] 200000 items were verified in 131.36259579658508 sec.the average number of ops            - 1522.0 per second
2020-11-16 16:34:16 | INFO | MainProcess | load_gen_task | [task.has_next] 200000 items were verified in 134.49612879753113 sec.the average number of ops            - 1487.0 per second
2020-11-16 16:34:16 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-3739-enterprise', '7.0.0-3739-enterprise', '7.0.0-3739-enterprise']
2020-11-16 16:34:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.105 port:8091 ssh_username:root, ip:172.23.104.107 port:8091 ssh_username:root, ip:172.23.104.201 port:8091 ssh_username:root]
2020-11-16 16:34:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',default bucket
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',default bucket
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',default bucket
2020-11-16 16:34:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 default
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 default
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 default
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 400000 == 400000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',default bucket
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',bucket0 bucket
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:34:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',bucket0 bucket
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 200000 == 200000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',bucket0 bucket
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.105:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.107:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.201:11210 bucket0
2020-11-16 16:34:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 400000 == 400000 expected on '172.23.104.105:8091''172.23.104.107:8091''172.23.104.201:8091',bucket0 bucket
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] DDoc Validation Started
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc0 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc1 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc2 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc0 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc1 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_ddoc2 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}}
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] DDoc Validation Successful
2020-11-16 16:34:20 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_data_all_buckets] DDoc Data Validation Started on bucket default. Expected Data Items 200000
2020-11-16 16:34:21 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000
2020-11-16 16:34:26 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.104.105:8092/default/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000
2020-11-16 16:34:28 | INFO | MainProcess | Cluster_Thread | [task.check] Server: 172.23.104.105, Design Doc: dev_ddoc0, View: views0, (200000 rows) expected, (66807 rows) returned
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 104 in step>, <FrameSummary file lib/tasks/task.py, line 3068 in check>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]
Mon Nov 16 16:34:28 2020
[<FrameSummary file /usr/local/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file testrunner.py, line 558 in run>, <FrameSummary file /usr/local/lib/python3.7/unittest/runner.py, line 176 in run>, <FrameSummary file /usr/local/lib/python3.7/unittest/case.py, line 676 in __call__>, <FrameSummary file /usr/local/lib/python3.7/unittest/case.py, line 628 in run>, <FrameSummary file pytests/view/createdeleteview.py, line 492 in rebalance_in_and_out_with_ddoc_ops>, <FrameSummary file pytests/view/createdeleteview.py, line 275 in _verify_ddoc_data_all_buckets>, <FrameSummary file lib/couchbase_helper/cluster.py, line 644 in query_view>, <FrameSummary file lib/tasks/future.py, line 160 in result>, <FrameSummary file lib/tasks/future.py, line 111 in __get_result>]
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.105:8091 => {'services': ['kv'], 'cpu_utilization': 10.07830260166709, 'mem_free': 3112906752, 'mem_total': 4220854272, 'swap_mem_used': 0, 'swap_mem_total': 3758092288}
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.107:8091 => {'services': ['kv'], 'cpu_utilization': 7.583058584833883, 'mem_free': 3320852480, 'mem_total': 4220854272, 'swap_mem_used': 0, 'swap_mem_total': 3758092288}
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.201:8091 => {'services': ['kv'], 'cpu_utilization': 7.598475222363406, 'mem_free': 3295662080, 'mem_total': 4201844736, 'swap_mem_used': 0, 'swap_mem_total': 3758092288}
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2020-11-16 16:34:28 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.105 with username:root, attempt#1 of 5
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.105 as root
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.107 with username:root, attempt#1 of 5
2020-11-16 16:34:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.107 as root
2020-11-16 16:34:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.136 with username:root, attempt#1 of 5
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.136 as root
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:34:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.201 with username:root, attempt#1 of 5
2020-11-16 16:34:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.201 as root
2020-11-16 16:34:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True
2020-11-16 16:34:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
2020-11-16 16:34:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2020-11-16 16:34:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
ERROR

======================================================================
ERROR: rebalance_in_and_out_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/view/createdeleteview.py", line 492, in rebalance_in_and_out_with_ddoc_ops
    self._verify_ddoc_data_all_buckets()
  File "pytests/view/createdeleteview.py", line 275, in _verify_ddoc_data_all_buckets
    result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)
  File "lib/couchbase_helper/cluster.py", line 644, in query_view
    return _task.result(timeout)
  File "lib/tasks/future.py", line 160, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 112, in __get_result
    raise self._exception
  File "lib/tasks/task.py", line 3045, in check
    raise QueryViewException(self.view_name, raised_error)
membase.api.exception.QueryViewException: Error occured querying view views0: {'from': '172.23.104.107:8092', 'reason': 'Design document `_design/dev_ddoc0` missing in database `default/master`.'}{'from': '172.23.104.201:8092', 'reason': 'Design document `_design/dev_ddoc0` missing in database `default/master`.'}

----------------------------------------------------------------------
Ran 1 test in 492.630s

FAILED (errors=1)
suite_tearDown (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: suite_tearDown (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: type object 'CreateDeleteViewTests' has no attribute 'suite_tearDown'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
Cluster instance shutdown with force
summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 0 , fail 1
failures so far...
view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops
testrunner logs, diags and results are available under /data/workspace/centos-p0-temp-vset00-00-temp_job/logs/testrunner-20-Nov-16_16-26-20/test_1

*** Tests executed count: 1

Run after suite setup for view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops
During the test, Remote Connections: 0, Disconnections: 40
('Thread', <TaskManager(Cluster_Thread, started 140257339549440)>, 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', <paramiko.Transport at 0x37762850 (unconnected)>, 'was not properly terminated, will be terminated now.')
Stopping the thread...
*** TestRunner ***
[description-setter] Could not determine description.
Archiving artifacts
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
Notifying upstream projects of job completion
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Not sending mail to unregistered user daver@couchbase.com
Not sending mail to unregistered user ajay.bhullar@couchbase.com
Not sending mail to unregistered user timofey.barmin@couchbase.com
Not sending mail to unregistered user Hemant.rajput@couchbase.com
Not sending mail to unregistered user ashwintrojan+github@gmail.com
Sending email to: pierre.regazzoni@couchbase.com evgeny.makarenko@couchbase.com thuan@couchbase.com arunkumar.senthilnathan@couchbase.com mihir.kamdar@couchbase.com asad.zaidi@couchbase.com vikas.chaudhary@couchbase.com ritam@couchbase.com umang.agrawal@couchbase.com balakumaran.gopal@couchbase.com jake.rawsthorne@couchbase.com hemant.rajput@couchbase.com
Finished: UNSTABLE