[python-sdk] ipv6 : UnknownHostError_0x15 (generated, catch UnknownHostError): <RC=0x15[DNS/Hostname lookup failed]

./testrunner -i fqdn.ini  -t buckettests.CreateBucketTests.test_ephemeral_buckets,default_bucket=false,sasl_buckets=2,bucket_size=100

Test Input params:

{'cluster_name': 'fqdn', 'ini': 'fqdn.ini', 'case_number': 1, 'sasl_buckets': '2', 'logs_folder': '/root/jenkins/workspace/centos-p0-ipv6-vset00-00-sanity/logs/testrunner-17-Dec-01_16-10-11/test_1', 'bucket_size': '100', 'default_bucket': 'false', 'num_nodes': 4}

Run before suite setup for buckettests.CreateBucketTests.test_ephemeral_buckets

2017-12-01 16:10:11 | WARNING | MainProcess | MainThread | [data_helper.<module>] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially

test_ephemeral_buckets (buckettests.CreateBucketTests) ... 2017-12-01 16:10:11,866 - root - INFO - connecting to s10505-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:10:12,038 - root - INFO - Connected to s10505-ip6.qe.couchbase.com

2017-12-01 16:10:12,321 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:10:12,748 - root - INFO - Node version in cluster 5.1.0-1480-enterprise

2017-12-01 16:10:12,755 - root - INFO - Node versions in cluster [u'5.1.0-1480-enterprise']

2017-12-01 16:10:12,755 - root - INFO - ==============  basetestcase setup was started for test #1 test_ephemeral_buckets==============

2017-12-01 16:10:12,770 - root - INFO - deleting existing buckets [] on s10505-ip6.qe.couchbase.com

2017-12-01 16:10:12,770 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:14,783 - root - INFO - deleting existing buckets [] on s10506-ip6.qe.couchbase.com

2017-12-01 16:10:14,783 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:16,794 - root - INFO - deleting existing buckets [] on s10507-ip6.qe.couchbase.com

2017-12-01 16:10:16,795 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:18,807 - root - INFO - deleting existing buckets [] on s10508-ip6.qe.couchbase.com

2017-12-01 16:10:18,807 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:20,825 - root - INFO - sleep for 10 secs.  ...

2017-12-01 16:10:30,840 - root - INFO - waiting for ns_server @ s10505-ip6.qe.couchbase.com:8091

2017-12-01 16:10:30,845 - root - INFO - ns_server @ s10505-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:30,848 - root - INFO - waiting for ns_server @ s10506-ip6.qe.couchbase.com:8091

2017-12-01 16:10:30,852 - root - INFO - ns_server @ s10506-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:30,856 - root - INFO - waiting for ns_server @ s10507-ip6.qe.couchbase.com:8091

2017-12-01 16:10:30,859 - root - INFO - ns_server @ s10507-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:30,863 - root - INFO - waiting for ns_server @ s10508-ip6.qe.couchbase.com:8091

2017-12-01 16:10:30,866 - root - INFO - ns_server @ s10508-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:30,867 - root - INFO - ==============  basetestcase cleanup was started for test #1 test_ephemeral_buckets ==============

2017-12-01 16:10:30,883 - root - INFO - deleting existing buckets [] on s10505-ip6.qe.couchbase.com

2017-12-01 16:10:30,883 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:32,893 - root - INFO - deleting existing buckets [] on s10506-ip6.qe.couchbase.com

2017-12-01 16:10:32,893 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:34,902 - root - INFO - deleting existing buckets [] on s10507-ip6.qe.couchbase.com

2017-12-01 16:10:34,903 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:36,911 - root - INFO - deleting existing buckets [] on s10508-ip6.qe.couchbase.com

2017-12-01 16:10:36,912 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely.

2017-12-01 16:10:38,933 - root - INFO - waiting for ns_server @ s10505-ip6.qe.couchbase.com:8091

2017-12-01 16:10:38,937 - root - INFO - ns_server @ s10505-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:38,941 - root - INFO - waiting for ns_server @ s10506-ip6.qe.couchbase.com:8091

2017-12-01 16:10:38,944 - root - INFO - ns_server @ s10506-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:38,948 - root - INFO - waiting for ns_server @ s10507-ip6.qe.couchbase.com:8091

2017-12-01 16:10:38,951 - root - INFO - ns_server @ s10507-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:38,955 - root - INFO - waiting for ns_server @ s10508-ip6.qe.couchbase.com:8091

2017-12-01 16:10:38,958 - root - INFO - ns_server @ s10508-ip6.qe.couchbase.com:8091 is running

2017-12-01 16:10:38,959 - root - INFO - ==============  basetestcase cleanup was finished for test #1 test_ephemeral_buckets ==============

Cluster instance shutdown with force

2017-12-01 16:10:38,965 - root - INFO - initializing cluster

2017-12-01 16:10:39,977 - root - INFO - server: ip:s10505-ip6.qe.couchbase.com port:8091 ssh_username:root, nodes/self: {'ip': u's10505-ip6.qe.couchbase.com', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@s10505-ip6.qe.couchbase.com', 'uptime': u'11798', 'mcdMemoryReserved': 3028, 'storageTotalRam': 3785, 'hostname': u's10505-ip6.qe.couchbase.com:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x286db90>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1480-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'fts', u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3459932160, 'memoryTotal': 3969355776, 'memoryQuota': 1761, 'mcdMemoryAllocated': 3028, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}

2017-12-01 16:10:39,978 - root - INFO - quota for fts service will be 256 MB

2017-12-01 16:10:39,978 - root - INFO - set both index and fts quota at node s10505-ip6.qe.couchbase.com 

2017-12-01 16:10:39,978 - root - INFO - pools/default params : ftsMemoryQuota=256

2017-12-01 16:10:39,981 - root - INFO - pools/default params : memoryQuota=1762

2017-12-01 16:10:39,986 - root - INFO - settings/indexes params : storageMode=plasma

2017-12-01 16:10:39,992 - root - INFO - /node/controller/setupServices params on s10505-ip6.qe.couchbase.com: 8091:services=kv%2Cfts&password=password&hostname=s10505-ip6.qe.couchbase.com&user=Administrator

2017-12-01 16:10:39,995 - root - ERROR - POST http://s10505-ip6.qe.couchbase.com:8091//node/controller/setupServices body: services=kv%2Cfts&password=password&hostname=s10505-ip6.qe.couchbase.com&user=Administrator headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password

2017-12-01 16:10:39,998 - root - INFO - This node is already provisioned with services, we do not consider this as failure for test case

2017-12-01 16:10:39,998 - root - INFO - settings/web params on s10505-ip6.qe.couchbase.com:8091:username=Administrator&password=password&port=8091

2017-12-01 16:10:40,042 - root - INFO - server: ip:s10506-ip6.qe.couchbase.com port:8091 ssh_username:root, nodes/self: {'ip': u's10506-ip6.qe.couchbase.com', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@s10506-ip6.qe.couchbase.com', 'uptime': u'453', 'mcdMemoryReserved': 3028, 'storageTotalRam': 3785, 'hostname': u's10506-ip6.qe.couchbase.com:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7fa0e03991d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1480-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3562528768, 'memoryTotal': 3969355776, 'memoryQuota': 2271, 'mcdMemoryAllocated': 3028, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}

2017-12-01 16:10:40,042 - root - INFO - pools/default params : memoryQuota=2018

2017-12-01 16:10:40,046 - root - INFO - settings/indexes params : storageMode=plasma

2017-12-01 16:10:40,051 - root - INFO - settings/web params on s10506-ip6.qe.couchbase.com:8091:username=Administrator&password=password&port=8091

2017-12-01 16:10:40,256 - root - INFO - server: ip:s10507-ip6.qe.couchbase.com port:8091 ssh_username:root, nodes/self: {'ip': u's10507-ip6.qe.couchbase.com', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@s10507-ip6.qe.couchbase.com', 'uptime': u'11800', 'mcdMemoryReserved': 3028, 'storageTotalRam': 3785, 'hostname': u's10507-ip6.qe.couchbase.com:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7fa0e03991d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1480-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3549249536, 'memoryTotal': 3969355776, 'memoryQuota': 2271, 'mcdMemoryAllocated': 3028, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}

2017-12-01 16:10:40,256 - root - INFO - pools/default params : memoryQuota=2018

2017-12-01 16:10:40,260 - root - INFO - settings/indexes params : storageMode=plasma

2017-12-01 16:10:40,266 - root - INFO - settings/web params on s10507-ip6.qe.couchbase.com:8091:username=Administrator&password=password&port=8091

2017-12-01 16:10:40,314 - root - INFO - server: ip:s10508-ip6.qe.couchbase.com port:8091 ssh_username:root, nodes/self: {'ip': u's10508-ip6.qe.couchbase.com', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@s10508-ip6.qe.couchbase.com', 'uptime': u'11800', 'mcdMemoryReserved': 3028, 'storageTotalRam': 3785, 'hostname': u's10508-ip6.qe.couchbase.com:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7fa0e03991d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1480-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3556687872, 'memoryTotal': 3969355776, 'memoryQuota': 2271, 'mcdMemoryAllocated': 3028, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}

2017-12-01 16:10:40,314 - root - INFO - pools/default params : memoryQuota=2018

2017-12-01 16:10:40,318 - root - INFO - settings/indexes params : storageMode=plasma

2017-12-01 16:10:40,324 - root - INFO - settings/web params on s10508-ip6.qe.couchbase.com:8091:username=Administrator&password=password&port=8091

2017-12-01 16:10:40,368 - root - INFO - **** add built-in 'cbadminbucket' user to node s10505-ip6.qe.couchbase.com ****

2017-12-01 16:10:40,423 - root - INFO - sleep for 10 secs.  ...

2017-12-01 16:10:50,433 - root - INFO - **** add 'admin' role to 'cbadminbucket' user ****

2017-12-01 16:10:50,451 - root - INFO - sleep for 10 secs.  ...

2017-12-01 16:11:00,461 - root - INFO - done initializing cluster

2017-12-01 16:11:01,400 - root - INFO - Node versions in cluster [u'5.1.0-1480-enterprise']

2017-12-01 16:11:01,400 - root - INFO - http://s10505-ip6.qe.couchbase.com:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=bucket0&saslPassword=password&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100

2017-12-01 16:11:01,404 - root - INFO - 0.00 seconds to create bucket bucket0

2017-12-01 16:11:01,430 - root - INFO - Node versions in cluster [u'5.1.0-1480-enterprise']

2017-12-01 16:11:01,430 - root - INFO - http://s10505-ip6.qe.couchbase.com:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=bucket1&saslPassword=password&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100

2017-12-01 16:11:01,434 - root - INFO - 0.00 seconds to create bucket bucket1

2017-12-01 16:11:01,434 - root - INFO - waiting for memcached bucket : bucket0 in s10505-ip6.qe.couchbase.com to accept set ops

2017-12-01 16:11:02,510 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 bucket0

2017-12-01 16:11:02,593 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 bucket0

2017-12-01 16:11:03,069 - root - INFO - bucket 'bucket0' was created with per node RAM quota: 100

2017-12-01 16:11:03,070 - root - INFO - waiting for memcached bucket : bucket1 in s10505-ip6.qe.couchbase.com to accept set ops

2017-12-01 16:11:04,144 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 bucket1

2017-12-01 16:11:04,206 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 bucket1

2017-12-01 16:11:04,428 - root - INFO - bucket 'bucket1' was created with per node RAM quota: 100

2017-12-01 16:11:04,477 - root - INFO - ==============  basetestcase setup was finished for test #1 test_ephemeral_buckets ==============

2017-12-01 16:11:04,484 - root - INFO - sleep for 10 secs.  ...

2017-12-01 16:11:14,503 - root - INFO - Node version in cluster 5.1.0-1480-enterprise

2017-12-01 16:11:14,504 - root - INFO - connecting to s10505-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:11:14,672 - root - INFO - Connected to s10505-ip6.qe.couchbase.com

2017-12-01 16:11:14,980 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:11:15,459 - root - INFO - Node versions in cluster [u'5.1.0-1480-enterprise']

2017-12-01 16:11:15,459 - root - INFO - http://s10505-ip6.qe.couchbase.com:8091/pools/default/buckets with param: bucketType=ephemeral&threadsNumber=3&authType=sasl&name=default&saslPassword=password&evictionPolicy=noEviction&flushEnabled=1&replicaNumber=1&ramQuotaMB=100

2017-12-01 16:11:15,465 - root - INFO - 0.01 seconds to create bucket default

2017-12-01 16:11:15,465 - root - INFO - waiting for memcached bucket : default in s10505-ip6.qe.couchbase.com to accept set ops

2017-12-01 16:11:16,567 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 default

2017-12-01 16:11:16,634 - root - INFO - creating direct client s10505-ip6.qe.couchbase.com:11210 default

2017-12-01 16:11:17,044 - root - INFO - bucket 'default' was created with per node RAM quota: 100

2017-12-01 16:11:17,045 - root - INFO - waiting for bucket creation to complete....

2017-12-01 16:11:17,067 - root - INFO - node s10505-ip6.qe.couchbase.com existing buckets : [u'bucket0', u'bucket1', u'default']

2017-12-01 16:11:17,068 - root - INFO - node s10505-ip6.qe.couchbase.com found bucket default

ERROR

2017-12-01 16:11:17,104 - root - INFO - connecting to s10505-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:11:17,204 - root - INFO - Connected to s10505-ip6.qe.couchbase.com

2017-12-01 16:11:17,513 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:11:17,823 - root - INFO - running command.raw on s10505-ip6.qe.couchbase.com: kill -9 $(ps aux | grep 'memcached'  | awk '{print $2}')

2017-12-01 16:11:17,918 - root - INFO - command executed successfully

2017-12-01 16:11:18,034 - root - INFO - connecting to s10506-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:11:18,134 - root - INFO - Connected to s10506-ip6.qe.couchbase.com

2017-12-01 16:11:18,440 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:11:18,819 - root - INFO - running command.raw on s10506-ip6.qe.couchbase.com: kill -9 $(ps aux | grep 'memcached'  | awk '{print $2}')

2017-12-01 16:11:18,914 - root - INFO - command executed successfully

2017-12-01 16:11:19,030 - root - INFO - connecting to s10507-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:11:19,130 - root - INFO - Connected to s10507-ip6.qe.couchbase.com

2017-12-01 16:11:19,436 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:11:19,788 - root - INFO - running command.raw on s10507-ip6.qe.couchbase.com: kill -9 $(ps aux | grep 'memcached'  | awk '{print $2}')

2017-12-01 16:11:19,882 - root - INFO - command executed successfully

2017-12-01 16:11:19,998 - root - INFO - connecting to s10508-ip6.qe.couchbase.com with username:root password:couchbase ssh_key:

2017-12-01 16:11:20,097 - root - INFO - Connected to s10508-ip6.qe.couchbase.com

2017-12-01 16:11:20,403 - root - INFO - It could be other operating systyem.  Go to check at other location

2017-12-01 16:11:20,781 - root - INFO - running command.raw on s10508-ip6.qe.couchbase.com: kill -9 $(ps aux | grep 'memcached'  | awk '{print $2}')

2017-12-01 16:11:20,877 - root - INFO - command executed successfully

Cluster instance shutdown with force

======================================================================

ERROR: test_ephemeral_buckets (buckettests.CreateBucketTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "pytests/buckettests.py", line 321, in test_ephemeral_buckets

    self._load_all_buckets(self.server, gen_load, "create", 0)

  File "pytests/basetestcase.py", line 878, in _load_all_buckets

    timeout_secs, proxy_client)

  File "pytests/basetestcase.py", line 852, in _async_load_all_buckets

    proxy_client))

  File "lib/couchbase_helper/cluster.py", line 142, in async_load_gen_docs

    _task = LoadDocumentsGeneratorsTask(server, bucket, [generator], kv_store, op_type, exp, flag, only_store_hash, batch_size)

  File "lib/tasks/task.py", line 1006, in __init__

    only_store_hash=only_store_hash, batch_size=batch_size, pause_secs=pause_secs, timeout_secs=timeout_secs)

  File "lib/tasks/task.py", line 945, in __init__

    timeout_secs=timeout_secs)

  File "lib/tasks/task.py", line 681, in __init__

    self.client = VBucketAwareMemcached(RestConnection(server), bucket)

  File "lib/sdk_client.py", line 558, in __init__

    self.client = SDKClient(self.bucket, hosts = [self.host], scheme = self.scheme, password = self.saslPassword)

  File "lib/sdk_client.py", line 34, in __init__

    self._createConn()

  File "lib/sdk_client.py", line 54, in _createConn

    self.cb = cluster.open_bucket(self.bucket)

  File "/usr/lib64/python2.7/site-packages/couchbase/cluster.py", line 100, in open_bucket

    rv = self.bucket_class(str(connstr), **kwargs)

  File "/usr/lib64/python2.7/site-packages/couchbase/bucket.py", line 251, in __init__

    self._do_ctor_connect()

  File "/usr/lib64/python2.7/site-packages/couchbase/bucket.py", line 260, in _do_ctor_connect

    self._connect()

_UnknownHostError_0x15 (generated, catch UnknownHostError): <RC=0x15[DNS/Hostname lookup failed], There was a problem while trying to send/receive your request over the network. This may be a result of a bad network or a misconfigured client or server, C Source=(src/bucket.c,792)>

----------------------------------------------------------------------

Ran 1 test in 69.024s

FAILED (errors=1)

summary so far suite buckettests.CreateBucketTests , pass 0 , fail 1