[memcached][GSI] "COUCHBASE: Unhandled memcached status=0x8" error observed during kv failover while kvOps running when index is being created

[2017-05-29 02:23:54,071] - [rest_client:783] ERROR - POST http://172.23.123.75:8093/query?statement=BUILD+INDEX+on+default%28employeee4b2f8fc590141f68f3d5ab240647badjob_title%2Cemployeee4b2f8fc590141f68f3d5ab240647badjoin_yr%29+USING+GSI body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 500 reason: unknown {

"requestID": "a30290d0-7cd6-4bcc-9cbc-0d7a371055e1",

"signature": null,

"results": [

],

"errors": [{"code":5000,"msg":"BuildIndexes - cause: Build index fails.  Index employeee4b2f8fc590141f68f3d5ab240647badjob_title will retry building in the background for reason: Bucket default In Recovery.\nBuild index fails.  Index employeee4b2f8fc590141f68f3d5ab240647badjoin_yr will retry building in the background for reason: Bucket default In Recovery.\n"}],

"status": "errors",

"metrics": {"elapsedTime": "243.16222ms","executionTime": "243.104996ms","resultCount": 0,"resultSize": 0,"errorCount": 1}

} auth: Administrator:password

[2017-05-29 02:23:54,073] - [task:88] ERROR - Unexpected exception [CBQError: host 172.23.123.75: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'BuildIndexes - cause: Build index fails.  Index employeee4b2f8fc590141f68f3d5ab240647badjob_title will retry building in the background for reason: Bucket default In Recovery.\nBuild index fails.  Index employeee4b2f8fc590141f68f3d5ab240647badjoin_yr will retry building in the background for reason: Bucket default In Recovery.\n', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'243.16222ms', u'executionTime': u'243.104996ms', u'result] caught

[2017-05-29 02:23:54,074] - [task:89] ERROR -   File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap

    self.__bootstrap_inner()

  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner

    self.run()

  File "lib/tasks/taskmanager.py", line 31, in run

    task.step(self)

  File "lib/tasks/task.py", line 75, in step

    self.execute(task_manager)

  File "lib/tasks/task.py", line 2566, in execute

    self.set_unexpected_exception(e)

  File "lib/tasks/task.py", line 89, in set_unexpected_exception

    self.log.error(''.join(traceback.format_stack()))

ERROR

[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 75, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 2566, 'execute', 'self.set_unexpected_exception(e)'), ('lib/tasks/task.py', 90, 'set_unexpected_exception', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]

Mon May 29 02:23:54 2017

[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/2i/recovery_2i.py', 297, 'test_failover', 'mid_recovery_tasks = self.async_run_operations(phase="in_between")'), ('pytests/2i/base_2i.py', 475, 'async_run_operations', 'tasks += self.async_multi_create_index(buckets, query_definitions)'), ('pytests/2i/base_2i.py', 174, 'async_multi_create_index', 'build_index_task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]

[2017-05-29 02:23:54,076] - [recovery_2i:33] INFO - <<<<<< WILL DROP THE INDEXES >>>>>

[2017-05-29 02:23:55,076] - [tuq_helper:76] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4b2f8fc590141f68f3d5ab240647badjob_title'

[2017-05-29 02:23:55,086] - [rest_client:3054] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4b2f8fc590141f68f3d5ab240647badjob_title%27

Exception in thread Cluster_Thread:

Traceback (most recent call last):

  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner

    self.run()

  File "lib/tasks/taskmanager.py", line 31, in run

    task.step(self)

  File "lib/tasks/task.py", line 75, in step

    self.execute(task_manager)

  File "lib/tasks/task.py", line 2646, in execute

    check = self.n1ql_helper._is_index_in_list(self.bucket, self.index_name, server = self.server)

  File "lib/couchbase_helper/tuq_helper.py", line 558, in _is_index_in_list

    res = self.run_cbq_query(query = query, server = server)

  File "lib/couchbase_helper/tuq_helper.py", line 113, in run_cbq_query

    raise CBQError(error_result, server.ip)

CBQError: CBQError: host 172.23.123.75: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'error: Bucket default In Recovery\n. Index employeee4b2f8fc590141f68f3d5ab240647badjob_title([`job_title`]). Index state: deferred', u'code': 12014}, {u'msg': u'error: Bucket default In Recovery\n. Index employeee4b2f8fc590141f68f3d5ab240647badjoin_yr([`join_yr`]). Index state: deferred', u'code': 12014}], u'results': [], u'metrics': {u'elapsedTime': u'243.550884ms', u'executionTime': u'243.4954ms', u'resultSize': 0, u'resultCount': 0, u'errorCount':

COUCHBASE: Unhandled memcached status=0x8 COUCHBASE: Unhandled memcached status=0x8