Bucket flush gives error: error 503 reason: unknown {"_":"Unexpected error: {error,{prepare_flush_failed,['ns_1@172.23.121.178'],[]}}"}

Error seen:

[2016-03-09 16:04:01,117] - [remote_util:2277] INFO - running command.raw on 172.23.121.177: ps aux | grep indexer

[2016-03-09 16:04:01,246] - [remote_util:2314] INFO - command executed successfully

(['498      26793 14.1  1.0 620072 40912 ?        Sl   16:03   0:01 /opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=523b79a7e229ad053e7f46bfe2e315f6 -storageMode=memory_optimized', 'root     26936  0.0  0.0 106060  1484 ?        Ss   16:04   0:00 bash -c ps aux | grep indexer', 'root     26940  0.0  0.0 103244   884 ?        S    16:04   0:00 grep indexer'], [])

[2016-03-09 16:04:08,521] - [rest_client:2410] INFO - Flush for bucket 'default' was triggered

[2016-03-09 16:04:09,553] - [rest_client:757] ERROR - http://172.23.121.177:8091/pools/default/buckets/standard_bucket0/controller/doFlush error 503 reason: unknown {"_":"Unexpected error: {error,{prepare_flush_failed,['ns_1@172.23.121.178'],[]}}"}

ERROR

======================================================================

ERROR: test_insert_json_values (tuqquery.tuq_dml.DMLQueryTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "pytests/tuqquery/tuq_dml.py", line 26, in setUp

    timeout=self.wait_timeout * 5)

  File "lib/couchbase_helper/cluster.py", line 1003, in bucket_flush

    return _task.result(timeout)

  File "lib/tasks/future.py", line 160, in result

    return self.__get_result()

  File "lib/tasks/future.py", line 112, in __get_result

    raise self._exception

BucketFlushFailed: unable to flush bucket standard_bucket0 on the host @ 172.23.121.177

----------------------------------------------------------------------

Ran 1 test in 23.888s

FAILED (errors=1)

test_insert_non_doc_null (tuqquery.tuq_dml.DMLQueryTests) ... [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 71, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 3644, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]

Test which fails:

./testrunner -i /tmp/n1ql_set1.ini gsi_type=memory_optimized,doc-per-day=6,standard_buckets=1,primary_indx_type=GSI,nodes_init=2 -t tuqquery.tuq_dml.DMLQueryTests.test_insert_json_values