XDCR elasticsearch replication fails: Failed to get cluster version information

When setting up ES with 4.6 Tony Hillman and I got the following error:

In the logs I see the following:

ReplicationSpecService 2017-01-04T16:26:05.813Z [INFO] Start ValidateAddReplicationSpec, sourceBucket=beer-sample, targetCluster=ES, targetBucket=beer-sample

ReplicationSpecService 2017-01-04T16:26:05.829Z [INFO] Result from local bucket look up: err_source=<nil>, time taken=16.375658ms

ReplicationSpecService 2017-01-04T16:26:05.834Z [INFO] Successfully retrieved target cluster reference. time take=5.017739ms

ReplicationSpecService 2017-01-04T16:26:05.835Z [INFO] Validated that source bucket and target bucket are not the same. time taken=373.365µs

ReplicationSpecService 2017-01-04T16:26:06.000Z [INFO] Result from remote bucket look up: err_target=<nil>, time taken=164.80997ms

ReplicationSpecService 2017-01-04T16:26:06.001Z [ERROR] Failed to get cluster version information, err=Can't get nodes information for cluster localhost:9091, err=Failed on calling host=localhost:9091, path=/pools/nodes, err=<nil>, statusCode=404

ReplicationSpecService 2017-01-04T16:26:06.001Z [INFO] Finished ValidateAddReplicationSpec. errorMap=map[toCluster:Failed to get cluster version information, err=Can't get nodes information for cluster localhost:9091, err=Failed on calling host=localhost:9091, path=/pools/nodes, err=<nil>, statusCode=404

]

AdminPort 2017-01-04T16:26:06.001Z [ERROR] Error creating replication. errorsMap=map[toCluster:Failed to get cluster version information, err=Can't get nodes information for cluster localhost:9091, err=Failed on calling host=localhost:9091, path=/pools/nodes, err=<nil>, statusCode=404

]

AdminPort 2017-01-04T16:26:06.001Z [INFO] Finished doCreateReplicationRequest call

These are the steps to reproduce, please note that these steps work on 4.5.1, so I have marked this as a regression:

1. Installing Java, as ES requires it

   wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm"

   sudo yum -y localinstall jdk-8u73-linux-x64.rpm

2. Installing ES

   wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.4.0/elasticsearch-2.4.0.rpm

   sudo yum -y localinstall elasticsearch-2.4.0.rpm

3. Installing the ES Couchbase Plugin

   cd /usr/share/elasticsearch/

   sudo bin/plugin install https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/2.2.4.0-update1/elasticsearch-transport-couchbase-2.2.4.0-update1.zip

4. Edit ES config setting the Username, password and maxConcurrentRequests

   sudo vim /etc/elasticsearch/elasticsearch.yml

   couchbase.password: password

   couchbase.username: Administrator

   couchbase.maxConcurrentRequests: 1024

5. Installing a plugin that provided a web UI for ES:

   sudo bin/plugin install mobz/elasticsearch-head

6. Start ES

   sudo /etc/init.d/elasticsearch start

7. Configure an index template for Elasticsearch to use on the information sent from Couchbase.

   curl -XPUT http://localhost:9200/_template/couchbase -d @plugins/transport-couchbase/couchbase_template.json

8. Creating an Index

   curl -XPUT http://localhost:9200/index-1

9. In the Couchbase UI config the XDCR replication to ES.