[system test] Index graphs are displayed as NA when indexer is down

1. Load data using tpcc workload on the buckets
2. The UI appears to be consistent when I started the run on Friday, today most of the buckets ( 3 days after) have empty stats for index stats.

The getIndexStatus shows "Failed to retrieve metadata from one of the index nodes 10.6.2.167"

Ketakis-MacBook-Pro:~ ketaki$ curl -v -u Administrator:password 10.6.2.164:9102/getIndexStatus | jq '.'

• About to connect() to 10.6.2.164 port 9102 (#0)
• Trying 10.6.2.164...
• Adding handle: conn: 0x7fe9eb003a00
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x7fe9eb003a00) send_pipe: 1, recv_pipe: 0
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 -::- -::- -::- 0* Connected to 10.6.2.164 (10.6.2.164) port 9102 (#0)
• Server auth using Basic with user 'Administrator'
  > GET /getIndexStatus HTTP/1.1
  > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
  > User-Agent: curl/7.30.0
  > Host: 10.6.2.164:9102
  > Accept: /
  >
  < HTTP/1.1 200 OK
  < Content-Type: application/json
  < Date: Tue, 26 May 2015 16:36:21 GMT
  < Transfer-Encoding: chunked
  <
  { [data not shown]
  100 2585 0 2585 0 0 7504 0 -::- -::- -::- 7492
• Connection #0 to host 10.6.2.164 left intact
  {
  "code": "error",
  "error": "Fail to retrieve cluster-wide metadata from index service",
  "failedNodes": [
  "10.6.2.167:9102"
  ],
  "status": [
  { "defnId": 17888528285049684000, "name": "PX_ORDER_LINE", "bucket": "ORDER_LINE", "isPrimary": true, "status": "Ready", "definition": "CREATE PRIMARY INDEX PX_ORDER_LINE ON ORDER_LINE USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 7231048663350699000, "name": "CU_ID_D_ID_W_ID", "bucket": "CUSTOMER", "secExprs": [ "`C_ID`", "`C_D_ID`", "`C_W_ID`" ], "status": "Ready", "definition": "CREATE INDEX CU_ID_D_ID_W_ID ON CUSTOMER(`C_ID`,`C_D_ID`,`C_W_ID`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 9609226901563103000, "name": "IT_ID", "bucket": "ITEM", "secExprs": [ "`I_ID`" ], "status": "Ready", "definition": "CREATE INDEX IT_ID ON ITEM(`I_ID`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 9632591663386833000, "name": "CU_W_ID_D_ID_LAST", "bucket": "CUSTOMER", "secExprs": [ "`C_W_ID`", "`C_D_ID`", "`C_LAST`" ], "status": "Ready", "definition": "CREATE INDEX CU_W_ID_D_ID_LAST ON CUSTOMER(`C_W_ID`,`C_D_ID`,`C_LAST`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 11372528481751935000, "name": "px_history", "bucket": "HISTORY", "isPrimary": true, "status": "Ready", "definition": "CREATE PRIMARY INDEX px_history ON HISTORY USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 14742045013308854000, "name": "seq_num", "bucket": "default", "secExprs": [ "`SequenceNumber`" ], "status": "Ready", "definition": "CREATE INDEX seq_num ON default(`SequenceNumber`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 1503142693051335400, "name": "NO_D_ID_W_ID", "bucket": "NEW_ORDER", "secExprs": [ "`NO_O_ID`", "`NO_D_ID`", "`NO_W_ID`" ], "status": "Ready", "definition": "CREATE INDEX NO_D_ID_W_ID ON NEW_ORDER(`NO_O_ID`,`NO_D_ID`,`NO_W_ID`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 15061203809293226000, "name": "PX_STOCK", "bucket": "STOCK", "isPrimary": true, "status": "Ready", "definition": "CREATE PRIMARY INDEX PX_STOCK ON STOCK USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,
  

  { "defnId": 5235702359980121000, "name": "PX_WAREHOUSE", "bucket": "WAREHOUSE", "isPrimary": true, "status": "Ready", "definition": "CREATE PRIMARY INDEX PX_WAREHOUSE ON WAREHOUSE USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ,

  { "defnId": 7653756408553295000, "name": "OR_O_ID_D_ID_W_ID", "bucket": "ORDERS", "secExprs": [ "`O_ID`", "`O_D_ID`", "`O_W_ID`", "`O_C_ID`" ], "status": "Ready", "definition": "CREATE INDEX OR_O_ID_D_ID_W_ID ON ORDERS(`O_ID`,`O_D_ID`,`O_W_ID`,`O_C_ID`) USING GSI", "hosts": [ "10.6.2.164:8091" ], "completion": 100 }

  ]
  }

Also the error.log on the index node shows
ns_server:error,2015-05-26T09:40:19.513-07:00,ns_1@10.6.2.167:index_status_keeper_worker<0.424.0>:index_rest:get_json:45]Request to http://127.0.0.1:9102/getIndexStatus failed: {error,
{econnrefused,
[{lhttpc_client,
send_request,1,
[

{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
{line,220}]},
{lhttpc_client,
execute,9,
[{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/lhttpc/lhttpc_client.erl"}

,

{line,169}

]},
{lhttpc_client,
request,9,
[

{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/lhttpc/lhttpc_client.erl"}

,

{line,92}

]}]}}

To figure out the issue, I did a couchbase stop/start service, but I could see the missing stats issue much before I did any restarts of couchbase-server.

Logs
https://s3.amazonaws.com/cb-customers/indexer/1/collectinfo-2015-05-26T162139-ns_1%4010.6.2.164.zip
https://s3.amazonaws.com/cb-customers/indexer/1/collectinfo-2015-05-26T162139-ns_1%4010.6.2.168.zip
https://s3.amazonaws.com/cb-customers/indexer/1/collectinfo-2015-05-26T162139-ns_1%4010.6.2.194.zip
https://s3.amazonaws.com/cb-customers/indexer/1/collectinfo-2015-05-26T162139-ns_1%4010.6.2.195.zip