select count(*) failed with error "Failed to get count for keyspace"

Steps:

• 3 KV nodes, 3 index+n1ql nodes

  +---------------+-------------+-----------------+-----------+----------+

  | Node          | Services    | CPU_utilization | Mem_total | Mem_free |

  +---------------+-------------+-----------------+-----------+----------+

  | 172.23.108.70 | index       | 12.9393807488   | 4.03 GiB  | 3.31 GiB |

  | 172.23.108.69 | kv          | 19.1076349684   | 4.03 GiB  | 3.32 GiB |

  | 172.23.108.74 | index, n1ql | 13.6958680804   | 4.03 GiB  | 3.13 GiB |

  | 172.23.108.67 | kv          | 46.9769876695   | 4.03 GiB  | 2.92 GiB |

  | 172.23.108.72 | index       | 18.8286985867   | 4.03 GiB  | 3.13 GiB |

  | 172.23.108.68 | kv          | 16.8279576129   | 4.03 GiB  | 3.28 GiB |

  +---------------+-------------+-----------------+-----------+----------+

• 3 buckets (1 couchstore, 1 magma and 1 Eph with replica=1)

  +---------+-----------+-----------------+----------+

  | Bucket  | Type      | Storage Backend | Replicas |

  +---------+-----------+-----------------+----------+

  | bucket1 | couchbase | couchstore      | 1        |

  | bucket2 | ephemeral | -               | 1        |

  | default | couchbase | magma           | 1        |

  +---------+-----------+-----------------+----------+

• Create index on each collection and wait for indexes to become ready

  > create index `{0}_{1}_{2}_index` on `{0}`.`{1}`.`{2}`(name) WITH {{ "defer_build": true, "num_replica": 1 }};

  > BUILD INDEX on `{0}`.`{1}`.`{2}` (`{0}_{1}_{2}_index`) USING GSI

• While indexes are building, kill memcached on KV nodes
• Once indexes are ready, run the following query on each collection

  SELECT count(*) as num_items from `{0}`.`{1}`.`{2}`

Observation:

During the select count query on Ephemeral bucket (bucket2), the query fails with following error,

IndexFailureException: com.couchbase.client.core.error.IndexFailureException: The server reported an issue with the underlying index {"completed":true,"coreId":"0x284d28f10000001f","errors":[{"code":12006,"message":"Failed to get count for keyspace default:bucket2 - cause: write tcp 172.23.108.74:46360->172.23.108.67:11210: write: broken pipe","retry":true}],"httpStatus":200,"idempotent":false,"lastDispatchedFrom":"172.23.106.205:59336","lastDispatchedTo":"172.23.108.74:8093","requestId":761591,"requestType":"QueryRequest","retried":0,"service":{"operationId":"937070d2-08a7-40e6-b5fc-6006b79482dd","statement":"SELECT count(*) as num_items from `bucket2`.`_default`.`collection-3`","type":"query"},"timeoutMs":75000,"timings":{"dispatchMicros":13445,"totalDispatchMicros":13445,"totalMicros":13775}} 

TAF test:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini -p get-cbcollect-info=False,skip_cluster_reset=False,rerun=False,upgrade_version=7.2.1-5878 -t bucket_collections.oso_backfill_test.KvOsoBackfillTests.test_dcp_backfill_config,nodes_init=6,replicas=1,num_items=5000,load_collections_exponentially=True,services_init=kv-kv-kv-index-index-index:n1ql,bucket_spec=multi_bucket.buckets_with_similar_hierarchy,oso_dcp_backfill=auto,simulate_error=kill_memcached,kv_quota_percent=100,index_quota_percent=100,err_itrs=20,GROUP=OSO_DCP_BACKFILL_P0,skip_collections_cleanup=True'