Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
Cheshire-Cat
-
Centos 7 64 bit; CB EE 7.0.0-5161
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Steps to Reproduce
1. create 3 kv, 3 fts, 3 index, 1 n1ql node.
2. Create 1 bucket, with 5 scopes, each with 1 collection. each collection has 10 items
Create 5 gsi indexes, 5 fts indexes
3. Rebalance-in 3 kv nodes & after rebalancing, create and drop and fts index for thousand times to create tombstones that will get purged
4. Rebalance-in 3 gsi nodes & after rebalancing, create and drop and fts index for thousand times to create tombstones that will get purged
5. Rebalance-in 3 fts nodes % fter rebalancing, create and drop and fts index for thousand times to create tombstones that will get purged
['[ns_server:debug,2021-05-18T01:01:03.003-07:00,ns_1@172.23.107.91:tombstone_agent<0.2956.6>:tombstone_agent:purge:195]Purged 13912 ns_config tombstone(s) up to timestamp 63788544001. Tombstones:\n']
|
6. Rebalance-out 6 nodes (2 from each service)
+----------------+----------+-----------------------+----------------+--------------+
|
| Nodes | Services | Version | CPU | Status |
|
+----------------+----------+-----------------------+----------------+--------------+
|
| 172.23.107.58 | fts | 7.0.0-5161-enterprise | 17.9577926334 | Cluster node |
|
| 172.23.107.44 | index | 7.0.0-5161-enterprise | 2.12014134276 | Cluster node |
|
| 172.23.105.175 | kv | 7.0.0-5161-enterprise | 3.1652360515 | Cluster node |
|
| 172.23.107.45 | index | 7.0.0-5161-enterprise | 2.23739097459 | --- OUT ---> |
|
| 172.23.121.78 | fts | 7.0.0-5161-enterprise | 16.8953351627 | Cluster node |
|
| 172.23.106.250 | index | 7.0.0-5161-enterprise | 0.479919171508 | Cluster node |
|
| 172.23.107.43 | fts | 7.0.0-5161-enterprise | 17.6747839749 | --- OUT ---> |
|
| 172.23.106.236 | kv | 7.0.0-5161-enterprise | 2.65908188914 | --- OUT ---> |
|
| 172.23.107.85 | kv | 7.0.0-5161-enterprise | 1.87516391293 | Cluster node |
|
| 172.23.106.251 | index | 7.0.0-5161-enterprise | 1.0989010989 | Cluster node |
|
| 172.23.106.233 | kv | 7.0.0-5161-enterprise | 1.98138297872 | Cluster node |
|
| 172.23.106.238 | n1ql | 7.0.0-5161-enterprise | 1.07891105257 | Cluster node |
|
| 172.23.107.54 | kv | 7.0.0-5161-enterprise | 1.64864151939 | --- OUT ---> |
|
| 172.23.107.78 | fts | 7.0.0-5161-enterprise | 16.8958223162 | --- OUT ---> |
|
| 172.23.121.74 | index | 7.0.0-5161-enterprise | 1.09917877448 | --- OUT ---> |
|
| 172.23.107.91 | fts | 7.0.0-5161-enterprise | 15.9317585302 | Cluster node |
|
+----------------+----------+-----------------------+----------------+--------------+
|
This rebalance appears to be stuck even though there are 5 fts indexes only. Waited for ~2 hours before collecting logs
Observations
on 172.23.12.78 fts.log
fts.log:2021-05-18T01:02:24.935-07:00 [WARN] feed_dcp_gocbcore: [fts0_65d41c0f38123762_f4e0a48a] Rollback to seqno: 0, vbuuid: 0 for vb: 223, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
fts.log:2021-05-18T01:02:35.270-07:00 [WARN] (GOCBCORE) memdClient read failure on conn `0636bb40ea5e862b/da14f9850834ed3a` : EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
|
fts.log:2021-05-18T01:02:35.272-07:00 [WARN] (GOCBCORE) memdClient read failure on conn `0636bb40ea5e862b/4f3c35579e796547` : EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
|
at gocbcore_utils.go:617
|
fts.log:2021-05-18T02:35:43.733-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc013570b40 failed to bootstrap: bucket not found -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
|
fts.log:2021-05-18T02:35:48.747-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc013570b40 failed to bootstrap: bucket not found -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
|
fts.log:2021-05-18T02:35:53.762-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc013570b40 failed to bootstrap: bucket not found -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:617
|
also "fts blocked dcp batches" at the bottom of the UI fts page : seems to have grown to a large number
