Major Description
Discovered this behaviour during an iteration of the experiment mentioned in MB-47441.
Setup:
- Cluster setup: 2 nodes, one with FTS and KV services enabled and another with only FTS enabled(which is added later followed by rebalance).
- FTS quota - 17GB, KV - 8GB
- System specific details - each node has 8 cores, 32 GB RAM and 128 GB storage with Centos 64-bit OS.
- Each index has 1 partition and indexes a field having 400 chars and 2 numeric fields; each document has 1000 chars and 2 numeric fields.
Details:
- Indexes were created as per the procedure mentioned in
MB-47441on the first node({FTS,KV}), till the count reached 1000. - The second node ({FTS}) was later added to the cluster and a rebalance was performed.
- There were indexes being created on the new node, however the operation was cut short and couldn't complete.
- Around the timestamp 2021-11-08T06:33:04, there was an OOM kill which did cause the rebalance to stop and fts to restart. Before the fts was restarted, observed the following message in debug.log:
[user:info,2021-11-08T06:33:04.140-08:00,ns_1@172.23.105.216:<0.23589.595>:ns_log:consume_log:76]Service 'fts' exited with status 137. Restarting. Messages:2021-11-08T06:32:50.399-08:00 [INFO] feed_dcp_gocbcore: releaseAgent, ref count decremented for DCPagent (key: fakedata:9c35726e73ec9ae9e091c23ebd11bb7d, agent: 0xc2f8cf8400, ref count: 5, number of agents for key: 167)2021-11-08T06:32:50.402-08:00 [INFO] feed_dcp_gocbcore: Close, name: fakedata_133_3001d169f9fe68c1_4c1c55842021-11-08T06:32:56.068-08:00 [INFO] pindex: fakedata_133_3001d169f9fe68c1_4c1c5584 Close started with remove: true2021-11-08T06:32:56.075-08:00 [INFO] pindex_bleve: batchWorker stopped for `fakedata_133_3001d169f9fe68c1_4c1c5584`2021-11-08T06:32:56.075-08:00 [INFO] pindex_bleve: batchWorker stopped for `fakedata_133_3001d169f9fe68c1_4c1c5584`2021-11-08T06:32:56.075-08:00 [INFO] pindex_bleve: batchWorker stopped for `fakedata_133_3001d169f9fe68c1_4c1c5584`2021-11-08T06:32:56.075-08:00 [INFO] pindex_bleve: batchWorker stopped for `fakedata_133_3001d169f9fe68c1_4c1c5584`2021-11-08T06:32:56.085-08:00 [INFO] pindex: fakedata_133_3001d169f9fe68c1_4c1c5584 Close completed successfully2021-11-08T06:32:56.178-08:00 [INFO] janitor: feeds to remove: 02021-11-08T06:32:56.178-08:00 [INFO] janitor: feeds to add: 0 |
