Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: query
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.74.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.106.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.106.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.241.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613021697/collectinfo-2021-02-11T053501-ns_1%40172.23.97.74.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-4432
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 3
Iteration : 1st

The N1QL workload to run queries against collection started at 2021-02-10T16:48:07. There are 2 N1QL nodes in the cluster - 172.23.97.149 & 172.23.97.150.

At 21:05, 172.23.97.150 got OOM killed by the OS and 21:14, 172.23.97.149 got OOM killed.

The attached graphs for 172.23.97.149 & 172.23.97.150 shows that the available memory gradually declined from the time the N1QL workload was started and ultimately got OOM killed. After restarting, we can see the gradual decline start again. The N1QL request rate is mostly constant along this time.

Attached logs for review.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Screen Shot 2021-02-10 at 11.07.22 PM.png
30 kB
10/Feb/21 11:16 PM
Screen Shot 2021-02-10 at 11.16.35 PM.png
37 kB
10/Feb/21 11:16 PM

Issue Links

is duplicated by

MB-44376 cbq-engine triggering Linux OOM-killer during rebalance

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

For Gerrit Dashboard: MB-44247
#	Subject	Branch	Project	Status	CR	V
146354,3	MB-39084 MB-44247 clear slice buffers before pooling, some pool cleanup	master	query	Status: MERGED	+2	+1
146459,3	MB-44247 truncate slices before pooling	master	query	Status: MERGED	+2	+1
146742,1	MB-44247 partially revert index pooling	master	query	Status: ABANDONED	0	0
146993,3	MB-44247 use pointer receive for index pool	master	query	Status: MERGED	+2	+1

Activity

People

Assignee:: Mihir Kamdar (Inactive)

Reporter:: Mihir Kamdar (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 10/Feb/21 11:17 PM

Updated:: 17/Jun/21 3:31 PM

Resolved:: 26/Feb/21 9:02 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 4 closed Gerrit changes

Hide There are 4 closed Gerrit changes

MB-39084 MB-44247 clear slice buffers before pooling, some pool cleanup: Gerrit Review:

MB-44247 truncate slices before pooling: Gerrit Review:

MB-44247 partially revert index pooling: Gerrit Review:

MB-44247 use pointer receive for index pool: Gerrit Review:

[System Test] : cbq-engine OOM killed on 2 nodes

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty