[MB-44846] [System Test] Seeing negative value for reclaim_pending storage stat - Couchbase database

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Incomplete
Affects Version/s: Cheshire-Cat
Fix Version/s: CheshireCat.Next
Component/s: storage-engine
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.25.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.137.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.155.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.157.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.5.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.67.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.69.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.104.70.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.105.107.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.105.111.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.106.188.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.108.103.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.120.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.121.117.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.121.3.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.123.27.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.123.28.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.251.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.252.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.96.253.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.119.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.121.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.97.242.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.98.135.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.11.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.20.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.21.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615286155/collectinfo-2021-03-09T103557-ns_1%40172.23.99.25.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-4626
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 2
Iteration : 1st

In the system test, on one of the nodes 172.23.104.67, seeing negative value for the reclaim_pending storage stat.

It appears in the indexer logs first time at 2021-03-09T01:54:03.442 for index "Index": "default:default_result", "Id": 12108459570607186121, "PartitionId": 0,

Attachments

Issue Links

duplicates

MB-45084 [System Test] negative value for reclaim_pending stat

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Lynn Straus added a comment - 10/Mar/21 3:06 PM

Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow.

Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.

Lynn Straus added a comment - 10/Mar/21 3:06 PM Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow. Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.

Akhil Mundroy added a comment - 05/Apr/21 10:29 PM

Appears to be dup of ~~MB-45084~~, seeing same behaviour with trim wctx:

2021-03-09T01:37:13.117-08:00 default:default_result 12108459570607186121 BackStore reclaim_pending : 151

2021-03-09T01:37:13.117-08:00 default:default_result 12108459570607186121 BackStore total_ctx : 30

2021-03-09T01:54:13.126-08:00 default:default_result 12108459570607186121 BackStore reclaim_pending : -16926

2021-03-09T01:54:13.126-08:00 default:default_result 12108459570607186121 BackStore total_ctx : 14

Akhil Mundroy added a comment - 05/Apr/21 10:29 PM Appears to be dup of MB-45084 , seeing same behaviour with trim wctx: 2021-03-09T01:37:13.117-08:00 default:default_result 12108459570607186121 BackStore reclaim_pending : 151 2021-03-09T01:37:13.117-08:00 default:default_result 12108459570607186121 BackStore total_ctx : 30 2021-03-09T01:54:13.126-08:00 default:default_result 12108459570607186121 BackStore reclaim_pending : -16926 2021-03-09T01:54:13.126-08:00 default:default_result 12108459570607186121 BackStore total_ctx : 14

Couchbase Server

[System Test] Seeing negative value for reclaim_pending storage stat

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty