Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.6.0
Affects Version/s: 7.6.0
Component/s: secondary-index
Labels:
- approved-for-trinity
- system-test
Environment:
7.6.0-2095

Triage:
Untriaged
Story Points:
0
Is this a Regression?:
Unknown

Description

Seen during iteration 17.

There are a few crashes in indexer because of the panics. Unsure if they have the same root cause -

172.23.106.176: crash

[user:info,2024-02-04T17:16:36.440-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:17:51.144-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:18:51.841-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:19:59.258-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:21:26.052-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:22:47.207-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:24:04.204-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:25:45.263-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:27:32.318-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

[user:info,2024-02-04T17:29:23.445-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages:

The first crash occurred during a rebalance -

2024-02-04T17:16:33.497-08:00 [Info] Timekeeper::stopTimer INIT_STREAM bucket2:scope_5:coll_5

fatal error: concurrent map read and map write

goroutine 68349406 [running]:

github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats.func1.2(0x5?, {0x0, 0x0})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4361 +0xbe

github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc02cdf7458)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x7b

github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats.func1()

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4378 +0x778

created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats in goroutine 151

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4311 +0x413

goroutine 1 [chan receive]:

iteration 18

Cbcollect logs:

iteration 17

Iteration 16 -

Cbcollect logs:

Iteration 15 -

Cbcollect logs:

Iteration 14 -

Cbcollect logs:

Iteration 13 -

Cbcollect logs:

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

For Gerrit Dashboard: MB-60666
#	Subject	Branch	Project	Status	CR	V
205180,5	MB-60666 Fix concurrent map read and map write error	unstable	indexing	Status: MERGED	+2	+1
205220,4	MB-60666 Initialize indexer and bucket stats for empty partition instances	unstable	indexing	Status: MERGED	+2	+1
205285,2	Merging fixes for MB-60362 and MB-60666	master	indexing	Status: MERGED	+2	+1
205295,2	Merging fix for MB-60666.	master	indexing	Status: MERGED	+2	+1

Activity

People

Assignee:: Pavan PB

Reporter:: Pavan PB

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Due:: 09/Feb/24

Created:: 05/Feb/24 12:23 AM

Updated:: 16/Feb/24 1:59 AM

Resolved:: 08/Feb/24 6:22 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 4 closed Gerrit changes

Hide There are 4 closed Gerrit changes

MB-60666 Fix concurrent map read and map write error: Gerrit Review:

MB-60666 Initialize indexer and bucket stats for empty partition instances: Gerrit Review:

Merging fixes for MB-60362 and MB-60666: Gerrit Review:

Merging fix for MB-60666.: Gerrit Review:

[System Test] Panics observed - invalid memory address or nil pointer dereference - updateSnapMapAndNotify

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty