Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-2095
-
Untriaged
-
0
-
Unknown
Description
Seen during iteration 17.
There are a few crashes in indexer because of the panics. Unsure if they have the same root cause -
172.23.106.176: crash |
[user:info,2024-02-04T17:16:36.440-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:17:51.144-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:18:51.841-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:19:59.258-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:21:26.052-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:22:47.207-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:24:04.204-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:25:45.263-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:27:32.318-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
[user:info,2024-02-04T17:29:23.445-08:00,ns_1@172.23.106.176:<0.18797.40>:ns_log:consume_log:76]Service 'index' exited with status 2. Restarting. Messages: |
|
The first crash occurred during a rebalance -
2024-02-04T17:16:33.497-08:00 [Info] Timekeeper::stopTimer INIT_STREAM bucket2:scope_5:coll_5 |
fatal error: concurrent map read and map write
|
|
goroutine 68349406 [running]: |
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats.func1.2(0x5?, {0x0, 0x0}) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4361 +0xbe |
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc02cdf7458) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x7b |
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4378 +0x778 |
created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats in goroutine 151 |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4311 +0x413 |
|
goroutine 1 [chan receive]: |
iteration 18
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707103721/collectinfo-2024-02-05T033849-ns_1%40172.23.97.67.zip
iteration 17
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707099498/collectinfo-2024-02-05T022841-ns_1%40172.23.97.67.zip
Iteration 16 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707095247/collectinfo-2024-02-05T011819-ns_1%40172.23.97.67.zip
Iteration 15 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707091026/collectinfo-2024-02-05T000727-ns_1%40172.23.97.67.zip
Iteration 14 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707086830/collectinfo-2024-02-04T225706-ns_1%40172.23.97.67.zip
Iteration 13 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707082804/collectinfo-2024-02-04T214711-ns_1%40172.23.97.67.zip