Loading...

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.2.0
Affects Version/s: 7.1.4, 7.2.0
Component/s: secondary-index
Labels:
- approved-for-7.2.0
- system-test
Environment:
7.1.4-3586

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.137.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.155.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.157.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.67.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.69.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.70.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.105.111.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.106.100.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.106.188.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.108.103.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.120.107.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.120.245.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.121.117.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.123.28.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.148.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.192.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.252.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.253.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.119.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.121.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.122.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.239.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.99.11.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.99.25.zip

Show
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.137.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.155.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.157.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.67.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.69.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.104.70.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.105.111.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.106.100.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.106.188.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.108.103.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.120.107.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.120.245.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.121.117.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.123.28.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.148.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.192.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.252.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.96.253.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.119.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.121.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.122.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.97.239.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.99.11.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1675653402/collectinfo-2023-02-06T031647-ns_1%40172.23.99.25.zip
Story Points:
0
Is this a Regression?:
Yes

Description

This issue is same as MB-54316. Created this ticket as I couldn't find any corresponding 7.1.4/7.2.0 BP ticket for that particular issue.

QE TEST

-test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_couchstore.yml

Day - 1
Cycle - 1
Scale - 3

Panic on GSI node 172.23.106.188

2023-02-05T07:44:33.670-08:00 [Warn] ClusterInfoCache:validateCache - Failed as len(c.nodes): 24 != len(c.nodesvs): 25

2023-02-05T07:44:33.670-08:00 [Info] Validation Failed while updating nodes and nodesvs.. Retrying(1)

2023-02-05T07:44:33.989-08:00 [Info] PeerPipe.doRecieve() : connection terminated for 172.23.104.137:9100. Reason = EOF. Cleaning Pipe.

2023-02-05T07:44:33.989-08:00 [Error] WatcherServer.runOnce() : Watcher terminated unexpectedly.

2023-02-05T07:44:33.990-08:00 [Info] PeerPipe.doRecieve() : connection terminated for 172.23.104.137:9100. Reason = EOF. Cleaning Pipe.

2023-02-05T07:44:33.990-08:00 [Error] WatcherServer.runOnce() : Watcher terminated unexpectedly.

2023-02-05T07:44:35.240-08:00 [Info] DDLServiceMgr::rebalanceDone: Returned

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xacfbd9]

goroutine 74939683 [running]:

github.com/couchbase/indexing/secondary/common.(*internalVersionCache).Get(0x0?, {0xc019b2f5a0?, 0xc01741f190?})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:266 +0x39

github.com/couchbase/indexing/secondary/common.(*internalVersionChecker).getVerFailNodes(0xc035e1db80, 0xc029d75290)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:490 +0x134

github.com/couchbase/indexing/secondary/common.(*internalVersionChecker).getVersion(0xc035e1db80)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:435 +0x3a5

github.com/couchbase/indexing/secondary/common.GetInternalClusterVersion({0x1437730?, 0xc02bf68280}, 0x1)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:765 +0x377

github.com/couchbase/indexing/secondary/manager/client.NewMetadataProvider({0x7fff6629f8e1, 0xe}, {0xc00b6f2408, 0x16}, 0x0, 0x0, {0x1434870?, 0xc005354d00})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:246 +0x3e5

github.com/couchbase/indexing/secondary/indexer.newMetadataProvider({0x7fff6629f8e1, 0xe}, 0xc02c2b0240, 0x20?, {0x1242fef, 0x1d})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1883 +0x745

github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).rebalanceDone(0xc00534e600, 0xc015516cb0, 0x0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:324 +0x17d

github.com/couchbase/indexing/secondary/indexer.notifyRebalanceDone(0x0?, 0x0?)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:302 +0x39

created by github.com/couchbase/indexing/secondary/indexer.(*RebalanceServiceManager).rebalanceOrFailoverDoneCallback.func1.1

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1753 +0x7a

Attachments

Issue Links

Clones

MB-55435 [System Test] Indexer panic: runtime error: invalid memory address or nil pointer dereference

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[BP to 7.2.0] - [System Test] Indexer panic: runtime error: invalid memory address or nil pointer dereference

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty