panics in Windows Q2 tests on 7.2.5-7549

We see throughput regression in Windows Q2 tests on 7.2.5-7553 and 7.2.5-7558. I check the runs and see panics in the runs on 7.2.5-7553 and 7.2.5-7558.

https://perf.jenkins.couchbase.com/job/zeus/12334/ 

2024-03-11T14:19:08 [ERROR] Following failures found: "panics": "./zeus-srv-05.perf.couchbase.com.zip": "cbcollect_info_ns_1@zeus-srv-05.perf.couchbase.com_20240311-211511/ns_server.query.log"

panic messages:

fatal error: sync: unlock of unlocked mutex
 
goroutine 2702970 [running]:
sync.fatal({0x277c284?, 0xcdb1ba?})
C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.21.6/go/src/runtime/panic.go:1061 +0x18
sync.(*Mutex).unlockSlow(0xc0522b5640, 0xffffffff)
C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.21.6/go/src/sync/mutex.go:229 +0x35
sync.(*Mutex).Unlock(0xc01b296780?)
C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.21.6/go/src/sync/mutex.go:223 +0x25
github.com/couchbase/query/execution.(*base).notifyParent(0xc0522b5400)
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:1067 +0x8d
github.com/couchbase/query/execution.serializedClose({0x2d96508, 0xc0522b5400}, 0xc0522b5400, 0xc7910ec1bd5c94?)
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:1010 +0x66
github.com/couchbase/query/execution.(*base).close(0xc006bab180, 0xc006bab180?)
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:299 +0x91
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1.2()
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:937 +0xc5
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1()
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:970 +0x436
github.com/couchbase/query/util.(*Once).Do(0x0?, 0x0?)
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:57 +0x2d
github.com/couchbase/query/execution.(*base).runConsumer(...)
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:893
github.com/couchbase/query/execution.(*InitialProject).RunOnce(0xc0a9e9ded8?, 0xc0a9e9dfd0?, {0x0?, 0x0?})
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/project_initial.go:59 +0x77
github.com/couchbase/query/execution.execOp({0x2d98518, 0xc006bab180}, 0xc0a9e9dfd0?, {0x0, 0x0})
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:542 +0x4e
created by github.com/couchbase/query/execution.(*base).fork in goroutine 2362
C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:554 +0x126

https://showfast.sc.couchbase.com/#/timeline/Windows/n1ql/Q1_Q3/MOI

Avg. Query Throughput (queries/sec), Q2, Singleton Unique Lookup, MOI, request_plus

https://showfast.sc.couchbase.com/#/timeline/Windows/n1ql/Q1_Q3/Plasma

Avg. Query Throughput (queries/sec), Q2, Singleton Unique Lookup, Plasma, request_plus