Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
Cypher
-
None
-
Untriaged
-
0
-
Unknown
Description
- Load some 100M vector data in KV
- Create index
CREATE INDEX `bigann_100M_PQ` ON `default1`(`color`,`embedding` VECTOR) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "172.23.110.66:18091","172.23.110.67:18091","172.23.110.69:18091" ], "num_partition":8 "dimension":128, "similarity":"L2_SQUARED", "description":"IVF,PQ32x8", "scan_nprobes":20 }
- Start a query load which is using the above index
SELECT RAW id FROM {collection} where color in ['green', 'red'] ORDER BY ANN(embedding, $qvec, 'L2_SQUARED', 20) ASC LIMIT 1000
- Create another index:
CREATE INDEX `bigann_100M_SQ` ON `default1`(`color`,`embedding` VECTOR) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "172.23.110.66:18091","172.23.110.67:18091","172.23.110.69:18091" ], "num_partition":8 "dimension":128, "similarity":"L2_SQUARED", "description":"IVF,SQ8", "scan_nprobes":20 }
- While the above index is still building the and query load is in progress somehow this index is picked up to serve the queries and indexer crashed.
2024-08-20T13:39:56.945-07:00 [Fatal] IndexScanSource2 - panic sync: WaitGroup is reused before previous Wait has returned detected while processing defnId:8993416600612108291, instId:13902447290580810882, index:default1/bigann_100M_SQ, type:vectorscanreq, partitions:[5 8] user:Administrator, span:<ud>(range (nil,nil incl:none))</ud>, limit:9223372036854775807, consistency:any_consistency, requestId:eba5f3a5-9a70-49f5-8720-cd214e4583e8
2024-08-20T13:39:56.983-07:00 [Fatal] goroutine 3349050648 [running]:
github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:179
github.com/couchbase/indexing/secondary/logging.StackTraceAll()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:319 +0x3c
github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource2).Routine.func1()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline_vector.go:790 +0xaa
panic({0x17e3400?, 0x21af360?})
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/runtime/panic.go:770 +0x132
sync.(*WaitGroup).Wait(0xc009f39f80?)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/sync/waitgroup.go:118 +0x74
github.com/couchbase/indexing/secondary/indexer.(*WorkerPool).Stop.func1()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline_vector.go:576 +0x36
sync.(*Once).doSlow(0xc00b336ba0?, 0x0?)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/sync/once.go:65
github.com/couchbase/indexing/secondary/indexer.(*WorkerPool).Stop(0xc011709440?)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline_vector.go:574 +0x3c
github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource2).Routine(0xc00e906070)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline_vector.go:924 +0xa05
github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt.func1()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:75 +0x32
created by github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt in goroutine 1065688906
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:74 +0x79
Attachments
Issue Links
- relates to
-
MB-63348 Panic observer during rebalance in while vector queries workload is running on the system.
- Open