Details
-
Bug
-
Resolution: Done
-
Major
-
Elixir
-
Untriaged
-
-
0
-
Unknown
Description
- Create 19 databases to avoid kv/gsi auto-scaling due to num_tenants.
- Create 10 collections(VolumeCollection0, VolumeCollection1....VolumeCollection0) in each database
- Create 100 indexes on each database i.e. 10 on each collection. This results in total 1900 indexes.
- Build 10 indexes per database at a time. Example:
Bucket0:VolumeCollection0 - 10 indexes under build
Bucket1:VolumeCollection0 - 10 indexes under build
Bucket2:VolumeCollection0 - 10 indexes under build
.
.
.
Bucket19:VolumeCollection0 - 10 indexes under build
- Observe that index build is getting started and aborted immediately. Which is reported in
MB-55276and discussed further inMB-55019 - Delete all the databases at this state and observe that indexer is trying to build the queued indexes for deleted databases which leads to more issues like rebalance failures
Rebalance exited with reason {service_rebalance_failed,index,
{worker_died,
{'EXIT',<0.11246.77>,
{task_failed,rebalance,
{service_error,
<<"[Bucket does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time.] Bucket volumetestbucket-5-cn0uqk does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time.">>}}}}}.
Rebalance Operation Id = 1f2fad995e51bc229f70a18dfefe75aa
followed by:
Service 'index' exited with status 2. Restarting. Messages:
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageRollbackDone(0x400007f400, {0x1a9a7c8?, 0x403c565c40?})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4102 +0x4a8
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0x400007f400, {0x1a9a7c8, 0x403c565c40})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1540 +0x41c
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400007f400)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1130 +0x274
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x27fd470?)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:556 +0x2024
main.main()
goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:206 +0xf90