Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55284

Indexer is trying to build queued indexes for a deleted database.

    XMLWordPrintable

Details

    Description

      1. Create 19 databases to avoid kv/gsi auto-scaling due to num_tenants.
      2. Create 10 collections(VolumeCollection0, VolumeCollection1....VolumeCollection0) in each database
      3. Create 100 indexes on each database i.e. 10 on each collection. This results in total 1900 indexes.
      4. Build 10 indexes per database at a time. Example:

        Bucket0:VolumeCollection0 - 10 indexes under build
        Bucket1:VolumeCollection0 - 10 indexes under build
        Bucket2:VolumeCollection0 - 10 indexes under build
        .
        .
        .
        Bucket19:VolumeCollection0 - 10 indexes under build
        

      5. Observe that index build is getting started and aborted immediately. Which is reported in MB-55276 and discussed further in MB-55019
      6. Delete all the databases at this state and observe that indexer is trying to build the queued indexes for deleted databases which leads to more issues like rebalance failures

        Rebalance exited with reason {service_rebalance_failed,index,
        {worker_died,
        {'EXIT',<0.11246.77>,
        {task_failed,rebalance,
        {service_error,
        <<"[Bucket does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time.] Bucket volumetestbucket-5-cn0uqk does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time.">>}}}}}.
        Rebalance Operation Id = 1f2fad995e51bc229f70a18dfefe75aa
        

        followed by:

        Service 'index' exited with status 2. Restarting. Messages:
        github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageRollbackDone(0x400007f400, {0x1a9a7c8?, 0x403c565c40?})
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4102 +0x4a8
        github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0x400007f400, {0x1a9a7c8, 0x403c565c40})
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1540 +0x41c
        github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400007f400)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1130 +0x274
        github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x27fd470?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:556 +0x2024
        main.main()
        goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:206 +0xf90
        

      cc: Deepkaran Salooja

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            varun.velamuri Varun Velamuri
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty