Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52837

[System Test] [Error] Planner - Unable to read index layout from cluster

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      [System Test] [Error] Encounter planner error for index.  Use round robin strategy for planning. Error: Unable to read index layout from cluster

      =======================

      Build : 7.1.2-3348
      Test : -test tests/2i/neo/test_neo_idx_clusterops_recovery.yml -scope tests/2i/neo/scope_neo_plasma_idx_dgm.yml
      Scale : 3

      Errors in indexer.log on node 172.23.107.2

      Errors in indexer.log

      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 480 252644755215671 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 481 239527019343654 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 483 78137246749009 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 484 89937510196727 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 485 32928067729160 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.751-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 487 10334790740752 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.862-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 459 (117703862092928 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.862-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 459 (117703862092928 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.862-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 491 (229123261642886 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.862-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 491 (229123261642886 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.925-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 459 117703862092928 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.925-07:00 [Info] TK StreamBegin INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd 491 229123261642886 0 67005 172.23.97.232:8091. HWT [0-0,0].
      2022-07-04T06:13:45.976-07:00 [Info] DATP[->dataport ":9103"] connection "172.23.97.235:35962" doAuth() ...
      2022-07-04T06:13:45.976-07:00 [Info] DATP[->dataport ":9103"] connection "172.23.97.235:35962" auth successful
      2022-07-04T06:13:45.977-07:00 [Info] DATP[->dataport ":9103"] new connection "172.23.97.235:35962" +3
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 543 (234350779971003 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 543 (234350779971003 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 575 (207525733542955 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 575 (207525733542955 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 607 (86377196692044 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 607 (86377196692044 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 639 (190329939091987 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 639 (190329939091987 0 67005 18) OSOFilter(Seq:761 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 671 (180541781189993 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 671 (180541781189993 0 67005 18) OSOFilter(Seq:761 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 703 (55993987011610 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 703 (55993987011610 0 67005 18) OSOFilter(Seq:761 Count:2 Start:1 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 735 (198308135773805 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:45.998-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 735 (198308135773805 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:46.108-07:00 [Error] Encounter planner error for index. Use round robin strategy for planning. Error: Unable to read index layout from cluster 127.0.0.1:8091. err = response status:500 cause: Unable to retrieve index metadata
      : Index:bucket5, HaMojxtswgudDpzr1, QqGvtFFFpc5NuA0vVMX1P, idx7_AOC0B8I296_idxprefix
      2022-07-04T06:13:46.108-07:00 [Info] send commit create request to watcher 172.23.97.217:9100 defnID 5907434721235466189
      2022-07-04T06:13:46.178-07:00 [Info] send commit create request to watcher 172.23.107.2:9100 defnID 5907434721235466189
      2022-07-04T06:13:46.178-07:00 [Info] send commit create request to watcher 172.23.107.3:9100 defnID 5907434721235466189
      2022-07-04T06:13:46.178-07:00 [Info] send commit create request to watcher 172.23.107.5:9100 defnID 5907434721235466189
      2022-07-04T06:13:46.178-07:00 [Info] send commit create request to watcher 172.23.97.216:9100 defnID 5907434721235466189
      2022-07-04T06:13:46.180-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 512 (61562058018129 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:46.180-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 512 (61562058018129 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:46.180-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 544 (248904577610700 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:46.180-07:00 [Info] OSOM Got OSO End INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 544 (248904577610700 0 67005 18) OSOFilter(Seq:764 Count:2 Start:1 End:0)
      2022-07-04T06:13:46.180-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 539 (199521312981960 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:46.181-07:00 [Info] OSOM Got OSO Start INIT_STREAM bucket4:_default:pn63PJe6UG2kmrn0cm8iGOP9ikd vb 538 (71593980155354 0 67005 17) OSOFilter(Seq:0 Count:0 Start:0 End:0)
      2022-07-04T06:13:46.305-07:00 [Info] Fetch new stats upon request by life cycle manager
      2022-07-04T06:13:46.315-07:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.97.235:9999 Topic INIT_STREAM_

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.107.2.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.107.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.107.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.215.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.216.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.217.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.227.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.232.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.235.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.236.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1656942097/collectinfo-2022-07-04T134139-ns_1%40172.23.97.237.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sai.teja Sai Krishna Teja
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty