Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37474

[System Test] : index building for one index is stuck - o2_result_rating - DISTRICT bucket

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 7.0.0-1165
      Test : -test tests/integration/test_allFeatures_madhatter.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Day : 2nd

      Seeing that index building for index o2_result_rating has been stuck for several hours. The logs on the indexer node 172.23.96.216 shows the following errors :

      2020-01-08T18:57:07.943-08:00 [Info] KVSender::sendAddInstancesRequest Projector 172.23.96.14:9999 Topic MAINT_STREAM_TOPIC_2e7c4745a69b67cd3f12a8d64e897e28
      		 
      Instances [indexInstance:<instId:12388856652348877335 state:IndexLoading  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> indexInstance:<instId:15306872969831463149 state:IndexActive  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]
      		 
      2020-01-08T18:57:07.949-08:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.96.14:9999 Topic MAINT_STREAM_TOPIC_2e7c4745a69b67cd3f12a8d64e897e28 IndexInst [indexInstance:<instId:12388856652348877335 state:IndexLoading  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> indexInstance:<instId:15306872969831463149 state:IndexActive  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]. Err feed.invalidBucket
      		 
      2020-01-08T18:57:07.949-08:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM DISTRICT Error Received feed.invalidBucket from 172.23.96.14:9999
      		 
      2020-01-08T18:57:07.949-08:00 [Info] KVSender::sendAddInstancesRequest Projector 172.23.96.18:9999 Topic MAINT_STREAM_TOPIC_2e7c4745a69b67cd3f12a8d64e897e28
      		 
      Instances [indexInstance:<instId:12388856652348877335 state:IndexLoading  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> indexInstance:<instId:15306872969831463149 state:IndexActive  definition:<defnID:14577526196373041913 bucket:DISTRICT isPrimary:false name:o2_result_rating using:0 exprType:N1QL secExpressions:<ud>([`rating` `result`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]

      Around the same time in projector (172.23.96.14) log, we see below:

      [root@localhost logs]# zgrep "invalid-bucket" projector.log*
      		 
      projector.log.6.gz:2020-01-08T18:57:07.954-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_2e7c4745a69b67cd3f12a8d64e897e28(127.0.0.1:8091)] ##7a addInstances() invalid-bucket "DISTRICT"
      		 
      projector.log.6.gz:2020-01-08T18:57:07.954-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_2e7c4745a69b67cd3f12a8d64e897e28(127.0.0.1:8091)] ##7a addInstances() invalid-bucket "DISTRICT"

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.113.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.114.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.151.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.152.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.154.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.68.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.104.78.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.18.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.183.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.190.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.191.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.207.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.209.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.210.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.212.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.214.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.215.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.216.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.219.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.220.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.221.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.223.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1578670278/collectinfo-2020-01-10T153119-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-37140 [System Test] : index building for one index is stuck

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-37689 [System Test] : index building for a partitioned primary index is stuck on 1 indexer node

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              varun.velamuri Varun Velamuri
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty