Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30575

[Backport MB-29269] System Test : KV swap rebalance hung as rebalance on index nodes never ends

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2497

      The integration longevity test has a step to do a swap rebalance of kv nodes. This is when secondary indexes are created and queries are running along with ongoing kv mutations.

      172.23.96.56 is the outgoing KV node. The index node rebalance progress is stuck at 99% for 14+ hrs. On one of the indexer nodes, 172.23.99.20, the indexer logs is full of errors like the following:

      2018-04-14T22:22:30.216-07:00 [Info] KVSender::sendAddInstancesRequest Projector 172.23.96.56:9999 Topic MAINT_STREAM_TOPIC_15fce2ca5de06fe73740e27404439ea9
      		 
      Instances [indexInstance:<instId:16823689694245000314 state:IndexLoading  definition:<defnID:16023322490735307480 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`result` `rating` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> indexInstance:<instId:6895671228834127721 state:IndexActive  definition:<defnID:16023322490735307480 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`result` `rating` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]
      		 
      2018-04-14T22:22:30.222-07:00 [Error] KVSender::sendAddInstancesRequest Unexpected Error During Add Instances Request Projector 172.23.96.56:9999 Topic MAINT_STREAM_TOPIC_15fce2ca5de06fe73740e27404439ea9 IndexInst [indexInstance:<instId:16823689694245000314 state:IndexLoading  definition:<defnID:16023322490735307480 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`result` `rating` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> indexInstance:<instId:6895671228834127721 state:IndexActive  definition:<defnID:16023322490735307480 bucket:default isPrimary:false name:default_result_ratings_claims_pindex using:0 exprType:N1QL secExpressions:<ud>([`result` `rating` `claim`])</ud> partitionScheme:KEY whereExpression:<ud>()</ud> > singlePartn:<nil>> ]. Err proto: can't skip unknown wire type 6 for protobuf.TimestampResponse
      		 
      2018-04-14T22:22:30.222-07:00 [Error] KVSender::addIndexForExistingBucket MAINT_STREAM default Error Received proto: can't skip unknown wire type 6 for protobuf.TimestampResponse from 172.23.96.56:9999

      Calling it a regression since this step was successful in the prev run for the same script on 5.5.0-2471

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-29269 System Test : KV swap rebalance hung as rebalance on index nodes never ends

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              deepkaran.salooja Deepkaran Salooja
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty