Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46551

[Query] - Error getting random entry from keyspace - cause: MCResponse status=KEY_ENOENT during upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • query
    • 6.6.2-9588 -> 7.0.0-5226
    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      Steps to Repro
      1. Run the following longevity on 6.6.2 for 3-4 days

      ./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      2. We have 27 node cluster in 6.6.2
      3. Add 6 nodes(1 of each service - 7.0.0-5226) and remove 6 nodes(6.6.2) and do a swap rebalance to upgrade the cluster.
      4. Failover 6 node(1 of each service - 6.6.2), upgrade, do a recovery and rebalance. Noticed errors like the following.

      172.23.106.117 : query

      Sample result for error message 'Error getting documents for infer.
      Error getting random entry from keyspace - cause: MCResponse status=KEY_ENOENT, opcode=0xb6, opaque=0, msg: ' at time 2021-05-25T09:29:59.691-07:00: [{u'completed_requests': {u'node': u'172.23.106.117:8091', u'errors': [{u'message': u'Error getting documents for infer.\nError getting random entry from keyspace - cause: MCResponse status=KEY_ENOENT, opcode=0xb6, opaque=0, msg: ', u'code': 0, u'key': u''}], u'scanConsistency': u'unbounded', u'state': u'completed', u'phaseOperators': {u'authorize': 1}, u'serviceTime': u'1.282805054s', u'remoteAddr': u'172.23.104.244:43354', u'elapsedTime': u'1.282884582s', u'resultSize': 0, u'requestTime': u'2021-05-25T09:29:59.691-07:00', u'statement': u'infer `CUSTOMER`  with {"infer_timeout":5, "max_schema_MB":1};', u'requestId': u'd274208a-664e-4f55-a7ce-8ca36c1e34ee', u'clientContextID': u'INTERNAL-0cd5e617-d9f5-43ab-b87c-c4e0f900a537', u'userAgent': u'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.
      0.4430.212 Safari/537.36 (Couchbase Query Workbench (6.6.2-9588-enterprise))', u'users': u'Administrator', u'resultCount': 0, u'errorCount': 1, u'phaseTimes': {u'authorize': u'40.249382ms', u'parse': u'410.08\xb5s', u'instantiate': u'14.985\xb5s', u'run': u'1.282333589s', u'plan': u'21.389\xb5s'}}}]
      

      This was not seen on upgrade from 6.6.2-9588 -> 7.0.0-5141. cbcollect_info attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            kamini.jagtiani Kamini Jagtiani
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty