Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46575

[UPGRADE] post swap rebalance upgrade seeing error "... enabled only after cluster is fully upgraded and there is no failed node"

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Regression jobs doing swap rebalance with 6.x to 7.0.0 fails on some cases post complete upgrade with following error. For example:

      2021-05-26 15:32:41,088 - root - INFO - query params : statement=CREATE+COLLECTION+%60travel-sample%60.%60post-upgrade%60.airport&scan_consistency=REQUEST_PLUS
      		 
      2021-05-26 15:32:41,096 - root - ERROR - POST http://172.23.100.14:8093/query?statement=CREATE+COLLECTION+%60travel-sample%60.%60post-upgrade%60.airport&scan_consistency=REQUEST_PLUS body:  headers: {'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=='} error: 500 reason: unknown b'{\n"requestID": "47d4cb54-96fc-4257-9f30-2d892dcaa466",\n"errors": [{"code":12021,"msg":"Scope not found in CB datastore default:travel-sample.post-upgrade"}],\n"status": "fatal",\n"metrics": {"elapsedTime": "6.204352ms","executionTime": "6.145016ms","resultCount": 0,"resultSize": 0,"serviceLoad": 1,"errorCount": 1}\n}\n' auth: Administrator:password
      		 
      2021-05-26 15:32:41,097 - root - ERROR - PROBLEM WITH RESULT. TYPE IS: <class 'dict'> AND CONTENT IS: {'requestID': '47d4cb54-96fc-4257-9f30-2d892dcaa466', 'errors': [{'code': 12021, 'msg': 'Scope not found in CB datastore default:travel-sample.post-upgrade'}], 'status': 'fatal', 'metrics': {'elapsedTime': '6.204352ms', 'executionTime': '6.145016ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 1, 'errorCount': 1}}
      		 
      2021-05-26 15:32:41,097 - root - ERROR - Query collection failed: CBQError: host 172.23.100.14: ERROR:{'requestID': '47d4cb54-96fc-4257-9f30-2d892dcaa466', 'errors': [{'code': 12021, 'msg': 'Scope not found in CB datastore default:travel-sample.post-upgrade'}], 'status': 'fatal', 'metrics': {'elapsedTime': '6.204352ms', 'executionTime': '6.145016ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 1, 'errorCount': 1}}

      Job for reference: http://qa.sc.couchbase.com/job/test_suite_executor/348958/consoleText

      Test flow is:

      • setup 4 nodes (kv:n1ql:index) with 6.0
      • remove node-1 and upgrade to 7.0
      • swap rebalance node-1 with node-2
      • at this point node-1 is on 7 and node-3 and node-4 are on 6
      • do some mix mode n1ql test (those are fine)
      • upgrade node-2 and swap rebalance with node-3
      • upgrade node-3 and swap reblance with node-4
      • at this point node-1, node-2 and node-3 are on 7.0 and node-4 is out of the cluster but still on 6.0
      • run n1ql post upgrade tests. The tests doing update stats and create index on scope fail with error above.

      See attachment for logs.

      Note that I cannot repro this manually or running with my own cluster.

      Note2: if i slightly tweak topology where node-4 is: kv:n1ql and other remain: kv:n1ql:index ... create index/update stats will work fine.

       

      Attachments

        For Gerrit Dashboard: MB-46575
        # Subject Branch Project Status CR V
        154747,2 MB-46575 : Update ServiceMap and retry if cluster version is old. unstable indexing Status: MERGED +2 +1

        Activity

          People

            pierre.regazzoni Pierre Regazzoni
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty