Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36962

Test test_employee_dataset_startkey_endkey_queries_rebalance_in fails

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • master
    • master
    • view-engine
    • None
    • Untriaged
    • Yes

    Description

      Summary

      As seen in KV-Engine post-commit job, the test test_employee_dataset_startkey_endkey_queries_rebalance_in from suite simple-test fails with the following error:

      2019-11-18 15:13:31 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9500/default/_design/test_view-c342a5a/_view/test_view-c342a5a?startkey=%5B2008%2C0%2C1%5D&endkey=%5B2008%2C7%2C1%5D&connection_timeout=60000&inclusive_end=false body:  headers: {'Content-Typ
      e': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 500 reason: undef {"error":"undef","reason":"[{couch_view_parser,start_context,[],[]},\n {couch_http_view_streamer,parse,3,\n                           [{file,\"/home/daver/repos/couchbase/server/
      couchdb/src/couch_index_merger/src/couch_http_view_streamer.erl\"},\n                            {line,23}]},\n {couch_index_merger,run_http_index_folder,5,\n                     [{file,\"/home/daver/repos/couchbase/server/couchdb/src/couch_index_merger/src/couch_index_merger.erl\"},\n               
             {line,642}]},\n {couch_index_merger,http_index_folder,5,\n                     [{file,\"/home/daver/repos/couchbase/server/couchdb/src/couch_index_merger/src/couch_index_merger.erl\"},\n                      {line,597}]}]"} auth: Administrator:asdasd           
      

      Looking at the view engine logs for node n_0 (attached), the first error I can see which seems related is:

      [couchdb:error,2019-11-18T15:09:49.985Z,couchdb_n_0@cb.local:<0.466.0>:couch_log:error:44]dcp client (default, mapreduce_view: default _design/test_view-f5b5e71 (prod/main)): Obtaining mutation from server timed out after 60.0 seconds [RequestId 1088, PartId 85, StartSeq 0, CurrentSnapshotSeq 1, EndSeq 1, SnapshotBoundary {0,1}, unacked bytes 620, Ack Threshold 4194304.0] Waiting...
      ...
      

      git bisect reports the following as the first bad commit:

      1b3e4c636cb71bc323a1f81463b31c3e300e7cf is the first bad commit                                                          
      commit e1b3e4c636cb71bc323a1f81463b31c3e300e7cf                                                                                                                                                  
      Author: Jim Walker <jim@couchbase.com>                                                                                                                                                           
      Date:   Thu May 23 13:02:07 2019 +0100                                                                                                                                                         
                                                                                                                                                                                             
          MB-23086: Use platform_cbassert                                                                                    
                                                                                                                   
          couchdb only requires the cbassert functionality, which is now                                                                  
          a separate library.                                                                              
                                                                                                              
          Change-Id: I88a7d20134b4f2d572ce1d1c7161229c9b7e9ee6                                                       
          Reviewed-on: http://review.couchbase.org/109566                                                       
          Reviewed-by: <ankit.prabhu@couchbase.com>                                                          
          Tested-by: <ankit.prabhu@couchbase.com>                
      

      Steps to Reproduce
      Build as per job kv_engine-master-post-commit - i.e. with AddressSanitizer enabled:

      make -j4 EXTRA_CMAKE_OPTIONS=-D CB_ADDRESSSANITIZER=1
      

      1. Run make-simple test (note you can modify conf/simple.conf to just run the

        make -C testrunner simple-test
        

        (Note: You can modify conf/simple.conf to just run the affected test view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in by commenting out the other tests).

      Expected Results
      All tests pass:

      Ran 1 test in 317.707s                                                                           
                                                                                                                
      OK                                                                                                      
      Cluster instance shutdown with force                                                             
      summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0                              
      testrunner logs, diags and results are available under /home/daver/repos/couchbase/server/testrunner/logs/testrunner-19-Nov-18_15-43-35/test_1
      Run after suite setup for view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in
      view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in  pass
      

      Actual Results
      The aforementioned test fails:

      Ran 1 test in 92.689s                                                                                                                                                                     
                                                                                                                                                                                                
      FAILED (failures=1)                                                                                                                                                                     
      Cluster instance shutdown with force                                                                                                                                                        
      grabbing logs (cluster-run)                                                                                                                                                                     
      summary so far suite view.viewquerytests.ViewQueryTests , pass 0 , fail 1                                                                                                            
      failures so far...                                                                                                                                                                            
      view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in                                                                                        
      testrunner logs, diags and results are available under /home/daver/repos/couchbase/server/testrunner/logs/testrunner-19-Nov-18_16-04-47/test_1                                                 
      Run after suite setup for view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in                                                                    
      view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in  fail
      

      Artefacts

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            drigby Dave Rigby (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty