Critical Description
The test works as follows:
1) start with 3 4.6.5 nodes with query, index, kv services
2) create travel sample bucket, standard_bucket0, and default bucket and load data and create indexes
travelsample has 31k docs, and standard_bucket0 and default each have 2016 docs
3) stop all servers and offline upgrade the cluster, wait for 3 minutes
4) run select count
for each bucket
5) create index for ansi joins query
6) execute ansi joins query
There are two issues. The first and more minor issue is that in step 4) the results only show a fraction of the number of docs in the system, i.e 700 docs in default when the ui shows 2016. It doesn't happen 100% of the time but it happens very often.
The main issue is that the ansi joins query, which is run via rest api call, returns an error. This error is a broken pipe error and some documents are returned with the error. The expected number of documents to be return from this query is 288, but the query returns 286 documents and the broken pipe error. Two documents are missing. Now iif after the query returns the error I go and run the query from the UI, it returns correct results and no error. If I do the same from a rest api call, I get correct results and no error. If I change the test to run select * from travel-sample, i get back all data from the bucket and no error. I have run the test with 4 nodes and with 3 nodes and the same issue happens every time.
I have attached the test output and server logs.