Details
-
Improvement
-
Resolution: Unresolved
-
Critical
-
Lithium
-
None
Description
SG Version: Any Lithium build
CBL. Version: Tested with prelithium which passed before : 2.8.0-173, also any lithium builds
SG Config: sg collect
Environment details:
[couchbase_servers] in one cluster
|
cbs1=172.23.104.194 |
cbs2=172.23.106.104 |
|
|
[sync_gateways] in one cluster
|
sgw1=172.23.106.105 |
sgw2=172.23.106.106 |
|
|
[load_balancers] for above SGWS |
lb1=172.23.106.108 |
Steps to Reproduce:
1. set up 2 CBS in one CBS cluster, 2 SGWS in one SGW cluster and load balancer for those 2 sgws
2. have sgw with these flags on : use_views on, xattrs on, delta sync on
3. Create user, session and docs on SG
4. Create docs on CBL with attachments
Starting continuous push_pull replication between TestServer(CBL) and SGW
5. Start another process in parallel
keep updating docs on CBL in the new process
6. In main process , Upgrade SGW one by one on cluster config list with flags : ssl on, xattrs on, delta sync on
7. Upgrade CBS one by one on cluster config list
8. Restart SGWs after the server upgrade
9. Stop the process which started at step #5
9. Gather CBL docs new revs for verification
10. Compare rev id, doc body and revision history of all docs on both CBL and SGW
Actual Result:
Some docs with attachments has miss match in the doc body.
Latest test mismatch :
Doc in CBL :
{'date_time_added': '2021-05-05 23:19:45.305216', 'location': 'california', 'updates': False, 'list_of_dicts': [{'friend_one': 'ilMaTlDiEj'}, {'friend_two': 'XXPaowZvMi'}], 'list': [4235271151415218383, 3633622526361190104], 'channels': ['sg_user_channel'], '_attachments': {'0528a8c3-4609-4a6d-91fd-ab147bebd456.png': {'data': '77+9UE5HChoKAAAACklIRFIAAAAKAAAACggGAAAA77+9Ms+9AAAB77+9SURBVHjvv70F77+9bSjvv70BAAfvv73vv73vv70jZ++/ve+/vd2577+9O++/vQvvv73vv73vv73vv71H77+9eUnvv71Ccu+/vRBfZO+/vUgyIu+/vRbvv73vv73vv71BKS/vv73vv70i77+9HNe0Xe+/ve+/vQgX77+977+977+9S++/ve+/ve+/ve+/ve+/ve+/vTDvv73vv73vv70PUxfvv73vv73vv71d77+977+977+9UMas77+977+9K3Xvv71mYxbdiQPvv73vv73vv73vv73vv73vv70z77+977+9Hu+/ve+/ve+/vU3vv71/77+9UHg277+977+9Eu+/ve+/vRcQVWfvv70R77+977+9bhAR77+977+9C8ObbQg777+9Tu+/vR8b77+9VhoGQ++/vVtc77+9VO+/ve+/vRF877+977+9Lxfvv73vv73vv71Bae+/vceT77+977+9c3woCUDvv73vv70HZO+/vRwi77+977+977+9au+/vc69Fyfvv70vMVLvv73vv73vv70VCu+/vWUL77+977+9VO+/vSgxQe+/ve+/ve+/ve+/vXsLcu+/vX8gybsR77+977+9B++/ve+/vQrvv70FVe+/ve+/ve+/vQc377+977+9Me+/vWXlp7JyPu+/vXwMHe+/ve+/ve+/vSzvv73vv73vv70Jdu+/ve+/ve+/vVzvv71AYQxW77+9R0hQOu+/ve+/ve+/vVZU77+977+90ZHvv70G77+9OO+/vdGWFiwvBu+/vV3vv70aYDhwUe+/vTrvv70iYe+/ve+/ve+/vVXvv73vv71mcF8IZO+/ve+/ve+/vTcn77+9zK3vv71l77+977+9SGLvv71477+977+977+977+9bBo/77+9N++/vQ7vv70R77+9D05TIe+/ve+/vT5h77+93rho77+977+9FGRSI++/ve+/vV/vv73vv73vv70h77+9Ne+/ve+/vVZ877+9Zkvvv73vv70W77+9TAzvv71N77+977+90IPvv73vv70777+977+977+977+9NxIAAAAASUVORO+/vUJg77+9'}, 'dfda8546-e2ef-4b24-812a-b13186ac6d22.png': {'data': '77+9UE5HChoKAAAACklIRFIAAAAKAAAACggGAAAA77+9Ms+9AAAB77+9SURBVHjvv70F77+9byjvv70BAAbvv71377+977+977+9Ye+/vS3vv73vv73vv71i77+977+977+9be+/vRMRZTfvv71dKVox77+9X++/ve+/vTrFp1NHOu+/vTvvv71/77+977+977+977+9Rk3duu+/ve+/vUQk77+977+9TO+/vXTvv73vv71KW++/vXJuN++/ve+/ve+/vXnvv73vv71Z77+9Z++/vTbvv70r77+977+977+977+9Q++/vX9Fc++/vRpjfF3vv71/77+9xLLvv70C77+977+9Xynvv70I77+977+977+9YzTvv70pMgRn77+977+977+9fu+/vXpfM0YVQMy1D++/ve+/vTp4x6chFXY9WDjvv73vv700SQ9T77+9A8WaTFjvv73vv71h77+977+9IO+/vc6A77+977+9EC7vv71LAO+/vXoW77+977+977+977+977+977+9fSfvv73vv73vv73Dv++/vX7vv71MPO+/vUlAyZxI77+9d++/ve+/vQnvv73vv73vv71VZS0UAznvv71aXe+/vRfvv71PDu+/vTrvv73vv70OUe+/vQrvv73vv73vv71YI++/ve+/ve+/vX0uXlrvv70c1oXvv73vv70x77+977+9RAvTs9ycTFZS77+9O++/vWzvv73vv70DD++/ve+/vRXvv70Z77+9ZnZc77+9Du+/ve+/ve+/ve+/vUIaWu+/ve+/ve+/vR/vv73Tl0vvv70PFT3vv73vv73Ekxkb77+977+9Shjvv70PUVPvv71M77+977+9Y1pO77+977+977+9GO+/vWbvv73vv73Eh39SKGnvv71rYO+/ve+/vQlHeW9Q77+977+977+977+977+977+9fu+/vQrvv70KexBe77+977+9bu+/vWIu77+9ER7vv73vv73vv73vv70QEdCU77+977+977+9GiPvv70/77+977+9eGPvv73vv73vv70TTu+/ve+/ve+/ve+/ve+/vQ4u77+9f++/vR7vv70o77+977+9I2IGIgAAAABJRU5E77+9QmDvv70='}}, 'dict_with_list': {'list': [True, True]}, 'dict': {'name': 'FbcSdONpjI'}, 'id': 'cbl_filter_1047', 'numOfUpdates': 3} |
Doc in SGW :
{
|
_attachments: {
|
0528a8c3-4609-4a6d-91fd-ab147bebd456.png: { |
digest: "sha1-ohjm5EaR32Ot6QwdINtogwrXV7A=", |
length: 846, |
revpos: 1, |
stub: true |
},
|
dfda8546-e2ef-4b24-812a-b13186ac6d22.png: {
|
digest: "sha1-Rd7MQHGHONEQ5723D4h0dGHkfWs=", |
length: 839, |
revpos: 1, |
stub: true |
}
|
},
|
_id: "cbl_filter_1047", |
_rev: "1-b69aba3260d8ba831dca71819b8dbe04f4a115e5", |
channels: [
|
"sg_user_channel"
|
],
|
date_time_added: "2021-05-05 23:19:45.305216", |
dict: {
|
name: "FbcSdONpjI" |
},
|
dict_with_list: {
|
list: [
|
true, |
true
|
]
|
},
|
id: "cbl_filter_1047", |
list: [
|
4235271151415218000, |
3633622526361190000
|
],
|
list_of_dicts: [
|
{
|
friend_one: "ilMaTlDiEj" |
},
|
{
|
friend_two: "XXPaowZvMi" |
}
|
],
|
location: "california", |
updates: 0 |
}
|
{color:red}'numOfUpdates': 3 key and value is missing in SGW {color} |
Expected Result: Doc body of all docs on SGW and CBL should match after the upgrade and replication completed
Error in logs :
2021-05-05T23:31:38.129-07:00 [ERR] c:[65ae75ab] Error during downloadOrVerifyAttachments for doc cbl_filter_1047/2-0be76a85c1dc25329271809ba68ba4f3a6839849: 403 Incorrect proof for attachment sha1-Rd7MQHGHONEQ5723D4h0dGHkfWs= -- db.(*blipHandler).handleRev() at blip_handler.go:777 |
Logs :
SGW LOGS: sgcollect info:
SGW1 log: sgcollect_info_sgw1.zip
SGW2. log: sg_collect_info_upgrade_sgw2.zip
Github link for the code: https://github.com/couchbaselabs/mobile-testkit/blob/master/testsuites/syncgateway/upgrade/testserver/sg_single_sgw/test_upgrade.py
Jenkins job failure link: http://uberjenkins.sc.couchbase.com:8080/job/sync-gateway-upgrade-centos-testserver/1196/testReport/
Pytest Command : pytest --timeout 360000 -s --mode=cc --server-version=6.6.0-7924 --server-upgraded-version=7.0.0-4907 --sync-gateway-version=2.7.0 --sync-gateway-upgraded-version=3.0.0-184 --use-views --xattrs --delta-sync --upgraded-server-ssl --upgraded-xattrs --upgraded-delta-sync --number-replicas=0 --upgraded-number-replicas=0 --liteserv-host=localhost --liteserv-port=8080 --liteserv-version=2.8.0-173 --liteserv-platform=ios --num-docs=10000 --cluster-conf=2each_lb --cbs-platform=centos7 testsuites/syncgateway/upgrade/testserver/sg_single_sgw/test_upgrade.py
It passes with delta sync off after the sgw upgrade
CBL logs on iOS : sgw_upgrade_cbl_logs.zip
Attachments
Issue Links
- relates to
-
CBL-2199 20% - Sprint 52
-
- Closed
-
