2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again.
Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message: MB-43796 use true_with_seqno_advanced for OSO mode
Couchbase Build Team
added a comment - Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message:
MB-43796 use true_with_seqno_advanced for OSO mode
Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message: MB-43796 enable OSO after cluster upgrade to 7.1
Couchbase Build Team
added a comment - Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message:
MB-43796 enable OSO after cluster upgrade to 7.1
1. Indexer needs to process a lot more mutations in OSO mode vs non-OSO.
Below is the projector ENDP stat for MAINT_STREAM after the test finishes.
OSO
2021-11-10T12:24:54.540-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,d4c4)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_a911101fa0309c36044eb6532996eaa9] stats: {"mutCount":9908014,"upsertCount":40317600,"deleteCount":0,"upsdelCount":0,"syncCount":81,"beginCount":205,"endCount":0,"snapCount":205,"flushCount":126901,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":5743048,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":17770,"latency.max":6711969895,"latency.avg":711535477,"latency.movingAvg":136782628,"endpChLen":0}
Non-OSO
2021-11-10T13:06:27.061-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,9e46)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_e81c7d5409fd5b5e20e168db1312a092] stats: {"mutCount":280,"upsertCount":0,"deleteCount":0,"upsdelCount":0,"syncCount":75,"beginCount":205,"endCount":0,"snapCount":0,"flushCount":275,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":0,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":9223372036854775807,"latency.max":-9223372036854775808,"latency.avg":0,"latency.movingAvg":0,"endpChLen":0}
2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again.
OSO
2021-11-10T11:57:45.342-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, s
copeId: , collectionId: , vbuckets: 1024 Crc64: 12148381171197068428 snapType DISK_SNAP -
{vbno, vbuuid, manifest, seqno, oscount, snapshot-start, snapshot-end}
{ 0 9a11417a8366 1 596 490 596 596}
{ 1 b612eda54df6 1 586 480 586 586}
{ 2 b1a1b2159ff9 1 593 479 593 593}
{ 3 183d014cfb3f 1 626 498 626 626}
{ 4 93dddf3e6c06 1 593 487 593 593}
{ 5 ae2da4b54760 1 620 494 620 620}
{ 6 e8d31d437176 1 622 501 622 622}
{ 7 240eb47a523b 1 615 492 615 615}
{ 8 4dec1f91ca80 1 616 491 616 616}
Non-OSO
2021-11-10T12:41:15.956-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, scopeId: , collectionId: , vbuckets: 1024 Crc64: 2435956341701773187 snapType DISK_SNAP -
{vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}
{ 0 4f7ed0f5e57f 1 49002 0 49002}
{ 1 6913c5f29bf6 1 48002 0 48002}
{ 2 774deaee7621 1 47902 0 47902}
{ 3 476e04b434ab 1 49802 0 49802}
{ 4 4874cc8df011 1 48702 0 48702}
{ 5 e3da329ebd1f 1 49402 0 49402}
{ 6 487288c0c025 1 50102 0 50102}
{ 7 254f9ff6bf89 1 49202 0 49202}
{ 8 20e3bb0ed1f4 1 49102 0 49102}