[MB-43796] OSO Enabled Initial Index build time greater than OSO Disabled - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: Cheshire-Cat
Fix Version/s: 7.1.0
Component/s: secondary-index
Labels:
- performance
Environment:
7.0.0-4253

Triage:
Untriaged
Story Points:
1
Is this a Regression?:
Unknown

Description

On running a 1000 Index test on Plasma with OSO Enabled , we see that the time for initial build of indexes is more than that with OSO Disabled

The test config is as follows :

Create a bucket with 50M Docs on 100 Collections at ~20% Resident Ratio (KV)
Create 1000 Indexes using PLASMA
Calculate time to build these 1000 Indexes

Results are as follows :

Bucket RR	OSO	Initial Build Time	Job
~20%	Enabled	20.3mins	http://perf.jenkins.couchbase.com/job/hemera/744/
~20%	Disabled	10.8mins	http://perf.jenkins.couchbase.com/job/hemera/745/
100%	Enabled	19.9mins	http://perf.jenkins.couchbase.com/job/hemera/747/
100%	Disabled	10.6mins	http://perf.jenkins.couchbase.com/job/hemera/746/

All runs are on 7.0.0-4253.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Screenshot 2022-01-21 at 4.59.17 PM.png
121 kB
21/Jan/22 3:29 AM
Screenshot 2022-01-21 at 4.59.37 PM.png
119 kB
21/Jan/22 3:29 AM

Issue Links

relates to

MB-49702 Magma rolling back to zero despite rollback seqno being within the 10min/10% window

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

View 10 older comments

Deepkaran Salooja added a comment - 17/Nov/21 5:55 PM

1. Indexer needs to process a lot more mutations in OSO mode vs non-OSO.

Below is the projector ENDP stat for MAINT_STREAM after the test finishes.

OSO

2021-11-10T12:24:54.540-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,d4c4)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_a911101fa0309c36044eb6532996eaa9] stats: {"mutCount":9908014,"upsertCount":40317600,"deleteCount":0,"upsdelCount":0,"syncCount":81,"beginCount":205,"endCount":0,"snapCount":205,"flushCount":126901,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":5743048,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":17770,"latency.max":6711969895,"latency.avg":711535477,"latency.movingAvg":136782628,"endpChLen":0}

Non-OSO

 2021-11-10T13:06:27.061-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,9e46)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_e81c7d5409fd5b5e20e168db1312a092] stats: {"mutCount":280,"upsertCount":0,"deleteCount":0,"upsdelCount":0,"syncCount":75,"beginCount":205,"endCount":0,"snapCount":0,"flushCount":275,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":0,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":9223372036854775807,"latency.max":-9223372036854775808,"latency.avg":0,"latency.movingAvg":0,"endpChLen":0}

2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again.

OSO

2021-11-10T11:57:45.342-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, s

copeId: , collectionId: , vbuckets: 1024 Crc64: 12148381171197068428 snapType DISK_SNAP -

    {vbno, vbuuid, manifest, seqno, oscount, snapshot-start, snapshot-end}

    {    0     9a11417a8366 1        596        490        596        596}

    {    1     b612eda54df6 1        586        480        586        586}

    {    2     b1a1b2159ff9 1        593        479        593        593}

    {    3     183d014cfb3f 1        626        498        626        626}

    {    4     93dddf3e6c06 1        593        487        593        593}

    {    5     ae2da4b54760 1        620        494        620        620}

    {    6     e8d31d437176 1        622        501        622        622}

    {    7     240eb47a523b 1        615        492        615        615}

    {    8     4dec1f91ca80 1        616        491        616        616}

Non-OSO

2021-11-10T12:41:15.956-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, scopeId: , collectionId: , vbuckets: 1024 Crc64: 2435956341701773187 snapType DISK_SNAP -

    {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}

    {    0     4f7ed0f5e57f 1      49002          0      49002}

    {    1     6913c5f29bf6 1      48002          0      48002}

    {    2     774deaee7621 1      47902          0      47902}

    {    3     476e04b434ab 1      49802          0      49802}

    {    4     4874cc8df011 1      48702          0      48702}

    {    5     e3da329ebd1f 1      49402          0      49402}

    {    6     487288c0c025 1      50102          0      50102}

    {    7     254f9ff6bf89 1      49202          0      49202}

    {    8     20e3bb0ed1f4 1      49102          0      49102}

Deepkaran Salooja added a comment - 17/Nov/21 5:55 PM 1. Indexer needs to process a lot more mutations in OSO mode vs non-OSO. Below is the projector ENDP stat for MAINT_STREAM after the test finishes. OSO 2021-11-10T12:24:54.540-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,d4c4)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_a911101fa0309c36044eb6532996eaa9] stats: {"mutCount":9908014,"upsertCount":40317600,"deleteCount":0,"upsdelCount":0,"syncCount":81,"beginCount":205,"endCount":0,"snapCount":205,"flushCount":126901,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":5743048,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":17770,"latency.max":6711969895,"latency.avg":711535477,"latency.movingAvg":136782628,"endpChLen":0} Non-OSO 2021-11-10T13:06:27.061-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,9e46)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_e81c7d5409fd5b5e20e168db1312a092] stats: {"mutCount":280,"upsertCount":0,"deleteCount":0,"upsdelCount":0,"syncCount":75,"beginCount":205,"endCount":0,"snapCount":0,"flushCount":275,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":0,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":9223372036854775807,"latency.max":-9223372036854775808,"latency.avg":0,"latency.movingAvg":0,"endpChLen":0} 2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again. OSO 2021-11-10T11:57:45.342-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, s copeId: , collectionId: , vbuckets: 1024 Crc64: 12148381171197068428 snapType DISK_SNAP - {vbno, vbuuid, manifest, seqno, oscount, snapshot-start, snapshot-end} { 0 9a11417a8366 1 596 490 596 596} { 1 b612eda54df6 1 586 480 586 586} { 2 b1a1b2159ff9 1 593 479 593 593} { 3 183d014cfb3f 1 626 498 626 626} { 4 93dddf3e6c06 1 593 487 593 593} { 5 ae2da4b54760 1 620 494 620 620} { 6 e8d31d437176 1 622 501 622 622} { 7 240eb47a523b 1 615 492 615 615} { 8 4dec1f91ca80 1 616 491 616 616} Non-OSO 2021-11-10T12:41:15.956-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, scopeId: , collectionId: , vbuckets: 1024 Crc64: 2435956341701773187 snapType DISK_SNAP - {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end} { 0 4f7ed0f5e57f 1 49002 0 49002} { 1 6913c5f29bf6 1 48002 0 48002} { 2 774deaee7621 1 47902 0 47902} { 3 476e04b434ab 1 49802 0 49802} { 4 4874cc8df011 1 48702 0 48702} { 5 e3da329ebd1f 1 49402 0 49402} { 6 487288c0c025 1 50102 0 50102} { 7 254f9ff6bf89 1 49202 0 49202} { 8 20e3bb0ed1f4 1 49102 0 49102}

Couchbase Build Team added a comment - 03/Dec/21 8:54 AM

Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message:
~~MB-43796~~ use true_with_seqno_advanced for OSO mode

Couchbase Build Team added a comment - 03/Dec/21 8:54 AM Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message: MB-43796 use true_with_seqno_advanced for OSO mode

Couchbase Build Team added a comment - 03/Dec/21 8:54 AM

Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message:
~~MB-43796~~ enable OSO after cluster upgrade to 7.1

Couchbase Build Team added a comment - 03/Dec/21 8:54 AM Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message: MB-43796 enable OSO after cluster upgrade to 7.1

Deepkaran Salooja added a comment - 03/Dec/21 11:35 AM

Vikas Chaudhary, please retest with 7.1.0-1836 or later.

Deepkaran Salooja added a comment - 03/Dec/21 11:35 AM Vikas Chaudhary , please retest with 7.1.0-1836 or later.

Vikas Chaudhary added a comment - 21/Jan/22 3:29 AM

Now both OSO enabled and disabled time is 12.4

Vikas Chaudhary added a comment - 21/Jan/22 3:29 AM Now both OSO enabled and disabled time is 12.4

People

Assignee:: Vikas Chaudhary

Reporter:: Prajwal‌ Kiran Kumar‌ (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 21/Jan/21 2:51 AM

Updated:: 21/Jan/22 3:29 AM

Resolved:: 03/Dec/21 11:35 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 3 closed Gerrit changes

Hide There are 3 closed Gerrit changes

MB-43796 disable projector seq order check in OSO mode: Gerrit Review:

MB-43796 enable OSO after cluster upgrade to 7.1: Gerrit Review:

MB-43796 use true_with_seqno_advanced for OSO mode: Gerrit Review:

PagerDuty

Couchbase Server

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty