Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

KV swap-rebalance is hung forever due to DcpConsumer missing wakeup to transmit StreamRequest

Description

  1. Create a 3KV, 2 GSI-N1QL node cluster.

  2. Create magma bucket and 10 collections.

  3. Create 100000000 items: CircularKey

  4. Scale OUT with Loading of docs by 1 kv node & 1 GSI node one at a time

  5. Scale OUT with Loading of docs by 1 kv node & 1 GSI node one at a time

  6. Scale IN with Loading of docs by 1 kv node & 1 GSI node one at a time

  7. Scale IN with Loading of docs by 1 kv node & 1 GSI node one at a time

  8. Scale Disk UP with Loading of docs

  9. Scale Disk DOWN with Loading of docs

  10. Scale Compute UP with Loading of docs - Vertical scaling which lead to swap rebalance of each node

  11. Scale Compute DOWN with Loading of docs - Vertical scaling which lead to swap rebalance of each node

  12. Scale Disk + Compute UP with Loading of docs - Vertical scaling which lead to swap rebalance of each node

  13. KV node swap rebalance is hung forever.

    Starting rebalance, KeepNodes = ['ns_1@svc-d-node-019.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com', 'ns_1@svc-d-node-020.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com', 'ns_1@svc-d-node-024.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com', 'ns_1@svc-qi-node-022.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com', 'ns_1@svc-qi-node-023.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com'], EjectNodes = ['ns_1@svc-d-node-021.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 6c6c808a64eabb6073e6ab6402a094af

QE test

sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/couchbase_capella_volume_2_new.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.hostedHospital.Murphy.test_rebalance,num_items=100000000,num_buckets=1,bucket_names=GleamBook,bucket_type=membase,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,maxttl=10,pc=10,gsi_nodes=2,cbas_nodes=3,fts_nodes=3,kv_nodes=3,n1ql_nodes=2,kv_disk=1500,n1ql_disk=50,gsi_disk=500,fts_disk=1500,cbas_disk=1500,kv_compute=n2-standard-16,gsi_compute=n2-standard-16,n1ql_compute=n2-standard-16,fts_compute=n2-standard-16,cbas_compute=n2-standard-16,mutation_perc=20,key_type=CircularKey,capella_run=true,services=data-index:query,rebl_services=data-index:query,max_rebl_nodes=27,provider=GCP,region=us-east1,type=PD-SSD,size=1500,ops_rate=100000,skip_teardown_cleanup=true,wait_timeout=14400,index_timeout=28800,runtype=dedicated,skip_init=false,rebl_ops_rate=10000,collections=10,expiry=true,vh_scaling=true,horizontal_scale=1,clients_per_db=10 -m rest'

Components

Affects versions

Fix versions

Labels

Environment

7.6.0-1690

Link to Log File, atop/blg, CBCollectInfo, Core dump

kv_rebl_hung → http://supportal.couchbase.com/snapshot/bbf72f339ff5758d98368d6daf2a2afb::0

Release Notes Description

None

Attachments

2
  • 30 Oct 2023, 04:25 PM
  • 30 Oct 2023, 01:13 AM

Activity

Ritesh Agarwal February 28, 2024 at 1:34 AM

Working fine on Enterprise Edition 7.6.0 build 2164

CB robot November 3, 2023 at 12:01 AM

Build couchbase-server-8.0.0-1454 contains kv_engine commit 4e1aab4 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-59350#icft=MB-59350: Re-notify DCP conns if timeslice exceeded & more data

CB robot November 2, 2023 at 5:44 PM

Build capella-analytics-1.0.0-1072 contains kv_engine commit 4e1aab4 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-59350#icft=MB-59350: Re-notify DCP conns if timeslice exceeded & more data

CB robot November 2, 2023 at 5:22 PM

Build couchbase-server-7.6.0-1743 contains kv_engine commit 4e1aab4 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-59350#icft=MB-59350: Re-notify DCP conns if timeslice exceeded & more data

Trond Norbye November 2, 2023 at 10:55 AM

That seems like the problem.. As you say it doesn't account for the timed timeslice. (We should probably look into replacing the "numEvent" timeslice with just using the clock as that's probably more fair)

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

Unknown

Triage

Untriaged

Story Points

Sprint

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created October 30, 2023 at 1:13 AM
Updated February 28, 2024 at 1:35 AM
Resolved November 3, 2023 at 10:04 AM
Instabug