Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44562

[System Test]: KV rebalance hung for default bucket

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • couchbase-bucket, fts
    • 7.0.0-4525
    • Triaged
    • 1
    • No
    • KV-Engine 2021-March

    Description

      Build: 7.0.0-4525

      Test:-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml

      Cycle: 1

      Test steps till kV rebalance

      • create 29 node cluster kv : 12, eventing ,cbas: 3, n1ql , fts: 2 , index 6 , back up : 1
      • create 10 buckets
      • 2 node xdcr setup
      • collection crud on bucket8 and bucket9 
      • create 4 collections on 4 buckets other the bucket8 and bucket9
      • deploy 4 eventing handlers
      • create indexes
      • load data to default bucket
      • rebalance in  https://172.23.106.100 as KV
      • load more data to default bucket
      • Rebalance in /out and swap 1 eventing node

      [2021-02-23T14:49:43-08:00, sequoiatools/couchbase-cli:7.0:83b899] server-add -c 172.23.108.103:8091 --server-add https://172.23.98.135 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      		 
      [2021-02-23T14:49:56-08:00, sequoiatools/couchbase-cli:7.0:83b9c1] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      		 
      [2021-02-23T15:21:04-08:00, sequoiatools/cmd:3e3c0f] 60
      		 
      [2021-02-23T15:22:11-08:00, sequoiatools/cmd:ca64ad] 300
      		 
      [2021-02-23T15:28:22-08:00, sequoiatools/couchbase-cli:7.0:5a3586] rebalance -c 172.23.108.103:8091 --server-remove 172.23.104.137 -u Administrator -p password
      		 
      [2021-02-23T16:01:23-08:00, sequoiatools/cmd:a0323f] 60
      		 
      [2021-02-23T16:02:31-08:00, sequoiatools/cmd:5af462] 300
      		 
      [2021-02-23T16:13:00-08:00, sequoiatools/couchbase-cli:7.0:e56a6d] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.137 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      		 
      [2021-02-23T16:13:14-08:00, sequoiatools/couchbase-cli:7.0:c5ac9e] rebalance -c 172.23.108.103:8091 --server-remove 172.23.121.3 -u Administrator -p password

      • Rebalance in/out/swap analytics node
      • create views
      • Rebalance in /out/swap index nodes
      • Remove 2 index nodes 172.23.99.11,172.23.104.69.  it fails due to index building is in progress
      • Remove kv node 172.23.106.100  it also fails as the indexer exits with 2 
      • Adding a kV node hung when we add .111

      [2021-02-24T01:01:02-08:00, sequoiatools/couchbase-cli:7.0:ba0e7a] server-add -c 172.23.108.103:8091 --server-add https://172.23.105.111 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2021-02-24T01:01:24-08:00, sequoiatools/couchbase-cli:7.0:c444cd] rebalance -c 172.23.108.103:8091 -u Administrator -p password

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-44919 KV is crashing {#0 0x00007f116996e61e in BackfillManager::moveFirstPendingToInitializing() () from /opt/couchbase/bin/../lib/libep.so}

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-52490 [30TB, 1% KV DGM, CBAS]: Rebalance in 1 KV node is stuck since 35 hours. No movement in data/vBuckets.

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened

          Bug - A problem which impairs or prevents the functions of the product. MB-44473 [System Test]:Rebalance to remove kv node taking more than 22 hrs

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty