Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-63300

[System Test] :- Rebalance stuck for 125+ hours on longevity cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • 7.2.6
    • couchbase-bucket
    • Enterprise Edition 7.2.6 build 8112
    • Untriaged
    • Linux x86_64
    • 0
    • Yes

    Description

      Script to Repro

      ./sequoia -client 172.23.97.180:2375 -provider file:centos_second_cluster.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 1 -repeat 0 -log_level 0 -version 7.2.6-8112 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      Rebalance has been stuck at bucket8 for 125+ hours now. See some error messages like below. Not sure if they are of any importance.

      172.23.218.183

      root@s12903-deb12:/opt/couchbase/var/lib/couchbase/logs# tail -f memcached.log.159013.txt
      2024-08-26T01:16:35.237988-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:2513
      2024-08-26T01:16:35.237999-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:1417
      2024-08-26T01:16:35.238002-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:2704
      2024-08-26T01:16:35.238005-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:1334
      2024-08-26T01:16:35.238009-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:2667
      2024-08-26T01:16:35.238012-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:1368
      2024-08-26T01:16:35.238038-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:2501
      2024-08-26T01:16:35.238042-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:1349
      2024-08-26T01:16:35.238045-07:00 WARNING (bucket8) PersistenceCallback::del: Fatal error in persisting DELETE on vb:768 seqno:2507
      
      

      Attachments

        Issue Links

          Activity

            People

              owend Daniel Owen
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty