Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51159

[System test upgrade] - "Unable to respond to caller given type VBMasterCheck" errors seen during online upgrade using swap rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • XDCR
    • 7.0.3-7031 -> 7.1.0-2373
    • Centos 64-bit
    • 1
    • Unknown

    Description

      Script to Repro
      1. Run CC longevity on 7.0.3 for 2-3 days

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_second_cluster.yml -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml -scale 3 -repeat 0 -log_level 0 -version 7.0.3-7031 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

      2. Change the encryption level to strict.
      3. Add 2 7.1 kv nodes and 2 7.1 indexing nodes, Remove 2 7.0.3 kv and 2 7.0.3 indexing nodes and do swap rebalance.

      On the newly added 7.1 kv nodes we see the following errors.

      172.23.106.137

      2022-02-24 01:05:31,789 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:37:06.037-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4245225472 timed out chan 0xc00013f2a8
      		 
      2022-02-24 01:05:31,793 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:42:44.165-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2499084288 timed out chan 0xc001a5c210
      		 
      2022-02-24 01:05:31,796 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:49:00.354-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2757689344 timed out chan 0xc000dadad0
      		 
      2022-02-24 01:05:31,799 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:54:34.213-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2749956096 timed out chan 0xc000dad150
      		 
      2022-02-24 01:05:31,803 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:54:52.251-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2580480000 timed out chan 0xc0001b1c80
      		 
      2022-02-24 01:05:31,804 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:00:24.266-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 375652352 timed out chan 0xc0025b4390
      		 
      2022-02-24 01:05:31,808 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:01:16.254-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2695168000 timed out chan 0xc00038ab48
      		 
      2022-02-24 01:05:31,812 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:06:05.085-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3007447040 timed out chan 0xc001a5c090
      		 
      2022-02-24 01:05:31,816 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:06:54.256-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 547946496 timed out chan 0xc002277fb0
      		 
      2022-02-24 01:05:31,820 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.7.gz:2022-02-23T09:09:06.804-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3640918016 timed out chan 0xc000126660
      		 
      2022-02-24 01:05:31,821 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:27.611-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4057989120 timed out chan 0xc0002bc7d8
      		 
      2022-02-24 01:05:31,825 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:27.966-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4269211648 timed out chan 0xc0021b2e18
      		 
      2022-02-24 01:05:31,829 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.014-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 1363410944 timed out chan 0xc000625de0
      		 
      2022-02-24 01:05:31,832 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.079-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 940769280 timed out chan 0xc0002bd638

      172.23.106.138

      2022-02-24 01:05:39,179 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:37:03.791-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2064908288 timed out chan 0xc000010ba0
      		 
      2022-02-24 01:05:39,183 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:42:41.482-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2458451968 timed out chan 0xc000011638
      		 
      2022-02-24 01:05:39,188 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:48:57.053-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4183162880 timed out chan 0xc00406cfe0
      		 
      2022-02-24 01:05:39,193 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:54:31.494-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4258201600 timed out chan 0xc00038e680
      		 
      2022-02-24 01:05:39,197 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T18:54:50.227-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 1256390656 timed out chan 0xc0000ae660
      		 
      2022-02-24 01:05:39,201 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:00:31.618-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 730202112 timed out chan 0xc000d34060
      		 
      2022-02-24 01:05:39,202 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:01:16.276-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 183894016 timed out chan 0xc001eda1d8
      		 
      2022-02-24 01:05:39,207 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:06:12.759-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 813826048 timed out chan 0xc000c60000
      		 
      2022-02-24 01:05:39,211 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2022-02-23T19:06:51.635-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3638886400 timed out chan 0xc000a6ef48
      		 
      2022-02-24 01:05:39,215 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.7.gz:2022-02-23T09:11:41.719-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2581331968 timed out chan 0xc0037bbf30
      		 
      2022-02-24 01:05:39,220 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.7.gz:2022-02-23T09:11:41.804-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 30867456 timed out chan 0xc0029c45a8
      		 
      2022-02-24 01:05:39,221 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:25:35.681-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 389742592 timed out chan 0xc0011b0ba8
      		 
      2022-02-24 01:05:39,226 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:25:35.730-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2444689408 timed out chan 0xc000419520
      		 
      2022-02-24 01:05:39,230 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:25:35.761-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3443916800 timed out chan 0xc0000b4f40
      		 
      2022-02-24 01:05:39,234 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:25:35.815-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3402629120 timed out chan 0xc0004195b8
      		 
      2022-02-24 01:05:39,238 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.157-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 296681472 timed out chan 0xc0001b8448
      		 
      2022-02-24 01:05:39,239 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.186-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4293328896 timed out chan 0xc000419180
      		 
      2022-02-24 01:05:39,243 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.297-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3769761792 timed out chan 0xc0000b57f8
      		 
      2022-02-24 01:05:39,248 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/goxdcr.log.9.gz:2022-02-23T00:26:28.423-08:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 35389440 timed out chan 0xc0000b5fd0

      cbcollect_info attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty