Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54333

[7.0.5] - Rebalance operations fails because of backup service crashing

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Script to Repro

      ./testrunner -i /tmp/testexec.30477.ini -p get-cbcollect-info=False,sdk_compression=False,compression_mode=active,get-cbcollect-info=True,sdk_compression=False,compression_mode=active,skip_log_scan=False,bucket_storage=couchstore -t failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,withMutationOps=True,services_init=kv-kv-kv-kv,services_out=kv
      

      This fails for more or less for all tc's. Just attaching one sample test.
      Rebalance in/out operations fail on 7.0.5 consistently because of backup service crashing as shown below.

      172.23.96.18311:35:16 PM   30 Oct, 2022

      Service 'backup' exited with status 2. Restarting. Messages:
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/bufio/bufio.go:144 +0x5d
      net/http.(*persistConn).readLoop(0xc0000ea6c0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:2093 +0x1ac
      created by net/http.(*Transport).dialConn
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:1750 +0x173e
       
      goroutine 101 [select]:
      net/http.(*persistConn).writeLoop(0xc0000ea6c0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:2392 +0xf5
      created by net/http.(*Transport).dialConn
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:1751 +0x1791
       
      goroutine 86 [runnable]:
      net/rpc.(*Server).ServeCodec.func1()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/rpc/server.go:478
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1571 +0x1
      created by net/rpc.(*Server).ServeCodec
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/rpc/server.go:478 +0x3fe
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-54333
          # Subject Branch Project Status CR V

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty