Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54333

[7.0.5] - Rebalance operations fails because of backup service crashing

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Test Blocker
    • Resolution: Fixed
    • 7.0.5
    • 7.0.5
    • tools
    • Enterprise Edition 7.0.5 build 7632
    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Script to Repro

      ./testrunner -i /tmp/testexec.30477.ini -p get-cbcollect-info=False,sdk_compression=False,compression_mode=active,get-cbcollect-info=True,sdk_compression=False,compression_mode=active,skip_log_scan=False,bucket_storage=couchstore -t failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,withMutationOps=True,services_init=kv-kv-kv-kv,services_out=kv
      

      This fails for more or less for all tc's. Just attaching one sample test.
      Rebalance in/out operations fail on 7.0.5 consistently because of backup service crashing as shown below.

      172.23.96.18311:35:16 PM   30 Oct, 2022

      Service 'backup' exited with status 2. Restarting. Messages:
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/bufio/bufio.go:144 +0x5d
      net/http.(*persistConn).readLoop(0xc0000ea6c0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:2093 +0x1ac
      created by net/http.(*Transport).dialConn
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:1750 +0x173e
       
      goroutine 101 [select]:
      net/http.(*persistConn).writeLoop(0xc0000ea6c0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:2392 +0xf5
      created by net/http.(*Transport).dialConn
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/http/transport.go:1751 +0x1791
       
      goroutine 86 [runnable]:
      net/rpc.(*Server).ServeCodec.func1()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/rpc/server.go:478
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1571 +0x1
      created by net/rpc.(*Server).ServeCodec
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/net/rpc/server.go:478 +0x3fe
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            owend Daniel Owen added a comment -

            Hi Balakumaran Gopal, Is this a recent regression? Do you have the last known working build?
            (We will continue to investigate in parallel).
            thanks

            owend Daniel Owen added a comment - Hi Balakumaran Gopal , Is this a recent regression? Do you have the last known working build? (We will continue to investigate in parallel). thanks

            Daniel Owen - This is the first time we have ever run any test on 7.0.5

            Balakumaran.Gopal Balakumaran Gopal added a comment - Daniel Owen - This is the first time we have ever run any test on 7.0.5
            owend Daniel Owen added a comment -

            OK thanks - but it works reliably on 7.0.4?

            owend Daniel Owen added a comment - OK thanks - but it works reliably on 7.0.4?

            Looked at the old greenboard link which looks good.

            http://greenboard.sc.couchbase.com/#!/server/7.0.4/7279

            Balakumaran.Gopal Balakumaran Gopal added a comment - Looked at the old greenboard link which looks good. http://greenboard.sc.couchbase.com/#!/server/7.0.4/7279
            Matt.Hall Matt Hall added a comment -

            Here is the actual stack trace:

            2022-10-30T23:13:01.401-07:00 INFO (Leader Manager) Stepped up as leader
            [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x467a9f]
             
            goroutine 98 [running]:
            runtime.throw({0xec3176?, 0xc0005a5328?})
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/panic.go:992 +0x71 fp=0xc0005a5258 sp=0xc0005a5228 pc=0x437a71
            runtime.sigpanic()
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/signal_unix.go:825 +0x2ec fp=0xc0005a52a8 sp=0xc0005a5258 pc=0x44dc6c
            aeshashbody()
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc0005a52b0 sp=0xc0005a52a8 pc=0x467a9f
            runtime.mapiternext(0xc00020e600)
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:934 +0x2cb fp=0xc0005a5320 sp=0xc0005a52b0 pc=0x41168b
            runtime.mapiterinit(0xc0004b1ec0?, 0xc0001baa10?, 0x100c0001ba940?)
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:861 +0x228 fp=0xc0005a5340 sp=0xc0005a5320 pc=0x411368
            reflect.mapiterinit(0x0?, 0xc0005a53e8?, 0x40eea5?)
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:1373 +0x19 fp=0xc0005a5368 sp=0xc0005a5340 pc=0x464319
            github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112
            github.com/json-iterator/go.(*mapEncoder).Encode(0xc0004b1e90, 0xc000208230, 0xc000131d40)
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_map.go:257 +0x21c fp=0xc0005a5420 sp=0xc0005a5368 pc=0xbe711c
            github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0001ba9f0, 0xc0004b18f0, 0xc00020e600?)
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect.go:219 +0x82 fp=0xc0005a5458 sp=0xc0005a5420 pc=0xbdfe82
            github.com/json-iterator/go.(*Stream).WriteVal(0xc000131d40, {0xd99740, 0xc0004b18f0})
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect.go:98 +0x158 fp=0xc0005a54c8 sp=0xc0005a5458 pc=0xbdf198
            github.com/json-iterator/go.(*frozenConfig).Marshal(0xc00039ce60, {0xd99740, 0xc0004b18f0})
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/config.go:299 +0xc9 fp=0xc0005a5560 sp=0xc0005a54c8 pc=0xbd6449
            github.com/json-iterator/go.Marshal(...)
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/adapter.go:34
            github.com/couchbase/cbbs/leader.opaqueToNodeInfo({0xc0004b27e0, 0x20}, {0xd99740?, 0xc0004b18f0?})
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:897 +0x4a fp=0xc0005a55c0 sp=0xc0005a5560 pc=0xc54d2a
            github.com/couchbase/cbbs/leader.(*DefaultLeader).DoRebalance(0xc0004a2b40, {{0xc0004b27a0, 0x20}, {0x0, 0x0, 0x0}, {0xc0004b27c0, 0x19}, {0xc000432000, 0x1, ...}, ...})
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:451 +0x1685 fp=0xc0005a5f60 sp=0xc0005a55c0 pc=0xc50605
            github.com/couchbase/cbbs/leader.(*DefaultLeader).StartTopologyChange.func2()
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:269 +0x58 fp=0xc0005a5fe0 sp=0xc0005a5f60 pc=0xc4eef8
            runtime.goexit()
                    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x46a6a1
            created by github.com/couchbase/cbbs/leader.(*DefaultLeader).StartTopologyChange
                    /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:269 +0x57c
            

            Looks like we're hitting the same json-iterator issue as MB-54199

            Matt.Hall Matt Hall added a comment - Here is the actual stack trace: 2022-10-30T23:13:01.401-07:00 INFO (Leader Manager) Stepped up as leader [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x467a9f]   goroutine 98 [running]: runtime.throw({0xec3176?, 0xc0005a5328?}) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/panic.go:992 +0x71 fp=0xc0005a5258 sp=0xc0005a5228 pc=0x437a71 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/signal_unix.go:825 +0x2ec fp=0xc0005a52a8 sp=0xc0005a5258 pc=0x44dc6c aeshashbody() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc0005a52b0 sp=0xc0005a52a8 pc=0x467a9f runtime.mapiternext(0xc00020e600) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:934 +0x2cb fp=0xc0005a5320 sp=0xc0005a52b0 pc=0x41168b runtime.mapiterinit(0xc0004b1ec0?, 0xc0001baa10?, 0x100c0001ba940?) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:861 +0x228 fp=0xc0005a5340 sp=0xc0005a5320 pc=0x411368 reflect.mapiterinit(0x0?, 0xc0005a53e8?, 0x40eea5?) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/map.go:1373 +0x19 fp=0xc0005a5368 sp=0xc0005a5340 pc=0x464319 github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112 github.com/json-iterator/go.(*mapEncoder).Encode(0xc0004b1e90, 0xc000208230, 0xc000131d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_map.go:257 +0x21c fp=0xc0005a5420 sp=0xc0005a5368 pc=0xbe711c github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0001ba9f0, 0xc0004b18f0, 0xc00020e600?) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect.go:219 +0x82 fp=0xc0005a5458 sp=0xc0005a5420 pc=0xbdfe82 github.com/json-iterator/go.(*Stream).WriteVal(0xc000131d40, {0xd99740, 0xc0004b18f0}) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect.go:98 +0x158 fp=0xc0005a54c8 sp=0xc0005a5458 pc=0xbdf198 github.com/json-iterator/go.(*frozenConfig).Marshal(0xc00039ce60, {0xd99740, 0xc0004b18f0}) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/config.go:299 +0xc9 fp=0xc0005a5560 sp=0xc0005a54c8 pc=0xbd6449 github.com/json-iterator/go.Marshal(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.18.7/pkg/mod/github.com/json-iterator/go@v1.1.10/adapter.go:34 github.com/couchbase/cbbs/leader.opaqueToNodeInfo({0xc0004b27e0, 0x20}, {0xd99740?, 0xc0004b18f0?}) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:897 +0x4a fp=0xc0005a55c0 sp=0xc0005a5560 pc=0xc54d2a github.com/couchbase/cbbs/leader.(*DefaultLeader).DoRebalance(0xc0004a2b40, {{0xc0004b27a0, 0x20}, {0x0, 0x0, 0x0}, {0xc0004b27c0, 0x19}, {0xc000432000, 0x1, ...}, ...}) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:451 +0x1685 fp=0xc0005a5f60 sp=0xc0005a55c0 pc=0xc50605 github.com/couchbase/cbbs/leader.(*DefaultLeader).StartTopologyChange.func2() /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:269 +0x58 fp=0xc0005a5fe0 sp=0xc0005a5f60 pc=0xc4eef8 runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x46a6a1 created by github.com/couchbase/cbbs/leader.(*DefaultLeader).StartTopologyChange /home/couchbase/jenkins/workspace/couchbase-server-unix/cbbs/leader/topology_manager.go:269 +0x57c Looks like we're hitting the same json-iterator issue as MB-54199
            Matt.Hall Matt Hall added a comment - Hit https://issues.couchbase.com/browse/MB-54339 on CV

            Build couchbase-server-7.0.5-7634 contains cbbs commit 1b1d605 with commit message:
            MB-54333 Update json-iterator to avoid panic

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.5-7634 contains cbbs commit 1b1d605 with commit message: MB-54333 Update json-iterator to avoid panic
            Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on 7.0.5-7635. Logs :- http://greenboard.sc.couchbase.com/#!/server/7.0.5/7635?features=COLLECTIONS,EPHEMERAL,COMPRESSION,DURABILITY

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty