Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54600

[BP 7.1.4] - XDCR - bucket topology service concurrent map iteration and map write

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 7.1.1, 7.1.2, 7.1.3
    • 7.1.4
    • XDCR
    • Untriaged
    • 1
    • No

    Description

      In customer situations, we're seeing the following concurrent map read and map write panic. Unfortunately, we didn't see this issue during 7.1.X testing.

      Note that the panics are coming from BucketTopologyService, which was added in 7.1.0, as MB-45983.

      fatal error: concurrent map read and map write
       
      goroutine 543 [running]:
      runtime.throw({0xf38bbc?, 0xc0321caea0?})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:992 +0x71 fp=0xc01fb17a70 sp=0xc01fb17a40 pc=0x43a971
      runtime.mapaccess2_faststr(0x108d440?, 0xc0326e3dc0?, {0xc0321caea0, 0x5b})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/map_faststr.go:117 +0x3d4 fp=0xc01fb17ad8 sp=0xc01fb17a70 pc=0x417c74
      github.com/couchbase/gomemcached/client.(*Client).StatsMapForSpecifiedStats(0xc010d5e000, {0xf1c885, 0x3}, 0x1f?)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1457 +0x1fd fp=0xc01fb17bb0 sp=0xc01fb17ad8 pc=0x717cbd
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsLegacyUpdater.func1()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:284 +0x389 fp=0xc01fb17d68 sp=0xc01fb17bb0 pc=0xcaba29
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce(0xc0003db900, {0xf23c46, 0xe}, 0xc0003903a0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1363 +0x571 fp=0xc01fb17f18 sp=0xc01fb17d68 pc=0xcb5af1
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1177 +0x1e5 fp=0xc01fb17fe0 sp=0xc01fb17f18 pc=0xcb3fe5
      runtime.goexit()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc01fb17fe8 sp=0xc01fb17fe0 pc=0x46cde1
      created by github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1161 +0x386
       
      goroutine 1 [chan receive, 57 minutes]:
      main.main()
              goproj/src/github.com/couchbase/goxdcr/main/main.go:275 +0x105b
      

      Similarly:

      fatal error: concurrent map iteration and map write
       
      goroutine 418 [running]:
      runtime.throw({0xf3f690?, 0x40?})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:992 +0x71 fp=0xc0179ad480 sp=0xc0179ad450 pc=0x43a971
      runtime.mapiternext(0xc0179ad5c0?)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/map.go:871 +0x4eb fp=0xc0179ad4f0 sp=0xc0179ad480 pc=0x414aab
      github.com/couchbase/goxdcr/base.GetKeysListFromStrStrMap(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2542
      github.com/couchbase/goxdcr/base.(*DcpStatsMapType).GreenClone(0xc019852370, 0xc0179ad838, 0xc0179ad828)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2600 +0x2ba fp=0xc0179ad690 sp=0xc0179ad4f0 pc=0x7d13da
      github.com/couchbase/goxdcr/service_impl.(*Notification).Clone(0xc016907d40, 0x1)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1934 +0x14f fp=0xc0179ad8d8 sp=0xc0179ad690 pc=0xcb9a6f
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getHighSeqnosUpdater.func1()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:796 +0x5c5 fp=0xc0179add68 sp=0xc0179ad8d8 pc=0xcb1505
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce(0xc000277b80, {0xf227d9, 0xc}, 0xc00041d590)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1363 +0x571 fp=0xc0179adf18 sp=0xc0179add68 pc=0xcb5af1
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1177 +0x1e5 fp=0xc0179adfe0 sp=0xc0179adf18 pc=0xcb3fe5
      runtime.goexit()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0179adfe8 sp=0xc0179adfe0 pc=0x46cde1
      created by github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1161 +0x386
      

      And a third signature:

      fatal error: concurrent map iteration and map write
       
      goroutine 247 [running]:
      runtime.throw({0xf3f690?, 0xc0228b7138?})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:992 +0x71 fp=0xc0299b6708 sp=0xc0299b66d8 pc=0x43a971
      runtime.mapiternext(0x6?)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/map.go:871 +0x4eb fp=0xc0299b6778 sp=0xc0299b6708 pc=0x414aab
      runtime.mapiterinit(0xc0005bb530?, 0xc02d063b60?, 0x80?)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/map.go:861 +0x228 fp=0xc0299b6798 sp=0xc0299b6778 pc=0x414568
      github.com/couchbase/goxdcr/utils.(*StringStringMapPool).Get(0xdf6420?, {0xc027801f00, 0x6, 0xc021b98ea0?})
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/utils/dataObjPools.go:246 +0x87 fp=0xc0299b6850 sp=0xc0299b6798 pc=0x892ac7
      github.com/couchbase/goxdcr/utils.(*StringStringMapPool).Get-fm({0xc027801f00?, 0xc0268843c0?, 0xc02c5f4920?})
              <autogenerated>:1 +0x37 fp=0xc0299b6880 sp=0xc0299b6850 pc=0xcdc497
      github.com/couchbase/goxdcr/base.(*StringStringMap).GreenClone(0xc025aa1ed0, 0xc0299b6a40)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2852 +0x176 fp=0xc0299b6978 sp=0xc0299b6880 pc=0x7d3356
      github.com/couchbase/goxdcr/service_impl.(*Notification).Clone(0xc00bcffd40, 0x1)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1950 +0x5fc fp=0xc0299b6bc0 sp=0xc0299b6978 pc=0xcb9f1c
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).runGC(0xc0005c8280)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1205 +0x10a fp=0xc0299b6f68 sp=0xc0299b6bc0 pc=0xcb42ca
      github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func1()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1148 +0x3e fp=0xc0299b6fe0 sp=0xc0299b6f68 pc=0xcb405e
      runtime.goexit()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0299b6fe8 sp=0xc0299b6fe0 pc=0x46cde1
      created by github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1141 +0x25e
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.4-3563 contains goxdcr commit a906ebf with commit message:
            MB-54600: ensure statsMapLegacy uses pool correctly

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.4-3563 contains goxdcr commit a906ebf with commit message: MB-54600 : ensure statsMapLegacy uses pool correctly

            Build couchbase-server-7.2.0-5093 contains goxdcr commit a906ebf with commit message:
            MB-54600: ensure statsMapLegacy uses pool correctly

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5093 contains goxdcr commit a906ebf with commit message: MB-54600 : ensure statsMapLegacy uses pool correctly

            People

              ritam.sharma Ritam Sharma
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty