Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49192

XDCR - collections data ignored path could leads to panic

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.0.0, 7.0.1, 7.0.2, 7.1.0
    • XDCR
    • Untriaged
    • 1
    • No

    Description

      Originally filed as MB-49176 for reproduction steps.

      Seems to happen under the scenarios:

      1. There are a lot of mutations that are ignored/not being mapped
      2. Of the mutations not being mapped/ignored, many of them belong in the source default collection
      3. Explicit mapping is set (seems to happen with explicit mapping in MB-49176), but not necessarily a pre-requisite

      When a DCP (uprEvent) is translated to a MCRequest, a source namespace obj/ptr is being shared in between them. This object is being "recycled" back to the data object pool twice. When this happens, the object's strings' get edited twice as well...

      1st time - http://src.couchbase.org/source/xref/cheshire-cat/goproj/src/github.com/couchbase/goxdcr/parts/router.go#1970-1976
      2nd time - http://src.couchbase.org/source/xref/cheshire-cat/goproj/src/github.com/couchbase/goxdcr/parts/dcp_nozzle.go#1954-1957

      Normally, this isn't that big of an issue when mutations are being replicated because the source namespace isn't used anymore once a MCRequest is composed with a target collection ID.

      However, when broken maps are present and mutation is not meant to be replicated, XDCR will need to print messages/logs and raise events. This leads to the panic stack of:

      panic: Empty source namespace
       
      goroutine 1046 [running]:
      github.com/couchbase/goxdcr/metadata.(*CollectionNamespaceMapping).AddSingleSourceNsMapping(0xc000236440, 0xc0012219b0, 0xc000e7f6e0, 0x2c)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/metadata/collections.go:1543 +0x2e9
      github.com/couchbase/goxdcr/metadata.(*CollectionNamespaceMapping).AddSingleMapping(...)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/metadata/collections.go:1566
      github.com/couchbase/goxdcr/parts.(*CollectionsRouter).recordUnroutableRequest(0xc0002363c0, 0x50fad60, 0xc0011112c0)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/parts/router.go:1125 +0xb0b
      github.com/couchbase/goxdcr/parts.(*Router).RouteCollection(0xc0001781e0, 0x50fad60, 0xc0011112c0, 0xc000036460, 0x48, 0xc001da3680, 0x0, 0x0)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/parts/router.go:1720 +0x712
      github.com/couchbase/goxdcr/parts.(*Router).Route(0xc0001781e0, 0x5076380, 0xc001da3680, 0x0, 0x0, 0x0)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/parts/router.go:1661 +0xd6a
      github.com/couchbase/goxdcr/connector.(*Router).Forward(0xc0003ded20, 0x5076380, 0xc001da3680, 0x0, 0x0)
              /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/connector/router.go:113 +0x1f5
      github.com/couchbase/goxdcr/parts.(*DcpNozzle).processData(0xc000140a00, 0x0, 0x0)
      

      as XDCR realizes that the source namespace (scopeName, collectionName) must not be empty.

      This is not always reproducible, however... as this requires some timing issues.. The specific test case shown in MB-49176 can reproduce it, but a similarly created test case without the use of travel-sample could not.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-49192
          # Subject Branch Project Status CR V

          Activity

            People

              neil.huang Neil Huang
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty