Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41890

[XDCR] goxdcr crashed when replication running

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • XDCR
    • Centos 7.6
    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Install Couchbase server 7.0.0-3291 on 8 centos servers.

      Create default bucket on both src and remote cluster

      Load 1000 keys at src cluster

      Create remote cluster and set replication from src to remote.

      At remote cluster, only ~300 keys receive, and hang.

      Check src cluster log, there are a lot of goxdcr crashed 

      Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
      2020-10-06T23:06:26.783-07:00 WARN GOXDCR.ResourceMgr: Could not retrieve runtime stats for 1e66dd75e9ff0cb7cf6a4365c770c751/default/default. err=Cannot find overview stats for 1e66dd75e9ff0cb7cf6a4365c770c751/default/default
      		 
      2020-10-06T23:06:26.786-07:00 INFO GOXDCR.RemoteClusterChangeListener: metakvCallback called on listener RemoteClusterChangeListener with path = /remoteCluster/7C4lBNPAX5LtsHxXa7PCGF7VDsRO5r6b7CSqbMLh7QU=
      		 
      2020-10-06T23:06:26.786-07:00 INFO GOXDCR.RemClusterSvc: metakvCallback called on path = /remoteCluster/7C4lBNPAX5LtsHxXa7PCGF7VDsRO5r6b7CSqbMLh7QU=
      		 
      panic: Cannot continue without retrieving remote cluster reference
      		 
       
      		 
      goroutine 27 [running]:
      		 
      github.com/couchbase/goxdcr/metadata_svc.(*CollectionsManifestAgent).populateRemoteClusterRefOnce(0xc0002be820)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/collections_manifest_service.go:715 +0x101
      		 
      created by github.com/couchbase/goxdcr/metadata_svc.(*CollectionsManifestAgent).Start
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/collections_manifest_service.go:744 +0x227

      Test to run 

      p3 testrunner.py  -i ../../ini/8-personal-tools.ini  -p get-cbcollect-info=False,bucket_storage=couchstore -t cbRecoverytests.cbrecovery.cbrecover_multiple_failover_swapout_reb_routine,items=1000,rdirection=unidirection,ctopology=chain,failover=source,fail_count=2,add_count=2,max_verify=1000

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-41318 [XDCR] panic: Cannot continue without requesting remote monitoring

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              neil.huang Neil Huang
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty