Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49359

[System Test][XDCR] panic observed in longevity with toy build - panic: runtime error: invalid memory address or nil pointer dereference

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • XDCR
    • Untriaged
    • 1
    • No

    Description

      While running the toy build provided for MB-48884, ran into this panic happening on multiple nodes:

       
      		 
      172.23.120.73 : crash
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-11-03T06:29:14.614-07:00,ns_1@172.23.120.73:<0.25836.381>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-11-03T07:30:22.156-07:00,ns_1@172.23.120.73:<0.25836.381>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
       
      		 
      172.23.96.122 : crash
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-11-03T06:16:10.625-07:00,ns_1@172.23.96.122:<0.10263.208>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-11-03T07:17:18.310-07:00,ns_1@172.23.96.122:<0.10263.208>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
       
      		 
      172.23.96.48 : crash
      		 
      [user:info,2021-11-03T06:33:49.367-07:00,ns_1@172.23.96.48:<0.6748.62>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:
      		 
      [user:info,2021-11-03T07:34:56.983-07:00,ns_1@172.23.96.48:<0.6748.62>:ns_log:consume_log:76]Service 'goxdcr' exited with status 2. Restarting. Messages:

      Stack trace from goxdcr logs:

      panic: runtime error: invalid memory address or nil pointer dereference
      		 
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x9160d3]
      		 
       
      		 
      goroutine 220 [running]:
      		 
      github.com/couchbase/goxdcr/peerToPeer.(*ReplicaCacheMonitor).GetReplicaInfo(0xc000124700, 0xc000440e40, 0xc0004a20f0, 0x2f, 0xc00031fdd0)
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaCache.go:139 +0x73
      		 
      github.com/couchbase/goxdcr/peerToPeer.(*ReplicaCacheImpl).GetReplicaInfo(0xc000124680, 0xc000132510, 0xc000ba3e40, 0x0, 0x0, 0xe95c80, 0xc000441b00, 0xc000c18880)
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaCache.go:92 +0x99
      		 
      github.com/couchbase/goxdcr/peerToPeer.(*ReplicaReplicatorImpl).reOrganizePopulateMap(0xc00084a380, 0xc000ffcde0, 0x1031313, 0x1)
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaReplicator.go:279 +0x10c
      		 
      github.com/couchbase/goxdcr/peerToPeer.(*ReplicaReplicatorImpl).pullAndSend(0xc00084a380, 0xc000ba3f44, 0x0)
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaReplicator.go:252 +0x57
      		 
      github.com/couchbase/goxdcr/peerToPeer.(*ReplicaReplicatorImpl).run(0xc00084a380)
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaReplicator.go:229 +0x268
      		 
      created by github.com/couchbase/goxdcr/peerToPeer.(*ReplicaReplicatorImpl).Start.func1
      		 
      	/home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/goxdcr/peerToPeer/replicaReplicator.go:202 +0x3e
      		 
      2021-11-03T06:29:14.865-07:00 INFO GOXDCR.ReplMgr: GOMAXPROCS=4
      		 
      2021-11-03T06:29:14.865-07:00 INFO GOXDCR.SecuritySvc: Security service started. Waiting for security context to be initialized
      		 
      2021-11-03T06:29:14.865-07:00 INFO GOXDCR.SecuritySvc: Received security change notification. code 7

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.151.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635951787/collectinfo-2021-11-03T150309-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MB-48884 [System Test] Few cluster operations failed due to error - Internal server error

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-49359
          # Subject Branch Project Status CR V
          165045,2 MB-49359 - return error if replica cache has not received info yet master goxdcr Status: MERGED +2 +1

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty